Accelerating serverless development with blueprints and governance
I spend lots of time talking to organizations about how to build and deploy serverless applications. While many people are still learning best practices, I also run into a bunch of folks who have formulated personal preferences on how they define and deploy cloud infrastructure resources. Many of them also have to operate within the choices and constraints of their organization.
In this post, I wanted to walk through my own personal preferences for building serverless applications and cover how I landed on those choices. This post is not about the “right” way but more about navigating trade-offs. I also wanted to cover how the lessons learned from that journey led me to start building the beginnings of a serverless landing zone. A serverless landing zone is aimed at organizations that are looking to build secure and governed capabilities for developing and deploying serverless applications.
I hope this is a helpful resource for you, especially if you are in the middle of trying to figure out your own personal best practices or are researching a serverless governance-in-depth strategy for your organization.
Refining how I define and deploy AWS resources
When I first started building on AWS, I used CloudFormation and wrote wrapper scripts in either python or bash to inject parameters and manage different actions for my templates. When I started doing more serverless development on AWS, I switched to using SAM, which provides transform macros for simplifying the definition of serverless resources. I also started using the SAM CLI for testing locally and for deploying resources into my AWS account. I particularly liked how sam deploy shows the deployment logs directly in the terminal, rather than having to check the AWS web console. Less mouse clicks! So I end up defining even non-serverless resources using SAM and the SAM CLI.
Eventually, I also switched to using make since it enabled me to write directives for single commands and for compositions of multiple commands. For example, I now have a standard makefile with directives to build, package, and deploy my functions, to test my function locally, and to invoke my deployed function. This gives me the granularity of an individual command while also being able to bundle multiple commands.
This approach also provides me with two additional benefits. These CLI commands tend to be highly parameterized. First, I don’t need to remember all the commands because they’re saved as make directives. Second, I can inject the variables via a separate environment file. Thus the makefile doesn’t change much after initially setting it up, and the environment file serves as my configuration file. This separation of directives and configuration could then allow one to leverage GitOps processes that drive deployment based on configuration files stored in git repositories.
Using this approach, I’ve built my own repository of templates (or otherwise referred to as blueprints for the rest of this post) that I use to start all my projects on AWS. This starts with blueprints for a basic Lambda function but extends to blueprints for API Gateway, EventBridge, and Step Functions. While it pales in comparison to the patterns collection on serverlessland.com, it is simply my opinionated approach to building these patterns.
Furthermore, I try to build in enterprise considerations into my starter kits, like including the required properties for a private API Gateway endpoint, specifying explicit logging configurations, or using OpenAPI specifications to document my API endpoints. I even build vanilla CloudFormation resources, like this blueprint for an EC2 bastion host.
Recognizing that not everyone uses CloudFormation or SAM, I also built out alternate blueprints using CDK, Terraform, and the Serverless Framework.
Considering governance requirements for enterprises
Blueprints as quick starter kits for particular architectural patterns are a good starting point. However, I also see that organizations need to implement comprehensive software delivery pipelines. It might include aspects that are depicted below:
You can replace the tools in the orange boxes with a wide variety of tools, depending on which development language and framework that you and your organization use.
I tend to see needs in two areas in this space: 1) those who have existing pipelines and are looking to augment their existing capabilities with additional controls and 2) those who might not have much and are looking to essentially build green field.
For those in the first camp, it’s unlikely that components of the pipeline will change unless there is a significant deficiency in the existing capability. For those in the second camp, it’s likely that guidance is needed with a lot of flexibility to customize the pipeline to suite the needs of your organization. In both cases, solving for specific needs with a modular approach seems an appropriate path to take. It allows for selective but prescriptive guidance on the implementation of tools and controls.
Implementing governance controls
In collaboration with a number of folks, I co-authored a guide for implementing governance in depth for serverless applications. The guide covers categories of controls as outlined by AWS Control Tower and then shows examples of how to implement those controls using AWS services. I understand that many might choose to implement those capabilities with open source alternatives, but this is just a starting point. Open source alternatives are something for future consideration.
Taking the content of the guide and expanding that into a talk, I co-presented this material at re:Invent 2023 and the recording for that talk can be found on YouTube. The talk walks through the journey that an organization might take in growing from zero or few controls to a set of controls that make sense for the organization. Those controls often result from incidents or on-going pains within the organization and a desire to automate the remediations to alleviate some day-to-day burden from operators.
Following up and looking ahead
I got some feedback from that re:Invent talk and am acting on some of the suggestions. As such, I have released of the code behind the examples that were shared in that re:Invent talk. That code can be found on AWS samples. I’ve implemented each control using SAM, CDK, and Terraform. To my earlier points, I understand many might already have their choice for infrastructure-as-code tools, so I aimed at showing the implementation in the most common tools that I see.
I’ve got a few backlog items that I hope to implement in that repository in the coming weeks: service control policies (preventative control) and integration with popular pipelines (specifically GitHub and GitLab). I’m open to hearing feedback for other feature requests.
Further out, I plan to implement some of the more common open source tools in this space. At the top of the list are: Open Policy Agent, Cloud Custodian, and some examples with tflint.
And even further out, I plan to implement a few opinionated end-to-end pipelines that leverage some of these tools. These could eventually become what I mentioned at the beginning of this post: a serverless landing zone. A landing zone would need to be far more comprehensive than what I’ve listed here, but one step at a time.. stay tuned!
