Monitoring and alerting on AWS service quotas
I recently had a customer engagement where they were trying to see how close Lambda function concurrent executions were coming to the account concurrency quota. They wanted an easy way to visualize this and send notifications when they were approaching that quota. I realized I didn’t know an easy way to do this. I reached out to my specialist teammates, and my colleague, Anton, was able to point me to a simple approach, which he also documented in a LinkedIn post.
Visualization in CloudWatch
By default, an AWS account allows for 1000 concurrency in a given region but that is a soft quota that can be raised. Operations folks may setup a widget in a CloudWatch dashboard that shows the ConcurrentExecutions metric, like below.
However, that doesn’t tell the operator how close the account is to the account quota. One could assume 1000, if he/she knows that’s the default limit, but what if the limit was raised at some point? You can add the service quota into the widget!
In the CloudWatch dashboard, edit the widget, select Add math, All functions, and SERVICE_QUOTA.
And by doing that, I can very easily visualize that I do indeed have 1000 as my account concurrency quota and that I am far from breaching that quota.
I observed one caveat. The approach that I outline above works when you do this across all functions for Lambda first.
When I tried to do it by function name first, I get an error in expression when it references a metric. I imagine this is because the function-level metric has an additional dimension versus account-level metrics.
However, if you do it at the account level first, and then add individual functions later, it works fine. Viola!
Alerting in Service Quotas
In the Service Quotas dashboard, you can also see the quota value and see the utilization of that quota.
You can click on the Concurrent executions quota to view more details.
Then from there, you can click on Create alarm to set a threshold alarm.
While the examples given here are for AWS Lambda, the same can apply for any other AWS service.
