Query Logs with CloudWatch Logs Insights
Write a Logs Insights query that filters by error and groups by service so the top error source is visible in seconds.
Task
Configure aws logs start-query --log-group-name /aws/lambda/api --query-string 'fields @timestamp, @message | filter level == "error" | stats count() by service' so the cloud resource enforces the control automatically without manual console steps.
When to use this
An audit finds that aws logs start-query --log-group-name /aws/lambda/api --query-string 'fields @timestamp, @message | filter level == "error" | stats count() by service' was configured manually and drifted, leaving a resource exposed for two weeks.
Prerequisites
- Terraform or IaC tooling installed
- Cloud provider credentials with target permissions
Steps
1. Configure aws logs start-query --log-group-name /aws/lambda/api --query-string 'fields @timestamp, @message | filter level == "error" | stats count() by service' in the project configuration file with the recommended defaults.
aws logs start-query --log-group-name /aws/lambda/api --query-string 'fields @timestamp, @message | filter level == "error" | stats count() by service'
Expected output: The configuration is accepted and aws logs start-query --log-group-name /aws/lambda/api --query-string 'fields @timestamp, @message | filter level == "error" | stats count() by service' runs without errors on the first invocation.
2. Run the check locally to verify aws logs start-query --log-group-name /aws/lambda/api --query-string 'fields @timestamp, @message | filter level == "error" | stats count() by service' produces the expected result before pushing.
pnpm test
Expected output: Local run matches CI behavior; no surprises after push.
3. Add the gate to the CI pipeline so aws logs start-query --log-group-name /aws/lambda/api --query-string 'fields @timestamp, @message | filter level == "error" | stats count() by service' blocks merge on failure.
- name: Run add cloudwatch log insight query
run: pnpm test
Expected output: Pull requests with failing aws logs start-query --log-group-name /aws/lambda/api --query-string 'fields @timestamp, @message | filter level == "error" | stats count() by service' are blocked from merging.
Troubleshooting
Problem
The check passes locally but fails in CI
Solution
Pin dependency versions and the runtime to match CI exactly; reproduce with the same Docker image or act.
Problem
Configuration is rejected with a schema error
Solution
Check the tool version matches the config schema; older versions may not support newer fields.
Limitations
- The gate only runs on events that trigger the workflow; locally pushed commits bypass it.
- Configuration drift is possible if someone edits settings outside version control.
Codify the control
Start by measuring the problem. Write a Logs Insights query that filters by error and groups by service so the top error source is visible in seconds. The most common mistake is adding the gate without first confirming it catches the issue you care about. Run aws logs start-query –log-group-name /aws/lambda/api –query-string ‘fields @timestamp, @message | filter level == “error” | stats count() by service’ against the current codebase and inspect the output before wiring it into CI.
aws logs start-query --log-group-name /aws/lambda/api --query-string 'fields @timestamp, @message | filter level == "error" | stats count() by service'Detect drift
Once the gate is in place, make failures actionable. A failing aws logs start-query –log-group-name /aws/lambda/api –query-string ‘fields @timestamp, @message | filter level == “error” | stats count() by service’ should point to the exact file and line, not a generic error. Pin the tool version in the lockfile so a upstream release does not silently change behavior. Review the gate output in every pull request just like a test result.