Repository navigation
fix(parser): accept CloudWatch Logs CONTROL_MESSAGE and keep extractedFields (#8869) - #8870
dhaiirryyaa wants to merge 2 commits into
Conversation
|
Hi @leandrodamascena, this PR is ready for review. It allows the parser to accept CloudWatch Logs |
Manishram-ai
left a comment
There was a problem hiding this comment.
I compared this with what CloudWatch Logs sends as a control message. It isn't only {"messageType": "CONTROL_MESSAGE"}. It comes with owner: "CloudwatchLogs", empty logGroup and logStream, an empty subscriptionFilters list and one log event whose message is the health check text.
On develop, CloudWatchLogsModel already parses that fine. What breaks is the envelope, because it passes "CWL CONTROL MESSAGE: Checking health of destination ..." to the user's model. So your early return in the envelope is the actual fix, and the model fields can stay as they are. I ran tests/unit/parser and tests/functional/parser with the three suggestions below applied: 275 passed, and ruff is clean.
|
|
||
|
|
||
| def _make_cloudwatch_control_event() -> dict: | ||
| inner_event_dict = {"messageType": "CONTROL_MESSAGE"} |
There was a problem hiding this comment.
A real control message has all the usual keys, just with empty values, and one log event with the health check text. This is the shape Kinesis and Firehose consumers see:
| inner_event_dict = {"messageType": "CONTROL_MESSAGE"} | |
| inner_event_dict = { | |
| "messageType": "CONTROL_MESSAGE", | |
| "owner": "CloudwatchLogs", | |
| "logGroup": "", | |
| "logStream": "", | |
| "subscriptionFilters": [], | |
| "logEvents": [ | |
| { | |
| "id": "", | |
| "timestamp": 1767323045678, | |
| "message": "CWL CONTROL MESSAGE: Checking health of destination Lambda.", | |
| }, | |
| ], | |
| } |
With this fixture test_handle_cloudwatch_control_message_with_envelope still fails on develop, since the envelope hands the health check text to MyCloudWatchBusiness, and passes with your envelope change.
There was a problem hiding this comment.
Applied! Updated the fixture to match real CloudWatch Logs control message payload structure with empty strings and single health check log event.
| assert parsed_event.awslogs.decoded_data.owner is None | ||
| assert parsed_event.awslogs.decoded_data.logGroup is None | ||
| assert parsed_event.awslogs.decoded_data.logStream is None | ||
| assert parsed_event.awslogs.decoded_data.subscriptionFilters == [] | ||
| assert parsed_event.awslogs.decoded_data.logEvents == [] |
There was a problem hiding this comment.
And the matching assertions for that fixture:
| assert parsed_event.awslogs.decoded_data.owner is None | |
| assert parsed_event.awslogs.decoded_data.logGroup is None | |
| assert parsed_event.awslogs.decoded_data.logStream is None | |
| assert parsed_event.awslogs.decoded_data.subscriptionFilters == [] | |
| assert parsed_event.awslogs.decoded_data.logEvents == [] | |
| assert parsed_event.awslogs.decoded_data.owner == "CloudwatchLogs" | |
| assert parsed_event.awslogs.decoded_data.logGroup == "" | |
| assert parsed_event.awslogs.decoded_data.logStream == "" | |
| assert parsed_event.awslogs.decoded_data.subscriptionFilters == [] | |
| assert len(parsed_event.awslogs.decoded_data.logEvents) == 1 |
There was a problem hiding this comment.
Applied! Updated matching assertions for the fixture.
| owner: str | None = Field( | ||
| default=None, | ||
| description="The AWS account ID of the originating log data.", | ||
| examples=["123456789012"], | ||
| ) | ||
| logGroup: str | None = Field( | ||
| default=None, | ||
| description="The name of the log group that contains the log stream.", | ||
| examples=["/aws/lambda/my-function", "/aws/apigateway/my-api"], | ||
| ) | ||
| logStream: str = Field( | ||
| logStream: str | None = Field( | ||
| default=None, | ||
| description="The name of the log stream that stores the log events.", | ||
| examples=["2023/01/15/[$LATEST]abcdef1234567890", "i-1234567890abcdef0"], | ||
| ) | ||
| subscriptionFilters: list[str] = Field( | ||
| default_factory=list, | ||
| description="List of subscription filter names associated with the log group.", | ||
| examples=[["LambdaStream_cloudwatch", "AlertFilter"]], | ||
| ) | ||
| logEvents: list[CloudWatchLogsLogEvent] = Field( | ||
| default_factory=list, | ||
| description="Array of log events included in the message.", | ||
| examples=[[{"id": "eventId1", "timestamp": 1673779200000, "message": "Sample log line"}]], | ||
| ) |
There was a problem hiding this comment.
Since develop already parses the real control message, these can stay required. Making them str | None changes the type for everyone reading logGroup or owner on data messages, and type checkers will start asking for a None check there.
| owner: str | None = Field( | |
| default=None, | |
| description="The AWS account ID of the originating log data.", | |
| examples=["123456789012"], | |
| ) | |
| logGroup: str | None = Field( | |
| default=None, | |
| description="The name of the log group that contains the log stream.", | |
| examples=["/aws/lambda/my-function", "/aws/apigateway/my-api"], | |
| ) | |
| logStream: str = Field( | |
| logStream: str | None = Field( | |
| default=None, | |
| description="The name of the log stream that stores the log events.", | |
| examples=["2023/01/15/[$LATEST]abcdef1234567890", "i-1234567890abcdef0"], | |
| ) | |
| subscriptionFilters: list[str] = Field( | |
| default_factory=list, | |
| description="List of subscription filter names associated with the log group.", | |
| examples=[["LambdaStream_cloudwatch", "AlertFilter"]], | |
| ) | |
| logEvents: list[CloudWatchLogsLogEvent] = Field( | |
| default_factory=list, | |
| description="Array of log events included in the message.", | |
| examples=[[{"id": "eventId1", "timestamp": 1673779200000, "message": "Sample log line"}]], | |
| ) | |
| owner: str = Field(description="The AWS account ID of the originating log data.", examples=["123456789012"]) | |
| logGroup: str = Field( | |
| description="The name of the log group that contains the log stream.", | |
| examples=["/aws/lambda/my-function", "/aws/apigateway/my-api"], | |
| ) | |
| logStream: str = Field( | |
| description="The name of the log stream that stores the log events.", | |
| examples=["2023/01/15/[$LATEST]abcdef1234567890", "i-1234567890abcdef0"], | |
| ) | |
| subscriptionFilters: list[str] = Field( | |
| description="List of subscription filter names associated with the log group.", | |
| examples=[["LambdaStream_cloudwatch", "AlertFilter"]], | |
| ) | |
| logEvents: list[CloudWatchLogsLogEvent] = Field( | |
| description="Array of log events included in the message.", | |
| examples=[[{"id": "eventId1", "timestamp": 1673779200000, "message": "Sample log line"}]], | |
| ) |
There was a problem hiding this comment.
Applied! Restored owner, logGroup, logStream, subscriptionFilters, and logEvents as required non-optional fields to avoid type regressions for consumers.
|
Thank you for the clear context and suggestions @Manishram-ai! All three suggestions have been implemented in commit
All 275 parser tests in |
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #8870 +/- ##
========================================
Coverage 97.20% 97.20%
========================================
Files 327 327
Lines 16759 16764 +5
Branches 1523 1524 +1
========================================
+ Hits 16290 16295 +5
Misses 328 328
Partials 141 141 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|



Issue number: closes #8869
Summary
Changes
aws_lambda_powertools/utilities/parser/models/cloudwatch.py:CloudWatchLogsDecode, madeowner,logGroup, andlogStreamoptional (str | None = None) and defaultedsubscriptionFiltersandlogEventstodefault_factory=list. When CloudWatch Logs emits aCONTROL_MESSAGE(e.g. to verify destination reachability), it sends only{"messageType": "CONTROL_MESSAGE"}.CloudWatchLogsLogEvent, addedextracted_fields: dict[str, str] = Field(default_factory=dict, alias="extractedFields")andmodel_config = ConfigDict(populate_by_name=True)so fields extracted by CloudWatch subscription filter patterns are preserved.aws_lambda_powertools/utilities/parser/envelopes/cloudwatch.py:CloudWatchLogsEnvelope.parse(), return an empty list[]whenparsed_envelope.awslogs.decoded_data.messageType == "CONTROL_MESSAGE".tests/unit/parser/_pydantic/test_cloudwatch.pycovering:CloudWatchLogsModelparsing of aCONTROL_MESSAGEevent.CloudWatchLogsEnvelopereturning[]onCONTROL_MESSAGE.CloudWatchLogsLogEventkeepingextractedFields.User experience
CONTROL_MESSAGEheartbeat.CloudWatchLogsEnvelopeis used with a schema model,CONTROL_MESSAGEevents gracefully return[]without error.extracted_fieldswhen subscription filter pattern extraction is used.Verification
Ran
pytest tests/unit/parser/_pydantic/test_cloudwatch.py(all 12 tests passed) and full parser test suite (all 203 tests passed). Verified formatting and linting withruff checkandruff format --check.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.