Skip to content

fix(parser): accept CloudWatch Logs CONTROL_MESSAGE and keep extractedFields (#8869) - #8870

Open
dhaiirryyaa wants to merge 2 commits into
aws-powertools:developfrom
dhaiirryyaa:fix/cloudwatch-parser-control-message
Open

dhaiirryyaa wants to merge 2 commits into
aws-powertools:developfrom
dhaiirryyaa:fix/cloudwatch-parser-control-message

Conversation

@dhaiirryyaa

Copy link
Copy Markdown
Contributor

Issue number: closes #8869

Summary

Changes

  • In aws_lambda_powertools/utilities/parser/models/cloudwatch.py:
    • In CloudWatchLogsDecode, made owner, logGroup, and logStream optional (str | None = None) and defaulted subscriptionFilters and logEvents to default_factory=list. When CloudWatch Logs emits a CONTROL_MESSAGE (e.g. to verify destination reachability), it sends only {"messageType": "CONTROL_MESSAGE"}.
    • In CloudWatchLogsLogEvent, added extracted_fields: dict[str, str] = Field(default_factory=dict, alias="extractedFields") and model_config = ConfigDict(populate_by_name=True) so fields extracted by CloudWatch subscription filter patterns are preserved.
  • In aws_lambda_powertools/utilities/parser/envelopes/cloudwatch.py:
    • In CloudWatchLogsEnvelope.parse(), return an empty list [] when parsed_envelope.awslogs.decoded_data.messageType == "CONTROL_MESSAGE".
    • Fixed copy-paste debug log message from "SNS model" to "CloudWatchLogs model".
  • Added unit tests in tests/unit/parser/_pydantic/test_cloudwatch.py covering:
    • CloudWatchLogsModel parsing of a CONTROL_MESSAGE event.
    • CloudWatchLogsEnvelope returning [] on CONTROL_MESSAGE.
    • CloudWatchLogsLogEvent keeping extractedFields.

User experience

  • Lambda functions triggered by CloudWatch Logs subscription filters will no longer fail validation when CloudWatch sends a CONTROL_MESSAGE heartbeat.
  • When CloudWatchLogsEnvelope is used with a schema model, CONTROL_MESSAGE events gracefully return [] without error.
  • Parsed log events retain extracted_fields when subscription filter pattern extraction is used.

Verification

Ran pytest tests/unit/parser/_pydantic/test_cloudwatch.py (all 12 tests passed) and full parser test suite (all 203 tests passed). Verified formatting and linting with ruff check and ruff format --check.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@dhaiirryyaa
dhaiirryyaa requested a review from a team as a code owner November 18, 2026 15:44
@boring-cyborg boring-cyborg Bot added the tests label Nov 18, 2026
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Nov 18, 2026
@dhaiirryyaa

Copy link
Copy Markdown
Contributor Author

Hi @leandrodamascena, this PR is ready for review. It allows the parser to accept CloudWatch Logs CONTROL_MESSAGE records while preserving extractedFields, resolving #8869. All unit tests and SonarQube quality gates are passing. Thanks!

@Manishram-ai Manishram-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I compared this with what CloudWatch Logs sends as a control message. It isn't only {"messageType": "CONTROL_MESSAGE"}. It comes with owner: "CloudwatchLogs", empty logGroup and logStream, an empty subscriptionFilters list and one log event whose message is the health check text.

On develop, CloudWatchLogsModel already parses that fine. What breaks is the envelope, because it passes "CWL CONTROL MESSAGE: Checking health of destination ..." to the user's model. So your early return in the envelope is the actual fix, and the model fields can stay as they are. I ran tests/unit/parser and tests/functional/parser with the three suggestions below applied: 275 passed, and ruff is clean.



def _make_cloudwatch_control_event() -> dict:
inner_event_dict = {"messageType": "CONTROL_MESSAGE"}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A real control message has all the usual keys, just with empty values, and one log event with the health check text. This is the shape Kinesis and Firehose consumers see:

Suggested change
inner_event_dict = {"messageType": "CONTROL_MESSAGE"}
inner_event_dict = {
"messageType": "CONTROL_MESSAGE",
"owner": "CloudwatchLogs",
"logGroup": "",
"logStream": "",
"subscriptionFilters": [],
"logEvents": [
{
"id": "",
"timestamp": 1767323045678,
"message": "CWL CONTROL MESSAGE: Checking health of destination Lambda.",
},
],
}

With this fixture test_handle_cloudwatch_control_message_with_envelope still fails on develop, since the envelope hands the health check text to MyCloudWatchBusiness, and passes with your envelope change.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied! Updated the fixture to match real CloudWatch Logs control message payload structure with empty strings and single health check log event.

Comment on lines +202 to +206
assert parsed_event.awslogs.decoded_data.owner is None
assert parsed_event.awslogs.decoded_data.logGroup is None
assert parsed_event.awslogs.decoded_data.logStream is None
assert parsed_event.awslogs.decoded_data.subscriptionFilters == []
assert parsed_event.awslogs.decoded_data.logEvents == []

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And the matching assertions for that fixture:

Suggested change
assert parsed_event.awslogs.decoded_data.owner is None
assert parsed_event.awslogs.decoded_data.logGroup is None
assert parsed_event.awslogs.decoded_data.logStream is None
assert parsed_event.awslogs.decoded_data.subscriptionFilters == []
assert parsed_event.awslogs.decoded_data.logEvents == []
assert parsed_event.awslogs.decoded_data.owner == "CloudwatchLogs"
assert parsed_event.awslogs.decoded_data.logGroup == ""
assert parsed_event.awslogs.decoded_data.logStream == ""
assert parsed_event.awslogs.decoded_data.subscriptionFilters == []
assert len(parsed_event.awslogs.decoded_data.logEvents) == 1

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied! Updated matching assertions for the fixture.

Comment on lines 40 to 64
owner: str | None = Field(
default=None,
description="The AWS account ID of the originating log data.",
examples=["123456789012"],
)
logGroup: str | None = Field(
default=None,
description="The name of the log group that contains the log stream.",
examples=["/aws/lambda/my-function", "/aws/apigateway/my-api"],
)
logStream: str = Field(
logStream: str | None = Field(
default=None,
description="The name of the log stream that stores the log events.",
examples=["2023/01/15/[$LATEST]abcdef1234567890", "i-1234567890abcdef0"],
)
subscriptionFilters: list[str] = Field(
default_factory=list,
description="List of subscription filter names associated with the log group.",
examples=[["LambdaStream_cloudwatch", "AlertFilter"]],
)
logEvents: list[CloudWatchLogsLogEvent] = Field(
default_factory=list,
description="Array of log events included in the message.",
examples=[[{"id": "eventId1", "timestamp": 1673779200000, "message": "Sample log line"}]],
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since develop already parses the real control message, these can stay required. Making them str | None changes the type for everyone reading logGroup or owner on data messages, and type checkers will start asking for a None check there.

Suggested change
owner: str | None = Field(
default=None,
description="The AWS account ID of the originating log data.",
examples=["123456789012"],
)
logGroup: str | None = Field(
default=None,
description="The name of the log group that contains the log stream.",
examples=["/aws/lambda/my-function", "/aws/apigateway/my-api"],
)
logStream: str = Field(
logStream: str | None = Field(
default=None,
description="The name of the log stream that stores the log events.",
examples=["2023/01/15/[$LATEST]abcdef1234567890", "i-1234567890abcdef0"],
)
subscriptionFilters: list[str] = Field(
default_factory=list,
description="List of subscription filter names associated with the log group.",
examples=[["LambdaStream_cloudwatch", "AlertFilter"]],
)
logEvents: list[CloudWatchLogsLogEvent] = Field(
default_factory=list,
description="Array of log events included in the message.",
examples=[[{"id": "eventId1", "timestamp": 1673779200000, "message": "Sample log line"}]],
)
owner: str = Field(description="The AWS account ID of the originating log data.", examples=["123456789012"])
logGroup: str = Field(
description="The name of the log group that contains the log stream.",
examples=["/aws/lambda/my-function", "/aws/apigateway/my-api"],
)
logStream: str = Field(
description="The name of the log stream that stores the log events.",
examples=["2023/01/15/[$LATEST]abcdef1234567890", "i-1234567890abcdef0"],
)
subscriptionFilters: list[str] = Field(
description="List of subscription filter names associated with the log group.",
examples=[["LambdaStream_cloudwatch", "AlertFilter"]],
)
logEvents: list[CloudWatchLogsLogEvent] = Field(
description="Array of log events included in the message.",
examples=[[{"id": "eventId1", "timestamp": 1673779200000, "message": "Sample log line"}]],
)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied! Restored owner, logGroup, logStream, subscriptionFilters, and logEvents as required non-optional fields to avoid type regressions for consumers.

@dhaiirryyaa

Copy link
Copy Markdown
Contributor Author

Thank you for the clear context and suggestions @Manishram-ai!

All three suggestions have been implemented in commit 604e841:

  1. Model Fields Unchanged: Restored owner, logGroup, logStream, subscriptionFilters, and logEvents as required types in CloudWatchLogsDecode, preserving strict type checking for consumers.
  2. Realistic Test Fixture: Updated _make_cloudwatch_control_event() to match real CloudWatch control messages (empty strings and single health-check log event).
  3. Updated Assertions: Verified all decoded fields against the real payload shape.

All 275 parser tests in tests/unit/parser and tests/functional/parser pass cleanly, and ruff is green.

@sonarqubecloud

Copy link
Copy Markdown

@codecov

codecov Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.20%. Comparing base (19c0798) to head (604e841).
⚠️ Report is 4 commits behind head on develop.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #8870   +/-   ##
========================================
  Coverage    97.20%   97.20%           
========================================
  Files          327      327           
  Lines        16759    16764    +5     
  Branches      1523     1524    +1     
========================================
+ Hits         16290    16295    +5     
  Misses         328      328           
  Partials       141      141           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: CloudWatchLogsModel and envelope reject CONTROL_MESSAGE events and drop extractedFields

2 participants