Skip to content

fix(logger): count buffer max_bytes in UTF-8 bytes - #8833

Merged
leandrodamascena merged 2 commits into
aws-powertools:developfrom
bardrop-oss:fix/logger-buffer-max-bytes
Nov 20, 2026
Merged

leandrodamascena merged 2 commits into
aws-powertools:developfrom
bardrop-oss:fix/logger-buffer-max-bytes

Conversation

@bardrop

@bardrop bardrop commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Issue number: closes #8830

Summary

Changes

LoggerBufferCache and KeyBufferCache sized items with len(str(item)), which counts characters. They now use one small helper, _item_size(), that returns len(str(item).encode("utf-8")). It is used in all three places (the "bigger than the whole buffer" check, add, and remove_oldest), so the running size stays consistent when items are evicted.

No docs change: docs/core/logger.md already says max_bytes is the "maximum size of the log buffer in bytes".

User experience

Before: with max_bytes=1600, logger.debug("é" * 1000) was buffered with no warning, though the record is about 2.2 KB in UTF-8.

After: the record is counted in bytes, so it is treated like any record bigger than the buffer: PowertoolsUserWarning("Cannot add item to the buffer...") and the log is written right away. ASCII-only logs behave the same as before.

Verification

I used an AI assistant (Claude) to help write this change. I ran the tests below myself.

New tests:

  • tests/unit/logger/required_dependencies/test_logger_buffer_cache.py: test_cache_size_counts_utf8_bytes, test_item_larger_than_cache_in_bytes
  • tests/functional/logger/required_dependencies/test_powertools_logger_buffer.py: test_log_record_exceeding_buffer_size_in_bytes (public Logger API only)

On develop (fix reverted, tests kept):

FAILED tests/unit/logger/required_dependencies/test_logger_buffer_cache.py::test_cache_size_counts_utf8_bytes
FAILED tests/unit/logger/required_dependencies/test_logger_buffer_cache.py::test_item_larger_than_cache_in_bytes
FAILED tests/functional/logger/required_dependencies/test_powertools_logger_buffer.py::test_log_record_exceeding_buffer_size_in_bytes
3 failed, 204 passed

With the fix:

$ uv run pytest -q tests/unit/logger tests/functional/logger
207 passed, 7 warnings

Full uv run pytest -q tests/unit tests/functional on this branch: 4056 passed, 73 skipped, 3 failed. The same 3 fail on develop without my change, and they come from test order, not this PR: once tests/functional/metrics/datadog is collected, ddtrace is imported and adds dd.* keys to log records, so test_append_context_keys_handles_empty_dict and test_log_dict_key_strip_nones see extra keys (both pass when tests/functional/logger runs alone). The third, test_datadog_write_to_log_with_env_variable, passes when run alone. On my machine some full runs also hung in the Redis idempotency tests on develop too, so I ran the logger tests separately as well.

ruff format --check and ruff check on the changed files: clean. mypy aws_lambda_powertools/logging/: "Success: no issues found in 17 source files".


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@bardrop
bardrop requested a review from a team as a code owner November 15, 2026 16:00
@boring-cyborg

boring-cyborg Bot commented Nov 15, 2026

Copy link
Copy Markdown

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Nov 15, 2026
@leandrodamascena leandrodamascena added this to the 3.42.0 milestone Nov 15, 2026

@leandrodamascena leandrodamascena left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @bardrop, second one in. Measuring in bytes in the three places that count keeps the running size honest when items get evicted, good call doing it through one helper. Goes out in 3.42.0, no need to rebase, you'll be in the release notes.

@sonarqubecloud

Copy link
Copy Markdown

@codecov

codecov Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.20%. Comparing base (9634e6f) to head (7637a60).

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #8833   +/-   ##
========================================
  Coverage    97.20%   97.20%           
========================================
  Files          327      327           
  Lines        16757    16759    +2     
  Branches      1522     1522           
========================================
+ Hits         16288    16290    +2     
  Misses         328      328           
  Partials       141      141           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@leandrodamascena
leandrodamascena merged commit d90afd5 into aws-powertools:develop Nov 20, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logger size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: LoggerBufferConfig.max_bytes counts characters, not bytes

2 participants