Skip to content

Faster charts and replies, spike-free air figures, tidier logs - #44

Merged
robdevops merged 22 commits into
mainfrom
claude/envirobot-next
Oct 17, 2026
Merged

Faster charts and replies, spike-free air figures, tidier logs#44
robdevops merged 22 commits into
mainfrom
claude/envirobot-next

Conversation

@robdevops

@robdevops robdevops commented Oct 17, 2026

Copy link
Copy Markdown
Owner

Speed

  • Charts are drawn with far fewer matplotlib objects and written as 256-colour PNGs (38% of the size); the palette step is quicker (same PNG bytes).
  • Questions no longer wait for the minute's refresh when the cache and memory already hold what they need, and their Ecowitt requests go first (a chart asked mid-refresh: ~2.0 s to ~0.2 s).
  • Long periods: one coverage read per check, the wind rose's days read in runs, months labelled once a day.
  • A chart is uploaded once: alerts send Telegram's copy to later chats, and period buttons edit by file id.
  • The button charts are drawn ahead every 5 minutes, so a press takes a few ms.
  • New TELEGRAM_PRELOAD_CHAT: upload a chart's other periods ahead (used for 30 minutes).
  • Each chart reply logs where its time went (Chart ms: ...).

Air quality

  • New ANTISPIKE_REQUIRE_CONSECUTIVE (default 2, 1/off disables): single-reading particle spikes are left out of PM charts, the air figures and day statistics, and the unhealthy-air alert needs the last N readings all over the mask level.

Logs

  • New LOG_WIDTH (default 71, 0 cuts none): every INFO line is cut with an ellipsis, a WARN or ERROR never is; WARNING prints as WARN; no module name; several startup lines shortened.
  • Fix: a queued Ecowitt request cancelled just as the one ahead finishes no longer raises ValueError.

Checks

python -m pytest -q (794 passed) and python scripts/eval_prompts.py (87 of 87).

Generated by Claude Code

claude added 22 commits October 17, 2026 16:15
@claude
Draw charts with fewer matplotlib objects, same pictures (the 7 sampl…
66c4c11 
…es in 74% of the time)

lib/charts.py: a stacked panel's tint is a rectangle on the figure, not an axes; rain behind
a line is drawn in its own panel (x on the time axis, y as a share of the height, its scale's
labels where a right-hand axis puts them) instead of a twin axes and a grid axes; rain, count
and rating bars are one collection each, not a patch per bar; the wind rose's 48 wedges are one
collection with the radius matplotlib gave the bars (_rose_radius, tested against it); dots are
one-point markers, not scatters; upper panels hide their unused time axis and titles are placed,
not searched for. Pixel for pixel the same, the rose's anti-aliasing aside (at most 3/255).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
Charts as 256-colour PNGs: 38% of the size, written faster, looking t…
a025049 
…he same

lib/charts.py _png: a chart has a few thousand colours, nearly all of them the in-between
shades on the edges of lines and letters. _palette keeps 256: from the commonest, each next
one the colour the palette so far draws worst (its pixels times its squared distance to the
nearest kept colour), so every colour that covers an area stays exact and no pixel of the
seven samples moves by more than 25/255 (mean under 0.06). Pillow's own quantisers were
slower or shifted the white and merged the panels' faint tint into it. The 7 samples go from
1004 to 380 KB, and writing them takes less time than savefig did.

tests/charts: regenerated (scripts/update_charts.py). The comparison's own cut to 64 colours
picks a little differently from the new pictures (up to 0.77% of pixels), though at full size
they match the old ones to within a few levels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
Log how long each chart reply took, step by step
bc6b928 
After a chart is sent, one line says where the time went:
"Chart times: data 12 ms, calculation 6 ms, drawing 112 ms, PNG 41 ms (22 KB), sending 812 ms".

- data: the time the history tools spent reading (cache, memory, Ecowitt; waits included), kept on the question's Turn.
- calculation: the tool's time less its data, shown when the chart is made in code.
- drawing and PNG: timed by charts.render, which now takes the Turn's times.
- sending: delivering the reply (or redrawing it in place).

The period button's cached chart and the alert sends also log how long sending took.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
Pick the chart palette in whole numbers, a channel at a time: same PN…
ad0dcec 
…Gs, PNG step in 64% of the time

The greedy palette now keeps each colour's red, green and blue as separate
integer arrays, and its score (pixels x squared distance) up to date only
where a colour got closer, instead of recomputing both over every colour for
each of the 256 picks. The values are the same whole numbers as before, so
the picks and the PNG bytes are identical (checked on all 15 layouts).

The PNG step for the 7 samples: 268 -> 171 ms (palette alone 2.7-5x faster).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
Questions never wait for the refresh, and their Ecowitt requests go f…
5929965 
…irst

- Reading without waiting: a history read first checks the cache and memory without the refresh's lock, and returns at once
  when they hold it all (the usual case: the refresh keeps memory under a minute old). Only a read that needs Ecowitt waits
  for a fetch in flight, as before. Counters count only the pass that answered.
- Urgent requests: a question's history requests (and its wind rose's) are marked urgent, so they go before the background
  refresh's queued requests.
- No refresh for charts made in code: such a chart no longer starts a refresh beside its own reads. Questions for the model
  still do.
- Less work for the refresh: it no longer merges the readings it fetches into a result it never reads.

Measured through the bot's chart path with Ecowitt answering in 0.6 s:
- A chart asked while the minute's refresh is fetching: Temperature 1d 2.0 s -> 0.16 s, Weather 1d 2.0 -> 0.34 s,
  Wind 1d 1.9 -> 0.17 s (it waited ~1.8 s for the refresh).
- With memory stale: Weather 1d and Wind 1d 5.0 s -> 1.0 s (one Ecowitt request instead of five).
- The history tools' results and chart specs for fixed past periods are unchanged (21 cases).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
Long periods: one coverage read per check, the wind rose's days read …
898a058 
…in runs, months labelled once a day

- One read of the coverage: a history read, a "held in the cache?" check and the wind rose's day-by-day check each read
  every group's coverage in one query and one thread hop (HistoryCache.coverage_of), instead of one hop per group and window.
  A hop costs more than the query, and far more while a chart is being drawn in another thread.
- The wind rose's 5-minute readings: consecutive days are read in one go. A 30-day wind chart reads them once instead of
  31 times; a 30-day weather chart makes 178 cache queries instead of 387.
- Monthly figures: each reading's month comes from its local day, worked out once a day, not a date conversion and strftime
  per reading.

The readings themselves are still read per group: every combined form was 10-33% slower.

History tool times (best of 3, before -> after):
- Wind 365d 433-458 -> 279-356 ms
- Weather 30d 289-312 -> 237-246 ms
- Wind 30d 177-199 -> 142-167 ms
- Temperature 90d 201-245 -> 176-183 ms
- Weather 7d 73-82 -> 58-68 ms
- Temperature 365d and Rain 365d: unchanged

The results and chart specs for 21 fixed past periods are identical. So are the wind questions through now, checked against
the previous code in the same process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
Send each chart picture once: Telegram's copy (its file id) for every…
eb2c922 
… later send

- Alerts: a chart alert is uploaded to the first chat only; the others are sent the file id Telegram gave it, so one upload
  serves every chat. The text-alone fallback is unchanged.
- Period buttons: the image cache keeps each chart's file id once it has been sent (the reply, or a redraw in place).
  Toggling back to a period drawn lately edits the message by that id, no upload. If Telegram refuses, it is drawn again,
  as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
Draw the button charts ahead every 5 minutes: a press is sent one at …
b8bf457 
…once

- Bot.predraw draws every chart the buttons ask for (made in code) and keeps it in the image cache with its write-up.
  It runs every 5 minutes after the startup warm-up, one chart at a time, and never while a question is being answered,
  so it can't slow one down.
- Answering from it: a chart question whose words are exactly its key ("Temperature chart 1d", "Wind chart 7d") is sent the
  chart drawn for those words in the last 5.5 minutes (ahead, or for someone's same question), by Telegram's copy once it
  has one. Its write-up goes into the chat's memory as a drawn one's does. Other words for the same period ("temperature
  chart today") are drawn as before.
- Background turns (Turn.ahead): their Ecowitt requests aren't urgent, and their routine log lines (tool calls, the
  history summaries) are at DEBUG. The first round logs one line; a chart that can't be drawn ahead is logged once, until
  it can again.
- Alerts still draw their own chart: a picture up to 5 minutes old could miss the reading the alert is about.

A button press drawn ahead takes 4-8 ms of the bot's time instead of 120-340 ms; the log says
"Chart times: drawn 40s before, sending 6 ms (Telegram's copy)".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
Ecowitt's request queue: a request cancelled just as the one ahead fi…
68fbfc4 
…nishes is a clean cancel

If a queued request was cancelled (its question timed out, or shutdown) and the request ahead finished before it had seen
the cancel, the release had already dropped it from the queue, and removing it again raised "ValueError: <Future
cancelled> is not in deque" instead of the cancel. Now it is only removed when it is still there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
LOG_WIDTH: long log lines quoting a request or a chat message are cut…
5d96d69 
… to 71 characters

A log line that quotes something (what was asked, the tool request and its short result, the reply, an alert, the
heads-up) is cut to LOG_WIDTH characters, default 71, an ellipsis (…) marking the cut; LOG_WIDTH=0 cuts none.
- Only the quote is cut. What comes before it on the line is kept whole, and so is every other line (timings, counts,
  warnings, tracebacks).
- The quote is put on one line.
- The fixed cuts these lines had (200, 300, 40, 30 characters) are gone, so 0 really shows everything.
- The period button's line now names the press before the question ("Period button 3M: Temperature chart 30d"), so
  the cut can't take the new period.

The log formatter moves to lib/logs.py with the cut; ecobot.py gives it the width once the settings are read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
TELEGRAM_PRELOAD_CHAT: upload a chart's other periods ahead, so a per…
f767184 
…iod button uploads nothing

With TELEGRAM_PRELOAD_CHAT set to a chat id, each chart sent (or switched) with period buttons, when made in code, has its
other periods drawn in the background and sent silently to that chat. The file ids go in the image cache, so pressing a
period button edits the message with Telegram's copy.
- One chart at a time, never while a question is being answered; not urgent Ecowitt requests; quiet logs.
- Used for 30 minutes (PRELOAD_TTL), then drawn afresh.
- If Telegram refuses an upload, one warning, and no more until restart. Unset: nothing changes.
- The drawing step of predraw is shared (Bot._draw_ahead).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
LOG_WIDTH applies to every log line, and the startup Alerts list wrap…
a294337 
…s at it

Every line of the log is cut to LOG_WIDTH (default 71, 0 cuts none), an ellipsis marking the cut, including each line of a
multi-line entry such as a traceback. A request, chat message, reply or alert is still put on one line first. The startup
"Alerts:" list wraps at the width (90 when LOG_WIDTH=0).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
Log lines no longer start with the module name
5f0c763 
"INFO Replied 1 in 0.1s ..." instead of "INFO lib.bot: Replied 1 in 0.1s ...".

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
Fix the log cut test's expected text
686c2f9 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
Shorten the log messages that LOG_WIDTH was cutting
b8ad38f 
Startup: Pollen (grass/asthma in a few words), ADMIN_ONLY, "@bot + model" and a separate Sources line, Keeping warm (wraps
like Alerts, both at the log width), AirGradient and Ecowitt archive lines, the live reading summary.
Per chart: "Chart ms: data 12, calc 6, draw 112, PNG 41 (22 KB), send 812", the history summary
("09-19..10-18: 13 reads (12 cache, 1 mem), 0 req, 8 refined"), the period button and drawn-ahead lines.
Two warnings are shorter. Lines that quote a request or reply are still cut, as they should be.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
Log WARNING as WARN and never cut a WARN or an ERROR to LOG_WIDTH
a9ffaea 
The ADMIN_ONLY startup line says "see: ADMIN_CHAT_IDS".

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
Ecowitt archive log lines start "Ecowitt:" without the word "archive"
031e551 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
Log the HTTP client's lines without the "HTTP Request:" prefix
e4a81f6 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
Leave single-reading spikes of the particles out of PM charts
def4796 
A PM1, PM2.5 or PM10 reading that is at least 2.5 times, and 5 µg/m³ above, both readings next to it (within 15 minutes) is
a glitch of the sensor: that moment's particle readings are left out of the drawing and of its peak labels. A rise lasting two
readings or more is the air and stays. The first and last readings are kept. Only charts change; the figures the tool gives
are as they were.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
Single-reading particle spikes are in none of the air figures either
57154fd 
The air history's highs, lows, averages and ratings, and the day statistics (days over a limit, a day's highest), now leave
out the single-reading spikes that charts already leave out, so the numbers and the pictures agree.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
The unhealthy-air alert counts a check only when the sensor's last tw…
fd5da17 
…o readings are both that high

The alert already needed two checks in a row, but each check looked at the latest reading alone, so a spike at a check counted
as one of the two. air.current() now also gives each metric's lower value of the last two readings (_both, for the alerts),
and a check is "bad" only when that is at or over the mask level. The "safe again" check is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@claude
ANTISPIKE_REQUIRE_CONSECUTIVE: the readings in a row a rise of the pa…
a5fe6e0 
…rticles needs (default 2, 1 or off: no rule)

One setting for the three behaviours: PM charts, the air figures (highs, lows, averages, days over a limit) and the
unhealthy-air alert. A stretch of fewer than N readings in a row, each at least 2.5 times and 5 µg/m3 above the readings
either side, is a glitch and is left out; with N = 3 two-reading spikes go too. The alert counts a check only when the
sensor's last N readings are all over the mask level (air.current() gives that as _sustained, hidden from the model).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBwHofjb1g9onaYRdLZMbf
@robdevops
robdevops enabled auto-merge October 17, 2026 19:35
@robdevops
robdevops merged commit a788e08 into main Oct 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@robdevops @claude