diff --git a/README.md b/README.md index 38bfe81..efa3177 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ A Telegram bot for a personal Ecowitt weather station and AirGradient air-quality sensor. Ask it about the weather or air quality (text or charts); it also sends silent alerts to every chat it is in: rain starting/stopping, gusts over 40 km/h (with the last day's wind chart), UV index of 10 or more in two readings in a row (with the last day's UV chart), indoor/outdoor temperatures crossing by more than 1°C (held back for the cooldown, then announced if it still stands, with the last day's temperature chart), -and unhealthy air (two readings in a row; with when it is safe again; both with the last day's particulates chart). Every alert carries one 🔔 button that opens up to two toggles for that alert type: Alert (on or off in this chat) and, while that is on and in a private chat, Sound (whether it makes a notification sound; alerts are silent unless you turn it on). Every toggle in the bot is a green button while on and a plain one with its name struck through while off. `/alerts` has two buttons, 🔔 Alerts and (in a private chat) 🔊 Sound; each opens a list of alert types (rain, gusts, UV, temperature crossing, particulates, pollen & asthma, forecast changes; Sound lists only those that are on) as toggles, green while on and with the name struck through while off, while one is open its own button stays (press it to close the list) and the other is hidden until you do; `/alerts on|off` still subscribes or unsubscribes every type. `/usage` is the Help & Alerts button. In a group anyone can open `/alerts`, but only admins can change anything (`lib/alerts/menu.py`). +and unhealthy air (two readings in a row; with when it is safe again; both with the last day's particulates chart). Every alert carries one 🔔 button that opens up to two toggles for that alert type: Alert (on or off in this chat) and, while that is on and in a private chat, Sound (whether it makes a notification sound; alerts are silent unless you turn it on). Every toggle in the bot is a blue button while on and a plain one with its name struck through while off. `/alerts` has two buttons, 🔔 Alerts and (in a private chat) 🔊 Sound; each opens a list of alert types (rain, gusts, UV, temperature crossing, particulates, pollen & asthma, forecast changes; Sound lists only those that are on) as toggles, blue while on and with the name struck through while off, while one is open its own button stays (press it to close the list) and the other is hidden until you do; `/alerts on|off` still subscribes or unsubscribes every type, and turning off every type one by one is the same as `/alerts off` (a type added later stays off too). A chat that blocks or removes the bot gets no more alerts but keeps its settings, which apply again when it next writes (or the bot is added back). `/usage` is the Help & Alerts button. In a group anyone can open `/alerts`, but only admins can change anything (`lib/alerts/menu.py`). ## Run @@ -29,12 +29,12 @@ The state and caches go in `/data` (its `DATA_DIR`), here a named volume. A host |---|---| | `TELEGRAM_BOT_TOKEN`, `XAI_API_KEY` | required | | `XAI_MODEL` (`grok-4.3`), `XAI_BASE_URL` (`https://api.x.ai/v1`) | model | -| `TZ` | e.g. `Australia/Melbourne` | +| `TZ` | only without an Ecowitt station: the station names its own time zone, which is used (a `TZ` that disagrees is logged). Without either, the system's zone (what `/etc/localtime` links to); e.g. `Australia/Melbourne` | | `ECOWITT_API_KEY`, `ECOWITT_APP_KEY` | enables Ecowitt | | `AIRGRADIENT_API_TOKEN`, `AIRGRADIENT_LOCATION_ID` | enables AirGradient | | `DATA_DIR` | optional: the folder that holds the bot's state (`bot_state.json`) and its SQLite caches (`ecowitt_cache.sqlite`, `airgradient_cache.sqlite`, `conditions_cache.sqlite`), kept apart from the code so a container can mount it (e.g. `/data`). A relative path is also accepted and is taken from the working directory; the folder is created if missing. Unset, a systemd service with `StateDirectory=` uses that folder (`STATE_DIRECTORY`, e.g. `/var/lib/ecobot`), else the app's own folder. Matplotlib's font cache goes in a `matplotlib` sub-folder (unless `MPLCONFIGDIR` is set). Default: the app's own folder, where the files have always been | | `AIRGRADIENT_DASHBOARD_URL` | optional "live chart" link under air-quality replies and alerts | -| `POLLEN=on` | optional: Melbourne grass pollen and thunderstorm asthma risk (scraped from melbournepollen.com.au) in the report, a `pollen_asthma` tool and an alert when High. Off by default; `POLLEN_DISTRICT` picks the district (default `Central`, never printed) | +| `POLLEN=on` | optional: Melbourne grass pollen and thunderstorm asthma risk (scraped from melbournepollen.com.au) in the report, a `pollen_asthma` tool and an alert when High. Off by default; `POLLEN_DISTRICT` picks the district (default `Central`, never printed; the others are East Gippsland, Mallee, North Central, North East, Northern Country, South West, West and South Gippsland and Wimmera, in any case; one the site doesn't have is warned about at start) | | `PLACE` | the name shown after the Pollen & asthma and Forecast headings in the report (default `Melbourne`) | | `FORECAST=on` | optional: the Open-Meteo daily forecast in the report and a `weather_forecast` tool. Off by default; `FORECAST_LAT` / `FORECAST_LON` set the location, else the weather station's own is used | | `ADMIN_ONLY` | on by default: the bot answers nobody (private chats and groups alike) except the admins and creators of the groups it is in; anyone else gets "Sorry, only group admins can use this bot." when they address it (ordinary group talk is ignored). The one exception is `/alerts` in a group, which anyone may open (its buttons and `/alerts on|off` stay admin-only). `ADMIN_ONLY=off` answers everyone, as before. The admins are asked of Telegram and remembered for 10 minutes; a group counts once the bot is added to it or someone speaks there | @@ -42,12 +42,14 @@ The state and caches go in `/data` (its `DATA_DIR`), here a named volume. A host | `ALERT_COOLDOWN_MINUTES` | optional: the rain and temperature-crossing alerts: how long it must stay dry before "the rain has stopped" is sent, and the least time between alerts of the same kind (default 30, from 5 to 150). A change that comes inside the cooldown is held and announced when it ends, if the state still differs from the last one alerted; if it is back where it was, nothing is sent | | `RAIN_QUIET_HOURS` | optional: local hours with no rain alerts (started, stopped), as `0-6` (the default; `22-6` wraps midnight; `off` for none). Rain that fell in them is summed up in one message once they end | | `CHART_ALL_FEELS_LIKE=on` | optional: "weather all week" (every reading, a panel each, sent with just its title; UV is left out as it has the same shape as solar) also draws the feels-like panel; off by default, as it nearly repeats temperature. Naming either ("plot feels like", "plot solar and uv") always works | -| `TUYA_DEVICE_ID`, `TUYA_CLIENT_ID`, `TUYA_CLIENT_SECRET` | enables the irrigation controller (a Tuya smart irrigation timer, through the Tuya cloud API): an Irrigation button in private chats (it replaces Wind) with its status and buttons to water for 5 to 30 minutes or switch the valve off, pause for 24 to 72 hours (the Pause button becomes Unpause while paused), refresh, and an Alerts menu to turn the chat's irrigation battery and irrigation pause alerts on or off (or type irrigation, water, tap or sprinkler); a daily check at 18:00 that delays it 24 hours when more than 1 mm fell in the last 24 hours or Open-Meteo gives today or tomorrow a 50% or better chance of 1 mm (needs Ecowitt and `FORECAST=on` for each half); an "irrigation pause" alert when it does, and an "irrigation battery" alert below 5%. `TUYA_BASE_URL` (default `https://openapi.tuyaeu.com`, the EU data centre) and `IRRIGATION_CHECK_HOUR` (default `18`) are optional | +| `TUYA_DEVICE_ID`, `TUYA_CLIENT_ID`, `TUYA_CLIENT_SECRET` | enables the irrigation controller (a Tuya smart irrigation timer, through the Tuya cloud API): an Irrigation button in private chats (it replaces Wind) with its status and buttons to water for 5 to 30 minutes or switch the valve off, pause for 24 to 72 hours (the Pause button becomes Unpause while paused), refresh, and an Alerts menu to turn the chat's irrigation battery and irrigation pause alerts on or off (or type irrigation, water, tap or sprinkler); a daily check at 18:00 that delays it 24 hours when more than 1 mm fell in the last 24 hours or Open-Meteo gives today or tomorrow a 50% or better chance of 1 mm (needs Ecowitt and `FORECAST=on` for each half; a forecast more than a day old doesn't count); on a second rainy day the bot's own delay is started again, and a delay you set is left alone; an "irrigation pause" alert when it does, and an "irrigation battery" alert below 5%. `TUYA_BASE_URL` (default `https://openapi.tuyaeu.com`, the EU data centre) and `IRRIGATION_CHECK_HOUR` (default `18`) are optional | | `CHART_ALL_VPD=on` | optional: "weather all week" also draws the vapour pressure deficit panel; off by default, as it is temperature and humidity combined (the station reports it for the outdoor sensor only). Naming it ("plot vpd") always works | | `TELEGRAM_PRELOAD_CHAT` | optional: a chat id (negative for a group or channel) the bot silently uploads charts to ahead of need. With it set, each chart sent with period buttons has its other periods drawn and uploaded there in the background, each on its own the moment it is drawn, starting as soon as the chart is sent and whatever else is being answered, so pressing a period button edits the message with Telegram's copy and uploads nothing. A chart uploaded this way is used for 30 minutes, then drawn afresh. The bot must be able to post in that chat; the uploaded messages are left there. If Telegram refuses an upload (the bot can't post there) none is tried again until a restart; a flood limit or a timeout only skips that photo. Unset: off | | `ANTISPIKE_REQUIRE_CONSECUTIVE` | optional: how many readings in a row a rise of the particles (PM1, PM2.5, PM10) needs to count as the air rather than a glitch of the sensor. Default `2`: a single-reading spike (at least 2.5 times, and 5 µg/m³ above, the readings either side) is left out of PM charts and of every air figure (highs, lows, averages, days over a limit), and the unhealthy-air alert counts a check only when the sensor's last 2 readings are all over the mask level. `3` also drops two-reading spikes, and so on; `1` or `off` turns the rule off. The live "air now" reading and the "safe again" check are unchanged | | `LOG_WIDTH` | optional: the most characters of a log line, cut with an ellipsis (…); `0` cuts none. Default 71. Every log line obeys it, whatever its level (a warning is printed as `WARN`); a request, chat message, reply or alert is also put on one line | -| `LOG_LEVEL` | optional: `DEBUG`, `INFO` (default), `WARNING` or `ERROR`. `DEBUG` adds the bot's own debug lines (the chart timings, charts it couldn't draw ahead); the libraries stay at their usual level. | +| `LOG_LEVEL` | optional: `DEBUG`, `INFO` (default), `WARNING` or `ERROR` (any other value keeps `INFO`, with a warning). `DEBUG` adds the bot's own debug lines (the chart timings, charts it couldn't draw ahead); the libraries stay at their usual level. | + +An on/off setting takes `on`, `yes`, `true` or `1`, and `off`, `no`, `false` or `0`; any other value keeps its default (and the log says so), so a typo can't switch `ADMIN_ONLY` off. A setting left blank is its default. ## Layout diff --git a/ecobot.py b/ecobot.py index d4344d5..98e5001 100644 --- a/ecobot.py +++ b/ecobot.py @@ -5,10 +5,12 @@ import asyncio import contextlib +import dataclasses import logging import os import signal import time +from collections.abc import Coroutine from lib.config import ROOT, Config, data_dir @@ -16,7 +18,7 @@ # Set before anything imports Matplotlib; an MPLCONFIGDIR in the environment wins. os.environ.setdefault("MPLCONFIGDIR", str(data_dir() / "matplotlib")) -from openai import AsyncOpenAI +from openai import AsyncOpenAI, Timeout from telegram import Update from telegram.ext import Application, Defaults @@ -25,16 +27,17 @@ from lib.airgradient import metrics as air_metrics from lib.alerts import (PARTICULATES_CHECK_SECONDS, AlertState, ForecastMonitor, IrrigationMonitor, Notifier, ParticulatesMonitor, PollenAndAsthmaMonitor, WeatherMonitor) -from lib.alerts.menu import LABELS, kinds_for -from lib.bot import PREDRAW_SECONDS, Bot, polling_error +from lib.alerts.menu import LABELS, available_kinds +from lib.bot import PREDRAW_SECONDS, TURN_SECONDS, Bot, polling_error from lib.ecowitt import Archive, Ecowitt from lib.forecast import Forecast from lib.irrigation import Tuya -from lib.irrigation.watchdog import CHECK_SECONDS as WATCHDOG_SECONDS +from lib.irrigation.watchdog import CHECK_SECONDS as VALVE_CHECK_SECONDS from lib.llm import Agent from lib.logs import LogFormat from lib.pollen import Pollen from lib.compose import Composer +from lib.sources import Source from lib.tools import Tools from lib.warm import every, safely @@ -44,33 +47,131 @@ logging.basicConfig(level=logging.INFO, handlers=[_handler]) logging.getLogger("httpx").setLevel(logging.WARNING) logging.getLogger("telegram.ext.Application").setLevel(logging.WARNING) # "Application started" etc. -_level = os.environ.get("LOG_LEVEL", "INFO").strip().upper() # LOG_LEVEL: DEBUG shows the bot's own debug lines (chart timings...), not the libraries' -if _level in ("DEBUG", "WARNING", "ERROR"): - for _name in ("ecobot", "lib"): - logging.getLogger(_name).setLevel(_level) log = logging.getLogger("ecobot") -async def start_sources(cfg: Config) -> list: - """Every configured data source that starts: the weather station and the air sensor, then the optional website sources - (pollen, forecast), which are off unless switched on. The forecast uses the station's location unless one is configured.""" - sources = [] +async def start_sources(cfg: Config) -> tuple[list[Source], Config]: + """Every configured data source that starts, and the settings in the time zone they run in. The weather station starts + first: it names its own time zone, which wins over TZ, so the air sensor and the optional website sources (pollen, + forecast, off unless switched on), and everything built after, use the station's. The forecast uses the station's + location unless one is configured.""" + sources: list[Source] = [] - async def start(source): + async def start(kind: type[Source], *args): + """A source made and started; one that fails either way is logged and left out.""" + source = None try: + source = kind(*args) await source.start() sources.append(source) except Exception: - log.exception("%s couldn't start - continuing without it", source.name) - await source.close() - for source in ([Ecowitt(cfg)] if cfg.ecowitt else []) + ([AirGradient(cfg)] if cfg.airgradient else []): - await start(source) + log.exception("%s couldn't start - continuing without it", kind.name) + if source is not None: + await safely(source.close) + if cfg.ecowitt: + await start(Ecowitt, cfg) + eco = sources[0] if sources else None + if eco and str(eco.tz) != str(cfg.tz): + if os.getenv("TZ"): + log.warning("TZ=%s is not the weather station's time zone; using the station's, %s", os.getenv("TZ"), eco.tz) + cfg = dataclasses.replace(cfg, tz=eco.tz) + if not getattr(cfg.tz, "key", None): + log.warning("The time zone has no name, only an offset (%s), so daylight saving will put the times out: set TZ, " + "e.g. TZ=Australia/Melbourne", cfg.tz) + if cfg.airgradient: + await start(AirGradient, cfg) if cfg.pollen: - await start(Pollen(cfg)) + await start(Pollen, cfg) if cfg.forecast: - eco = next((s for s in sources if isinstance(s, Ecowitt)), None) - await start(Forecast(cfg, (eco.latitude, eco.longitude) if eco and eco.latitude is not None else None)) - return sources + await start(Forecast, cfg, (eco.latitude, eco.longitude) if eco and None not in (eco.latitude, eco.longitude) else None) + return sources, cfg + + +class Tasks: + """The tasks the bot runs in the background: one that stops with an error is logged as it does, and close() cancels them.""" + + def __init__(self): + self.running: set[asyncio.Task] = set() + + def start(self, work: Coroutine) -> asyncio.Task: + task = asyncio.create_task(work, name=getattr(work, "__qualname__", None)) + self.running.add(task) + task.add_done_callback(self._ended) + return task + + def _ended(self, task: asyncio.Task): + self.running.discard(task) + if not task.cancelled() and (error := task.exception()): + log.error("%s stopped: %s: %s", task.get_name(), type(error).__name__, error, exc_info=error) + + async def close(self): + for task in self.running: + task.cancel() + await asyncio.gather(*self.running, return_exceptions=True) + + +def start_alerts(cfg: Config, bot: Bot, notify: Notifier, tasks: Tasks) -> tuple[list[str], WeatherMonitor | None]: + """Hook up the alert monitor of every source that started and of the irrigation controller. The weather alerts start in + warm_up, once the station has been read. Returns the alert types now running, and the weather monitor.""" + eco, air, pollen, forecast = (bot.by_name.get(name) for name in ("Ecowitt", "AirGradient", "Pollen", "Forecast")) + running: list[str] = [] + weather = WeatherMonitor(eco, bot.state, notify, cfg.alert_cooldown_minutes * 60, cfg.rain_quiet_hours) if eco else None + if air: + tasks.start(every(PARTICULATES_CHECK_SECONDS, ParticulatesMonitor(air, bot.state, notify).check)) + running += ParticulatesMonitor.KINDS + if forecast: + forecast.warmer.after.append(ForecastMonitor(forecast, bot.state, notify).check) # after each refresh (in the day) + running += ForecastMonitor.KINDS + if pollen: + pollen.warmer.after.append(PollenAndAsthmaMonitor(pollen, bot.state, notify).check) # after each refresh (in the day) + running += PollenAndAsthmaMonitor.KINDS + if bot.irrigation: # the daily check (rain delay, battery) and the watchdog that closes the valve after a run + tasks.start(IrrigationMonitor(bot.irrigation, bot.state, notify, cfg.tz, cfg.irrigation_check_hour, eco, forecast).run()) + tasks.start(every(VALVE_CHECK_SECONDS, bot.watchdog.check)) + running += IrrigationMonitor.KINDS + return running, weather + + +async def warm_up(bot: Bot, weather: WeatherMonitor | None, running: list[str], tasks: Tasks): + """Fetch everything once, together, and log one summary line; then start what waited for it: the weather alerts, the + archives (copying each source's history into its cache) and the button charts drawn ahead.""" + eco, air = bot.by_name.get("Ecowitt"), bot.by_name.get("AirGradient") + started, parts = time.monotonic(), [] + + async def ecowitt_part(): + # One request at a time (Ecowitt rate-limits): warm first, so the alerts' own look at the last 3 hours is answered + # from what was just fetched. A failed first refresh must not stop the alerts or the archive: safely() logs it + parts.append(await safely(eco.warm, True) or "Ecowitt failed") + try: + await weather.init() + eco.warmer.after.append(weather.check) # checked on every fresh refresh + running[:0] = WeatherMonitor.KINDS + await weather.check() + except Exception: + log.exception("Weather alerts couldn't start") + # The full history is copied into the cache in the background (a few minutes the first time), so questions rarely + # need Ecowitt inline + tasks.start(Archive(eco).loop()) + + async def air_part(): + parts.append(await safely(air.warm, True) or "AirGradient failed") + tasks.start(safely(air_backfill)) # even if the sensor was unreachable just now + + async def air_backfill(): + started = time.monotonic() + fetched, failed, total = await air.backfill() + log.info("AirGradient archive: %d days cached, %d failed, %d held, %.0fs", fetched, failed, total, time.monotonic() - started) + + try: + await asyncio.gather(*([ecowitt_part()] if eco else []), *([air_part()] if air else [])) + except Exception: + log.exception("Startup warm-up failed (the regular refreshes will catch up)") + log.info("Startup warm-up: %s, %.1fs", ", ".join(parts) or "nothing to fetch", time.monotonic() - started) + tasks.start(every(PREDRAW_SECONDS, bot.predraw)) # the button charts, ready before they are pressed + on = [k for k in LABELS if k in running] # the line says what started, not what is configured + log.info("Alerts: %s: %d chats", ", ".join(LABELS[k] for k in on) or "none", len(bot.state.alert_chats())) + if missing := [LABELS[k] for k in available_kinds(bot.by_name, bot.irrigation is not None) if k not in running]: + log.warning("Alerts NOT running: %s (see the error above)", ", ".join(missing)) def version() -> str: @@ -99,134 +200,68 @@ async def main(): log.info("Starting ecobot %s", version()) cfg = Config.from_env() _handler.formatter.width = cfg.log_width # every log line is cut to it from here on + for name in ("ecobot", "lib"): # the bot's own lines: the libraries keep theirs + logging.getLogger(name).setLevel(cfg.log_level) air_metrics.REQUIRE_CONSECUTIVE = cfg.antispike_require_consecutive intent.FEELS_LIKE_IN_ALL = cfg.chart_all_feels_like intent.VPD_IN_ALL = cfg.chart_all_vpd - sources = await start_sources(cfg) - if not sources: - raise SystemExit("No data source is working - nothing to talk about") - eco = next((s for s in sources if isinstance(s, Ecowitt)), None) - air = next((s for s in sources if isinstance(s, AirGradient)), None) - pollen = next((s for s in sources if isinstance(s, Pollen)), None) - forecast = next((s for s in sources if isinstance(s, Forecast)), None) - - composer = Composer(eco, air) if eco and air else None # charts and comparisons across the two sources - tools = Tools([t for s in sources for t in s.tools] + (composer.tools if composer else [])) - agent = Agent(AsyncOpenAI(api_key=cfg.xai_api_key, base_url=cfg.xai_base_url), cfg.xai_model, tools) - state = AlertState(cfg.state_path) - if cfg.admin_only: - log.info("ADMIN_ONLY=on: only admins are answered (see: ADMIN_CHAT_IDS)") - if not cfg.admin_chat_ids and not any(chat < 0 for chat in state.chats): - log.warning("Admin only is on but no group is known yet and ADMIN_CHAT_IDS is empty, so nobody can use the bot: " - "add it to a group, list your user id in ADMIN_CHAT_IDS, or set ADMIN_ONLY=off") - irrigation = Tuya(cfg) if cfg.irrigation else None - bot = Bot(cfg, agent, sources, state, irrigation) - - app = (Application.builder().token(cfg.telegram_token).concurrent_updates(True) - .defaults(Defaults(disable_notification=True)) # silent messages - .connect_timeout(10).read_timeout(15).write_timeout(15) # default 5s is tight - .build()) - bot.register(app) - - stop = asyncio.Event() - loop = asyncio.get_running_loop() - for sig in (signal.SIGINT, signal.SIGTERM): - loop.add_signal_handler(sig, stop.set) - - tasks: list[asyncio.Task] = [] - archive = Archive(eco) if eco else None - try: - async with app: # one task runs everything, so connections open and close cleanly - await app.start() - await app.updater.start_polling(allowed_updates=Update.ALL_TYPES, drop_pending_updates=False, - error_callback=polling_error) # messages sent while starting are answered (lib.bot.PENDING_MAX_SECONDS) - try: # any startup failure below still runs the shutdown steps (and shows the real error) - log.info("@%s + %s", app.bot.username, cfg.xai_model) - log.info("Sources: %s", ", ".join(s.name for s in sources)) - kinds = kinds_for((s.name for s in sources), irrigation is not None) - notify = Notifier(app.bot, state) - await bot.refresh_keyboards(app.bot) # chats whose buttons are out of date are told, with the new ones - - # Alerts - running: list[str] = [] # the alert types whose monitors are hooked up (the weather ones once they have started) - if eco: - monitor = WeatherMonitor(eco, state, notify, cfg.alert_cooldown_minutes * 60, cfg.rain_quiet_hours) - if air: - particulates_monitor = ParticulatesMonitor(air, state, notify) - tasks.append(asyncio.create_task(every(PARTICULATES_CHECK_SECONDS, particulates_monitor.check))) - running += ParticulatesMonitor.KINDS - if forecast: - forecast.warmer.after.append(ForecastMonitor(forecast, state, notify).check) # after each refresh (in the day) - running += ForecastMonitor.KINDS - if pollen: - pollen.warmer.after.append(PollenAndAsthmaMonitor(pollen, state, notify).check) # after each refresh (in the day) - running += PollenAndAsthmaMonitor.KINDS - if irrigation: # the daily check (rain delay, battery) and the watchdog that closes the valve after a run - tasks.append(asyncio.create_task(IrrigationMonitor(irrigation, state, notify, cfg.tz, cfg.irrigation_check_hour, - eco, forecast).run())) - tasks.append(asyncio.create_task(every(WATCHDOG_SECONDS, bot.watchdog.check))) - running += IrrigationMonitor.KINDS - - # Keeping warm: everything questions need, refreshed before they arrive - tasks.extend(asyncio.create_task(s.warmer.run()) for s in sources) - log.info("Keeping warm: %s", ", ".join(f"{s.name} {s.warmer.interval:.0f}s" for s in sources)) - - async def startup_warmup(): - """Fetch everything once, together, and log one summary line.""" - started, parts = time.monotonic(), [] - - async def ecowitt_part(): - # One request at a time (Ecowitt rate-limits): warm first, so the alerts' - # own look at the last 3 hours is answered from what was just fetched - # A failed first refresh must not stop the alerts or the archive: safely() logs it - parts.append(await safely(eco.warm, True) or "Ecowitt failed") - try: - await monitor.init() - eco.warmer.after.append(monitor.check) # checked on every fresh refresh - running[:0] = WeatherMonitor.KINDS - await monitor.check() - except Exception: - log.exception("Weather alerts couldn't start") - # The full history is copied into the cache in the background (a few minutes - # the first time), so questions rarely need Ecowitt inline - tasks.append(asyncio.create_task(archive.loop())) - - async def air_part(): - parts.append(await safely(air.warm, True) or "AirGradient failed") - tasks.append(asyncio.create_task(safely(air_backfill))) # even if the sensor was unreachable just now - - async def air_backfill(): - started = time.monotonic() - fetched, failed, total = await air.backfill() - log.info("AirGradient archive: %d days cached, %d failed, %d held, %.0fs", - fetched, failed, total, time.monotonic() - started) - - try: - await asyncio.gather(*([ecowitt_part()] if eco else []), *([air_part()] if air else [])) - except Exception: - log.exception("Startup warm-up failed (the regular refreshes will catch up)") - log.info("Startup warm-up: %s, %.1fs", ", ".join(parts) or "nothing to fetch", - time.monotonic() - started) - tasks.append(asyncio.create_task(every(PREDRAW_SECONDS, bot.predraw))) # the button charts, ready before they are pressed - on = [k for k in LABELS if k in running] # the line says what started, not what is configured - log.info("Alerts: %s: %d chats", ", ".join(LABELS[k] for k in on) or "none", len(state.alert_chats())) - if missing := [LABELS[k] for k in kinds if k not in running]: - log.warning("Alerts NOT running: %s (see the error above)", ", ".join(missing)) - - tasks.append(asyncio.create_task(startup_warmup())) - await stop.wait() - finally: - for task in tasks: - task.cancel() - with contextlib.suppress(asyncio.CancelledError): - await task - await app.updater.stop() - await app.stop() - finally: + # Whatever starts is closed on the way out, the last started first, however the bot stops (a failure part-way included) + async with contextlib.AsyncExitStack() as stack: + sources, cfg = await start_sources(cfg) # in the weather station's time zone from here on for source in sources: - await source.close() + stack.push_async_callback(safely, source.close) + if not sources: + raise SystemExit("No data source is working - nothing to talk about") + by_name = {s.name: s for s in sources} + eco, air = by_name.get("Ecowitt"), by_name.get("AirGradient") + composer = Composer(eco, air) if eco and air else None # charts and comparisons across the two sources + tools = Tools([t for s in sources for t in s.tools] + (composer.tools if composer else [])) + # A model request may run as long as the bot gives a whole question (it then asks again, a reasoning step lower); a + # dropped connection, a 429 or a 5xx is tried again twice, at once, inside that time + client = AsyncOpenAI(api_key=cfg.xai_api_key, base_url=cfg.xai_base_url, timeout=Timeout(TURN_SECONDS, connect=5.0), + max_retries=2) + stack.push_async_callback(safely, client.close) + state = AlertState(cfg.state_path) + if cfg.admin_only: + log.info("ADMIN_ONLY=on: only admins are answered (see: ADMIN_CHAT_IDS)") + if not cfg.admin_chat_ids and not any(chat < 0 for chat in state.chats): + log.warning("Admin only is on but no group is known yet and ADMIN_CHAT_IDS is empty, so nobody can use the bot: " + "add it to a group, list your user id in ADMIN_CHAT_IDS, or set ADMIN_ONLY=off") + irrigation = Tuya(cfg) if cfg.irrigation else None if irrigation: - await irrigation.close() + stack.push_async_callback(safely, irrigation.close) + bot = Bot(cfg, Agent(client, cfg.xai_model, tools), sources, state, irrigation) + stack.push_async_callback(bot.close) + + app = (Application.builder().token(cfg.telegram_token).concurrent_updates(True) + .defaults(Defaults(disable_notification=True)) # silent messages + .connect_timeout(10).read_timeout(15).write_timeout(15) # default 5s is tight + .build()) + bot.register(app) + stop = asyncio.Event() + for sig in (signal.SIGINT, signal.SIGTERM): + asyncio.get_running_loop().add_signal_handler(sig, stop.set) + await stack.enter_async_context(app) # one task runs everything, so connections open and close cleanly + await app.start() + stack.push_async_callback(app.stop) + await app.updater.start_polling(allowed_updates=Update.ALL_TYPES, drop_pending_updates=False, + error_callback=polling_error) # messages sent while starting are answered (lib.bot.PENDING_MAX_SECONDS) + stack.push_async_callback(app.updater.stop) + tasks = Tasks() + stack.push_async_callback(tasks.close) + for source in sources: + stack.push_async_callback(source.warmer.stop) + + log.info("@%s + %s", app.bot.username, cfg.xai_model) + log.info("Sources: %s", ", ".join(s.name for s in sources)) + notify = Notifier(app.bot, state) + await bot.refresh_keyboards(app.bot) # chats whose buttons are out of date are told, with the new ones + running, weather = start_alerts(cfg, bot, notify, tasks) + for source in sources: # keeping warm: everything questions need, refreshed before they arrive + tasks.start(source.warmer.run()) + log.info("Keeping warm: %s", ", ".join(f"{s.name} {s.warmer.interval:.0f}s" for s in sources)) + tasks.start(warm_up(bot, weather, running, tasks)) + await stop.wait() if __name__ == "__main__": diff --git a/lib/admins.py b/lib/admins.py index c4f0da6..207d525 100644 --- a/lib/admins.py +++ b/lib/admins.py @@ -4,9 +4,11 @@ added to it or someone speaks in it) or of a group named by a negative id in ADMIN_CHAT_IDS. The lists are asked for once and kept for ten minutes. ADMIN_CHAT_IDS is added to that, never instead of it: a positive id is a user (it bootstraps a bot that is in no group yet), a negative id is a group, whose own admins are then looked up and whose anonymous admins are let in: a message an admin sends "as the -group" carries the group as its `sender_chat` and no real user, so it is allowed only when that group's id is listed. +group" carries the group as its `sender_chat` and no real user, so it is allowed only when that group's id is listed. The groups' +lists are asked for together, and messages that arrive together while a list is being fetched wait for that one lookup. """ +import asyncio import logging import time @@ -21,36 +23,36 @@ def __init__(self, state, extra_ids=(), ttl: float = TTL_SECONDS, clock=time.mon self.state, self.extra, self.ttl, self.clock = state, frozenset(extra_ids), ttl, clock self._cache: dict[int, tuple[float, frozenset[int]]] = {} # group -> (when fetched, its admins) self._failing: set[int] = set() + self._looking: dict[int, asyncio.Lock] = {} # group -> held while its list is fetched def groups(self) -> list[int]: - """The groups whose admins count: the ones the bot knows, and the ones ADMIN_CHAT_IDS names.""" - known = {c for c in (self.state.chats if self.state else {}) if c < 0} + """The groups whose admins count: the ones the bot is in (not one that removed it), and the ones ADMIN_CHAT_IDS names.""" + known = {c for c in self.state.present() if c < 0} return sorted(known | {i for i in self.extra if i < 0}) async def _admins_of(self, bot, chat_id: int) -> frozenset[int]: - cached = self._cache.get(chat_id) - if cached and self.clock() - cached[0] < self.ttl: - return cached[1] - try: - members = await bot.get_chat_administrators(chat_id) - ids = frozenset(m.user.id for m in members if not getattr(m.user, "is_bot", False)) - except Exception as e: # the bot was removed, Telegram is down: keep what was known, try again soon, say it once - if chat_id not in self._failing: - self._failing.add(chat_id) - log.warning("Couldn't list the admins of %s (%s: %s)", chat_id, type(e).__name__, e) - ids = cached[1] if cached else frozenset() - self._cache[chat_id] = (self.clock() - self.ttl + RETRY_SECONDS, ids) + async with self._looking.setdefault(chat_id, asyncio.Lock()): # one lookup at a time: who waited finds it done + cached = self._cache.get(chat_id) + if cached and self.clock() - cached[0] < self.ttl: + return cached[1] + try: + members = await bot.get_chat_administrators(chat_id) + ids = frozenset(m.user.id for m in members if not getattr(m.user, "is_bot", False)) + except Exception as e: # the bot was removed, Telegram is down: keep what was known, try again soon, say it once + if chat_id not in self._failing: + self._failing.add(chat_id) + log.warning("Couldn't list the admins of %s (%s: %s)", chat_id, type(e).__name__, e) + ids = cached[1] if cached else frozenset() + self._cache[chat_id] = (self.clock() - self.ttl + RETRY_SECONDS, ids) + return ids + self._failing.discard(chat_id) + self._cache[chat_id] = (self.clock(), ids) return ids - self._failing.discard(chat_id) - self._cache[chat_id] = (self.clock(), ids) - return ids async def allowed(self, bot, sender_ids) -> bool: """Is one of these ids (the sender's user id and, for a message sent as a chat, that chat's id) allowed?""" ids = {i for i in sender_ids if isinstance(i, int)} if ids & self.extra: return True - for chat_id in self.groups(): - if ids & await self._admins_of(bot, chat_id): - return True - return False + lists = await asyncio.gather(*(self._admins_of(bot, chat_id) for chat_id in self.groups())) + return any(ids & admins for admins in lists) diff --git a/lib/airgradient/__init__.py b/lib/airgradient/__init__.py index 0961ac5..4c07dd6 100644 --- a/lib/airgradient/__init__.py +++ b/lib/airgradient/__init__.py @@ -1,4 +1,4 @@ from .metrics import pm25_aqi, rating -from .source import AirGradient +from .source import AirGradient, Latest -__all__ = ["AirGradient", "pm25_aqi", "rating"] +__all__ = ["AirGradient", "Latest", "pm25_aqi", "rating"] diff --git a/lib/airgradient/days.py b/lib/airgradient/days.py index 9776b4e..58131e5 100644 --- a/lib/airgradient/days.py +++ b/lib/airgradient/days.py @@ -3,13 +3,13 @@ days particles passed 25 µg/m³. The counting, ranking, grouping and bar charts are shared with weather_days (lib/daytable.py).""" import asyncio -import json import logging -from datetime import date, datetime, timedelta, tzinfo +from datetime import date, datetime, tzinfo from statistics import fmean -from ..daytable import analyse, grouped_as_asked, parameters, validate +from ..daytable import analyse, grouped_as_asked, parameters, prepare from ..timeutil import day_bounds, local_date, now_local +from ..tools import json_result from .metrics import ALL_METRICS, CHART_UNITS, without_spikes log = logging.getLogger(__name__) @@ -17,6 +17,7 @@ HOURLY_MAX_READINGS = 30 # a day with this few readings holds hourly averages (24), not 5-minute ones (288) SUFFIX_KIND = {"max": max, "min": min, "avg": fmean} FIELDS = {f"{m}_{kind}": CHART_UNITS[m] for m in ALL_METRICS for kind in SUFFIX_KIND} +ALWAYS_SHOWN = ("pm2_5_max",) # every listed day shows it; days rank by it when nothing else is named PARAMETERS = parameters(list(FIELDS), "pm2_5_max > 25 and co2_max > 800") DESCRIPTION = ("Find, rank or count DAYS by the air-quality sensor's readings, from the stored history (any period, fast): for " @@ -44,26 +45,15 @@ def daily_values(rows: list[dict], shown: list[str], tz: tzinfo) -> dict[str, di def find_air_days(store, tz: tzinfo, args: dict, now: datetime, turn=None) -> dict: - today = now.date() - try: - first = date.fromisoformat(str(args["start_date"])[:10]) - last = min(date.fromisoformat(str(args["end_date"])[:10]), today - timedelta(days=1)) - except (KeyError, ValueError) as e: - return {"error": f"bad date ({e}); use 'YYYY-MM-DD'"} - if first > last: - return {"error": "start_date must be before end_date (today isn't final, so the latest day is yesterday)"} - if problem := validate(args, FIELDS): - return {"error": problem} - where = args.get("where") or [] - sort_by = args.get("sort_by") or (where[0]["field"] if where else args.get("of") or "pm2_5_max") - args = {**args, "sort_by": sort_by} - shown = list(dict.fromkeys(["pm2_5_max", sort_by, *(c["field"] for c in where), *([args["of"]] if args.get("of") else [])])) - days = [first + timedelta(days=k) for k in range((last - first).days + 1)] - rows = store.load(day_bounds(first, tz)[0], day_bounds(last, tz)[1]) + asked = prepare(args, FIELDS, ALWAYS_SHOWN, now.date()) + if isinstance(asked, str): + return {"error": asked} + days, shown = asked.days, asked.shown + rows = store.load(day_bounds(asked.first, tz)[0], day_bounds(asked.last, tz)[1]) counts = store.counts(days) values = daily_values(rows, shown, tz) checked = [d for d in days if any(d in values[n] for n in shown)] - got = analyse(values, shown, {n: FIELDS[n] for n in shown}, args, checked, first, last, tz, turn) + got = analyse(values, asked, checked, tz, turn) out = got.out hourly = [d for d in checked if (counts.get(d) or 0) <= HOURLY_MAX_READINGS] if hourly and any(n.endswith(("_max", "_min")) for n in shown): @@ -74,10 +64,10 @@ def find_air_days(store, tz: tzinfo, args: dict, now: datetime, turn=None) -> di out["note_missing"] = f"{missing} day(s) in this period have no stored readings (not downloaded yet, or the sensor was off) and are not counted." if not checked: out["note"] = "No stored readings for this period yet (the bot may still be downloading history)." - log.info("Air days %s to %s: %d checked, %d match", first, last, len(checked), len(got.matches)) + log.info("Air days %s to %s: %d checked, %d match", asked.first, asked.last, len(checked), len(got.matches)) return out async def air_days_tool(store, tz: tzinfo, args: dict, turn=None) -> str: result = await asyncio.to_thread(find_air_days, store, tz, grouped_as_asked(args, turn), now_local(tz), turn) - return json.dumps(result, ensure_ascii=False, separators=(",", ":")) + return json_result(result) diff --git a/lib/airgradient/metrics.py b/lib/airgradient/metrics.py index ac082ce..65fc3b9 100644 --- a/lib/airgradient/metrics.py +++ b/lib/airgradient/metrics.py @@ -2,6 +2,8 @@ from datetime import datetime +from ..series import AIR + # name -> (fields to try, in order: corrected first), unit METRICS = { "pm2_5": (("pm02_corrected", "pm02"), "µg/m³"), @@ -12,26 +14,12 @@ "nox_index": (("noxIndex", "nox_index"), "relative index (1 = baseline)"), } -# Traffic-light ratings: value <= first -> good, <= second -> poor, above -> very poor. -# Particles follow the US AQI (very poor = AQI 151+, the same level as the mask alerts); -# PM1 has no standard, so it uses PM2.5's; CO2, VOC and NOx follow AirGradient's colour scales. -RATINGS = { - "pm2_5": (9.0, 55.4), - "pm10": (54.0, 254.0), - "pm1": (9.0, 55.4), - "co2": (799.0, 1499.0), - "voc_index": (150.0, 250.0), - "nox_index": (20.0, 150.0), -} - -LABELS = {"pm2_5": "PM2.5", "pm10": "PM10", "pm1": "PM1", "co2": "CO₂", - "voc_index": "VOC index", "nox_index": "NOx index"} -CHART_UNITS = {"pm2_5": "µg/m³", "pm10": "µg/m³", "pm1": "µg/m³", "co2": "ppm", - "voc_index": "", "nox_index": ""} -ALL_METRICS = list(LABELS) -# Only these have a lowest reading worth marking on a chart: CO2 never falls to zero (it has an outdoor floor near 400 ppm), -# the others are often 0 or at their baseline, so their lowest says nothing. -MARK_LOW = ("co2",) +# Traffic-light ratings (series.AIR): value <= first -> good, <= second -> poor, above -> very poor. +RATINGS = {name: m.limits for name, m in AIR.items()} +LABELS = {name: m.label for name, m in AIR.items()} +CHART_UNITS = {name: m.unit for name, m in AIR.items()} +ALL_METRICS = list(AIR) +MARK_LOW = tuple(name for name, m in AIR.items() if m.low) # the metrics whose lowest is marked on a chart # Metrics charted together, one panel each on a shared time axis, top to bottom: the particles on one axis, then CO2, VOC and # NOx each on its own (their scales differ too much to share an axis honestly). A panel with none of the requested metrics # is left out. diff --git a/lib/airgradient/source.py b/lib/airgradient/source.py index 294cbbc..c6db0fd 100644 --- a/lib/airgradient/source.py +++ b/lib/airgradient/source.py @@ -13,26 +13,27 @@ import asyncio import statistics -import json import logging import time from datetime import date, datetime, timedelta, UTC from itertools import pairwise +from typing import NamedTuple import httpx from .. import intent from ..captions import CHART_HINT, wants_chart from ..config import Config -from ..lines import Plotted, build_line +from ..sources import Source +from ..lines import Plotted, Sample, build_line from ..panels import panel_for from ..specs import Chart, Line -from ..timeutil import local_date, now_local, to_local, when -from ..tools import Tool, Turn +from ..timeutil import day_bounds, day_text, local_date, local_epoch, parse_local, to_local, when +from ..tools import Tool, Turn, json_result from ..warm import Warmer from .days import DESCRIPTION as DAYS_DESCRIPTION, PARAMETERS as DAYS_PARAMETERS, air_days_tool from . import metrics -from .metrics import AIR_PANELS, ALL_METRICS, MARK_LOW, LABELS, METRICS, epoch, normalise, pm25_aqi, rating, value_of, without_spikes +from .metrics import AIR_PANELS, ALL_METRICS, MARK_LOW, LABELS, METRICS, normalise, pm25_aqi, rating, value_of, without_spikes from .store import AirStore log = logging.getLogger(__name__) @@ -70,7 +71,14 @@ "happened. chart=true sends a graph.") -class AirGradient: +class Latest(NamedTuple): + """The latest reading as the air_quality tool gives it, and what the particulates alert also needs.""" + reading: dict # the sensor, the time, and each metric's value, unit and rating + at: datetime | None # when it was taken (UTC) + sustained: dict[str, float] # each metric's lowest over the last REQUIRE_CONSECUTIVE readings: what the air has been that long + + +class AirGradient(Source): """AirGradient client with its data kept warm and cached: - the current reading and today's history are refreshed every few minutes (warm); - finished days are fetched once and stored on disk, so they never need fetching again.""" @@ -104,28 +112,25 @@ def wants(self, text: str) -> bool: """Should a question start refreshing this source? Only air-quality questions.""" return intent.mentions_air(text) - def poke(self): - self.warmer.poke() - - async def close(self): - await self.client.aclose() - self.store.close() - - async def _get(self, kind: str, **params): + async def _get(self, kind: str, **params) -> dict | list: + """The reply: an object for "current", a list of readings for "past". A period with no readings is an empty list (AirGradient + says so with a 404); a reply of any other shape raises, so it is never stored as days without readings.""" self.requests += 1 res = await self.client.get(API.format(loc=self.loc, kind=kind), params={**params, "token": self.token}) - if res.status_code == 404 and kind == "past": # "No data available" for the period + if res.status_code == 404 and kind == "past" and "no data" in res.text.lower(): # "No data available" for the period return [] if res.status_code != 200: raise RuntimeError(f"AirGradient {kind} failed: HTTP {res.status_code}") - return res.json() + data = res.json() + if kind == "past" and isinstance(data, dict) and isinstance(data.get("data"), list): + data = data["data"] + if not isinstance(data, list if kind == "past" else dict): + raise RuntimeError(f"AirGradient {kind} sent something unexpected: {res.text[:100]}") + return data def _lock(self, key) -> asyncio.Lock: return self._locks.setdefault(key, asyncio.Lock()) - def _now(self) -> datetime: - return now_local(self.tz) - # ---------- cached data ---------- async def _current_row(self, refresh: bool = False) -> dict: async with self._lock("current"): @@ -135,37 +140,37 @@ async def _current_row(self, refresh: bool = False) -> dict: def _final(self, day: date) -> bool: """Over for more than an hour, so its data won't change.""" - return self._now() > datetime.combine(day, datetime.min.time()) + timedelta(days=1, hours=1) + return self.now() > datetime.combine(day, datetime.min.time()) + timedelta(days=1, hours=1) async def _held(self, days: list[date]) -> dict[date, int | None]: """Readings stored per finished day, in one query (None: not fetched yet, or not finished).""" stored = await asyncio.to_thread(self.store.counts, days) return {d: stored[d] if self._final(d) else None for d in days} - def _utc(self, local: datetime) -> str: - return local.replace(tzinfo=self.tz).astimezone(UTC).strftime("%Y-%m-%dT%H:%M:%SZ") + def _unstored(self, held: dict[date, int | None]) -> list[date]: + """The finished days in `held` (see _held) that aren't stored yet.""" + return [d for d, n in held.items() if n is None and self._final(d)] - async def _past(self, start: datetime, end: datetime) -> list[dict]: - """Readings between two local times, normalised and oldest first.""" - data = await self._get("past", **{"from": self._utc(start), "to": self._utc(end)}) - raw = data if isinstance(data, list) else data.get("data", []) if isinstance(data, dict) else [] + async def _past(self, start: int, end: int) -> list[dict]: + """Readings between two epochs (never past now), normalised and oldest first.""" + utc = lambda ts: datetime.fromtimestamp(ts, UTC).strftime("%Y-%m-%dT%H:%M:%SZ") + raw = await self._get("past", **{"from": utc(start), "to": utc(min(end, time.time()))}) return sorted((normalise(r) for r in raw if isinstance(r, dict) and r.get("timestamp")), key=lambda r: r["ts"]) - async def _fetch_days(self, days: list[date]): - """Fetch finished days and store them, in as few requests as the API allows (it takes up to - 10 days at a time; we use 9).""" - days = sorted(days) + async def _fetch_days(self, days: list[date]) -> dict[date, int]: + """Fetch finished days and store them, in as few requests as the API allows (it takes up to 10 days at a time; we use 9). + Returns how many readings each day has stored.""" + days, stored = sorted(days), {} while days: window = [d for d in days if (d - days[0]).days < MAX_REQUEST_DAYS] days = days[len(window):] - start = datetime.combine(window[0], datetime.min.time()) - end = datetime.combine(window[-1], datetime.min.time()) + timedelta(days=1) by_day: dict = {d: [] for d in window} - for r in await self._past(start, min(end, self._now())): + for r in await self._past(day_bounds(window[0], self.tz)[0], day_bounds(window[-1], self.tz)[1]): by_day.get(local_date(r["ts"], self.tz), []).append(r) for d, day_rows in by_day.items(): - await asyncio.to_thread(self.store.save_day, d, day_rows) + stored[d] = await asyncio.to_thread(self.store.save_day, d, day_rows) self._recent.pop(d, None) + return stored async def _day_rows(self, day: date, refresh: bool = False) -> list[dict]: """All of one local day's readings, normalised. Finished days come from the disk cache (and @@ -174,25 +179,23 @@ async def _day_rows(self, day: date, refresh: bool = False) -> list[dict]: final = self._final(day) async with self._lock(day): if final: - if await asyncio.to_thread(self.store.day_count, day) is None: + if self._unstored(await self._held([day])): await self._fetch_days([day]) return await asyncio.to_thread(self.store.load_day, day) cached = self._recent.get(day) if cached and not refresh and time.time() - cached[1] <= FRESH_SECONDS: return cached[0] - start = datetime.combine(day, datetime.min.time()) - rows = await self._past(start, min(start + timedelta(days=1), self._now())) + rows = await self._past(*day_bounds(day, self.tz)) self._recent[day] = (rows, time.time()) return rows async def warm(self, fresh: bool = True) -> str: """Current reading, today, and the last week of finished days.""" - before, today = self.requests, self._now().date() + before, today = self.requests, self.now().date() await self._current_row(fresh) await self._day_rows(today, fresh) week = [today - timedelta(days=n) for n in range(1, WARM_DAYS + 1)] - held = await self._held(week) - await self._fetch_days([d for d in week if self._final(d) and held[d] is None]) # one request for all of them + await self._fetch_days(self._unstored(await self._held(week))) # one request for all of them for day in week: if not self._final(day): # a day that just ended is still refetched for a while; finished ones are stored await self._day_rows(day) @@ -206,7 +209,7 @@ async def backfill(self, pace: float | None = None, empty_stop: int | None = Non pace = BACKFILL_PACE if pace is None else pace empty_stop = BACKFILL_EMPTY_STOP if empty_stop is None else empty_stop empty_before_data = BACKFILL_EMPTY_BEFORE_DATA if empty_before_data is None else empty_before_data - today = self._now().date() + today = self.now().date() fetched = failed = failed_run = empty_run = total = 0 seen_data = False n = 1 @@ -214,12 +217,10 @@ async def backfill(self, pace: float | None = None, empty_stop: int | None = Non days = [today - timedelta(days=k) for k in range(n, min(n + MAX_REQUEST_DAYS, max_days + 1))] n += MAX_REQUEST_DAYS counts = await self._held(days) - missing = [d for d in days if self._final(d) and counts[d] is None] - if missing: + if missing := self._unstored(counts): try: - await self._fetch_days(missing) + counts.update(await self._fetch_days(missing)) fetched, failed_run = fetched + len(missing), 0 - counts.update(await asyncio.to_thread(self.store.counts, missing)) if fetched // 30 != (fetched - len(missing)) // 30: log.info("AirGradient archive: %d day(s) fetched so far, now at %s", fetched, days[-1]) except Exception as e: @@ -249,13 +250,9 @@ async def handle(self, args: dict, turn: Turn | None = None) -> str: try: chart = wants_chart(args, turn) if args.get("start_date") or args.get("end_date") or chart: - return json.dumps(await self.history(args.get("start_date"), args.get("end_date"), - chart=chart, metrics=args.get("metrics"), turn=turn), - ensure_ascii=False) - reading = await self.current() - reading.pop("_time_utc", None) - reading.pop("_sustained", None) - return json.dumps(reading, ensure_ascii=False) + return json_result(await self.history(args.get("start_date"), args.get("end_date"), + chart=chart, metrics=args.get("metrics"), turn=turn)) + return json_result(await self.current()) except Exception as e: log.warning("AirGradient request failed: %s", e) return f"Error: couldn't read the air-quality sensor ({e})" @@ -264,12 +261,6 @@ async def _days(self, args: dict, turn: Turn | None = None) -> str: """The air_days tool: count, rank and total days from the stored readings.""" return await air_days_tool(self.store, self.tz, args, turn) - async def _previous_row(self, ts: int) -> dict | None: - """The newest of today's cached readings older than `ts` and within MAX_GAP_SECONDS of it, else None.""" - today = to_local(ts, self.tz).date() - rows = [r for r in await self._day_rows(today) if r["ts"] < ts] - return rows[-1] if rows and ts - rows[-1]["ts"] <= MAX_GAP_SECONDS else None - async def _previous_rows(self, ts: int, count: int) -> list[dict]: """Up to `count` of today's cached readings just before `ts`, each within MAX_GAP_SECONDS of the next, oldest first.""" today = to_local(ts, self.tz).date() @@ -281,62 +272,67 @@ async def _previous_rows(self, ts: int, count: int) -> list[dict]: after = r["ts"] return got - async def current(self) -> dict: + async def latest(self) -> Latest: """The latest reading. A rating (and PM2.5's AQI band) is the level the last two readings both reached, so a passing - spike is not called poor until the next reading agrees; the value shown is always the latest one.""" + spike is not called poor until the next reading agrees; the value shown is always the latest one. The readings before it + come from today's history: when that can't be read, the latest is rated alone.""" row = await self._current_row() - previous = await self._previous_row(epoch(row["timestamp"])) if row.get("timestamp") else None - out = {"sensor": row.get("locationName"), "sensor_type": row.get("locationType"), - "time": when(epoch(row["timestamp"]), self.tz) if row.get("timestamp") else None} - if row.get("timestamp"): # for the air alerts' staleness check (removed before the model sees it) - out["_time_utc"] = datetime.fromisoformat(row["timestamp"].replace("Z", "+00:00")) + at = datetime.fromisoformat(row["timestamp"].replace("Z", "+00:00")) if row.get("timestamp") else None + need = metrics.REQUIRE_CONSECUTIVE - 1 # the readings before the latest that `sustained` takes in + try: + earlier = await self._previous_rows(int(at.timestamp()), max(need, 1)) if at else [] + except Exception as e: # only the history failing: the latest reading still stands + log.warning("AirGradient: today's readings unavailable (%s); rating the latest alone", e) + earlier = [] + previous, held = (earlier[-1] if earlier else {}), (earlier[-need:] if need else []) + reading = {"sensor": row.get("locationName"), "sensor_type": row.get("locationType"), + "time": when(int(at.timestamp()), self.tz) if at else None} + sustained = {} for name, (_, unit) in METRICS.items(): v = value_of(row, name) - if v is not None: - before = previous.get(name) if previous else None # today's rows are already normalised - rated = v if before is None else min(v, before) # thresholds only rise: the lower value's zone is the one both reached - out[name] = {"value": v, "unit": unit, "rating": rating(name, rated)} - if out[name]["rating"] != rating(name, v): - out[name]["rating_note"] = "rated on the last two readings: the previous one was lower" - out[name]["_rated"] = rated - if "pm2_5" in out: - aqi, band = pm25_aqi(out["pm2_5"]["_rated"]) - out["pm2_5"].update(aqi_us=aqi, band=band) - sustained = {} # each metric's lowest value over the last REQUIRE_CONSECUTIVE readings: what the air has been for that long - if row.get("timestamp"): - earlier = await self._previous_rows(epoch(row["timestamp"]), max(metrics.REQUIRE_CONSECUTIVE - 1, 0)) - for name, entry in out.items(): - if isinstance(entry, dict) and "value" in entry: - sustained[name] = min([entry["value"], *(r[name] for r in earlier if name in r)]) - for entry in out.values(): - if isinstance(entry, dict): - entry.pop("_rated", None) - out["_sustained"] = sustained # for the alerts (removed before the model sees it) - return out + if v is None: + continue + rated = min(v, previous.get(name, v)) # thresholds only rise: the lower value's zone is the one both reached + reading[name] = {"value": v, "unit": unit, "rating": rating(name, rated)} + if reading[name]["rating"] != rating(name, v): + reading[name]["rating_note"] = "rated on the last two readings: the previous one was lower" + if name == "pm2_5": + aqi, band = pm25_aqi(rated) + reading[name].update(aqi_us=aqi, band=band) + sustained[name] = min([v, *(r[name] for r in held if name in r)]) + return Latest(reading, at, sustained) + + async def current(self) -> dict: + """The latest reading as the air_quality tool gives it (see latest).""" + return (await self.latest()).reading async def rows(self, t0: datetime, t1: datetime) -> tuple[list[dict], int, list[date]]: """(readings in [t0, t1] (naive local times), days that only exist as hourly averages, older days still waiting for - the backfill and left out). Finished days not yet stored are fetched first, up to MAX_INLINE_DAYS.""" + the backfill and left out). Finished days not yet stored are fetched first, up to MAX_INLINE_DAYS. A single-reading spike + of the particles is a glitch of the sensor: it is taken out here, once, so no figure or chart has it.""" days = [t0.date() + timedelta(days=k) for k in range((t1.date() - t0.date()).days + 1)] held = await self._held(days) - missing = [d for d in days if self._final(d) and held[d] is None] + missing = self._unstored(held) skipped = missing[:-MAX_INLINE_DAYS] if len(missing) > MAX_INLINE_DAYS else [] # the oldest wait for the backfill - await self._fetch_days([d for d in missing if d not in skipped]) - rows = [] - for d in days: - if d not in skipped: # finished days are all stored by now; today and a just-ended day come via _day_rows - rows += await (asyncio.to_thread(self.store.load_day, d) if self._final(d) else self._day_rows(d)) - counts = await self._held([d for d in days if d not in skipped]) - hourly = sum(1 for c in counts.values() if c and c <= HOURLY_MAX_READINGS) - lo_ts, hi_ts = t0.replace(tzinfo=self.tz).timestamp(), t1.replace(tzinfo=self.tz).timestamp() - return [r for r in rows if lo_ts <= r["ts"] <= hi_ts], hourly, skipped + held.update(await self._fetch_days([d for d in missing if d not in skipped])) + finished = [d for d in days if self._final(d)] # all stored by now but the skipped: read in one query + rows = (await asyncio.to_thread(self.store.load, day_bounds(finished[0], self.tz)[0], day_bounds(finished[-1], self.tz)[1]) + if finished else []) + for d in days[len(finished):]: # today, and a day that has just ended: fetched, not stored + rows += await self._day_rows(d) + hourly = sum(1 for n in held.values() if n and n <= HOURLY_MAX_READINGS) + lo, hi = local_epoch(t0, self.tz), local_epoch(t1, self.tz) + return [r for r in without_spikes(rows) if lo <= r["ts"] <= hi], hourly, skipped async def history(self, start: str | None, end: str | None, chart: bool = False, metrics: list[str] | None = None, turn: Turn | None = None) -> dict: turn = turn or Turn() - now = self._now() - parse = lambda s, d: datetime.fromisoformat(str(s).replace("T", " ")) if s else d - t0, t1 = parse(start, now - timedelta(days=1)), min(parse(end, now), now) + now = self.now() + try: # a date the model sends, with or without an offset; a bad one is its mistake, not the sensor's + t0 = parse_local(start, self.tz) if start else now - timedelta(days=1) + t1 = min(parse_local(end, self.tz), now) if end else now + except ValueError as e: + return {"error": f"bad date ({e}); use 'YYYY-MM-DD HH:MM:SS'"} trimmed = t0 < t1 - timedelta(days=MAX_DAYS) t0 = max(t0, t1 - timedelta(days=MAX_DAYS)) if t0 >= t1: @@ -345,10 +341,9 @@ async def history(self, start: str | None, end: str | None, chart: bool = False, before, started = self.requests, time.perf_counter() rows, hourly, skipped = await self.rows(t0, t1) turn.times["data"] = turn.times.get("data", 0.0) + time.perf_counter() - started # for the reply's timing line - rows = without_spikes(rows) # a single-reading spike of the particles is a glitch of the sensor: in no figure or chart log.log(logging.DEBUG if turn.ahead else logging.INFO, "AirGradient %s to %s: %d readings, %d req", f"{t0:%Y-%m-%d}", f"{t1:%m-%d %H:%M}", len(rows), self.requests - before) - out = {"sensor_type": "outdoor", "period": f"{t0:%a} {t0.day} {t0:%b %Y} - {t1:%a} {t1.day} {t1:%b %Y}", + out = {"sensor_type": "outdoor", "period": f"{day_text(t0)} - {day_text(t1)}", "readings": len(rows)} for name, (_, unit) in METRICS.items(): pts = [(r["ts"], r[name]) for r in rows if name in r] @@ -375,7 +370,8 @@ async def history(self, start: str | None, end: str | None, chart: bool = False, if not rows: out["note"] = "No readings for this period." if chart and rows: - chart_spec = self._chart([m for m in ALL_METRICS if m in (metrics or ["pm2_5"])] or ["pm2_5"], rows, out["period"]) + chart_spec = self._chart([m for m in ALL_METRICS if m in (metrics or ["pm2_5"])] or ["pm2_5"], rows, out["period"], + (t0.date(), t1.date())) if chart_spec: turn.charts.append(chart_spec) out["chart"] = CHART_HINT @@ -389,16 +385,16 @@ def _line(self, name: str, rows: list[dict]) -> tuple[Line, Plotted] | None: if len(pts) < 2: return None gap = max(60, round(statistics.median(b[0] - a[0] for a, b in pairwise(pts)))) - plotted = build_line([(t, v, None, None, gap) for t, v in pts], self.tz, pts[-1][0] - pts[0][0]) + plotted = build_line([Sample(t, v, None, None, gap) for t, v in pts], self.tz, pts[-1][0] - pts[0][0]) if plotted is None: return None lo, hi = min(pts, key=lambda p: p[1]), max(pts, key=lambda p: p[1]) return plotted.spec(LABELS[name], {"high": hi, **({"low": lo} if name in MARK_LOW else {})}, name), plotted - def _chart(self, names: list[str], rows: list[dict], period: str) -> Chart | None: + def _chart(self, names: list[str], rows: list[dict], period: str, days: tuple[date, date] | None = None) -> Chart | None: """These metrics as one chart. One metric is drawn large, with its rating zones. Several go into panels on a - shared time axis, grouped by AIR_PANELS (the particles together, the rest each alone).""" - rows = without_spikes(rows) # (a no-op for the rows history() gives it; a caller with its own rows gets the same drawing) + shared time axis, grouped by AIR_PANELS (the particles together, the rest each alone). `period` is the subtitle's + text for it, `days` its first and last day (for the caption).""" drawn = {n: got for n in names if (got := self._line(n, rows))} if not drawn: return None @@ -406,11 +402,11 @@ def _chart(self, names: list[str], rows: list[dict], period: str) -> Chart | Non (name, (line, plotted)), = drawn.items() subtitle = (f"{period} · {plotted.name}" + (", range shaded" if plotted.low else "") + " · records marked" if not plotted.raw else f"{period} · AirGradient readings") - return Chart(LABELS[name], subtitle, [panel_for(name, [line])]) + return Chart(LABELS[name], subtitle, [panel_for(name, [line])], period=days) panels = [] for group in AIR_PANELS: members = [m for m in group if m in drawn] if members: panels.append(panel_for(members, [drawn[m][0] for m in members])) return Chart("Air quality", f"{period} · AirGradient readings" - + (" · range shaded" if any(plotted.low for _, plotted in drawn.values()) else ""), panels) + + (" · range shaded" if any(plotted.low for _, plotted in drawn.values()) else ""), panels, period=days) diff --git a/lib/airgradient/store.py b/lib/airgradient/store.py index 0e3d8bd..554f080 100644 --- a/lib/airgradient/store.py +++ b/lib/airgradient/store.py @@ -33,14 +33,8 @@ def close(self): with self._lock: self.db.close() - def day_count(self, day: date) -> int | None: - """Readings stored for this finished day, or None if it hasn't been fetched.""" - with self._lock: - row = self.db.execute("SELECT n FROM days WHERE loc=? AND day=?", (self.loc, day.isoformat())).fetchone() - return row[0] if row else None - def counts(self, days: list[date]) -> dict[date, int | None]: - """day_count for many days in one query.""" + """Readings stored for each of these finished days, in one query: None for a day that hasn't been fetched.""" if not days: return {} with self._lock: @@ -48,7 +42,8 @@ def counts(self, days: list[date]) -> dict[date, int | None]: (self.loc, min(days).isoformat(), max(days).isoformat())).fetchall()) return {d: found.get(d.isoformat()) for d in days} - def save_day(self, day: date, rows: list[dict]): + def save_day(self, day: date, rows: list[dict]) -> int: + """Store one day's readings, in place of any it had; returns how many it now holds.""" lo, hi = day_bounds(day, self.tz) rows = [r for r in rows if lo <= r["ts"] < hi] with self._lock: @@ -58,6 +53,7 @@ def save_day(self, day: date, rows: list[dict]): [(self.loc, r["ts"], *(r.get(c) for c in COLUMNS)) for r in rows]) self.db.execute("INSERT OR REPLACE INTO days VALUES (?, ?, ?)", (self.loc, day.isoformat(), len(rows))) self.db.commit() + return len(rows) def load(self, start_ts: int, end_ts: int) -> list[dict]: """Readings with start_ts <= ts < end_ts, oldest first.""" diff --git a/lib/alerts/__init__.py b/lib/alerts/__init__.py index c8405d4..02dc94a 100644 --- a/lib/alerts/__init__.py +++ b/lib/alerts/__init__.py @@ -1,8 +1,8 @@ from .particulates import CHECK_SECONDS as PARTICULATES_CHECK_SECONDS, ParticulatesMonitor from .forecast import ForecastMonitor from .irrigation import IrrigationMonitor -from .notify import AlertState, Notifier, with_footer +from .notify import AlertState, Notifier from .pollen_and_asthma import PollenAndAsthmaMonitor from .weather import WeatherMonitor -__all__ = ["PARTICULATES_CHECK_SECONDS", "ParticulatesMonitor", "AlertState", "ForecastMonitor", "IrrigationMonitor", "Notifier", "PollenAndAsthmaMonitor", "WeatherMonitor", "with_footer"] +__all__ = ["PARTICULATES_CHECK_SECONDS", "ParticulatesMonitor", "AlertState", "ForecastMonitor", "IrrigationMonitor", "Notifier", "PollenAndAsthmaMonitor", "WeatherMonitor"] diff --git a/lib/alerts/confirm.py b/lib/alerts/confirm.py index 664e860..336df57 100644 --- a/lib/alerts/confirm.py +++ b/lib/alerts/confirm.py @@ -3,7 +3,9 @@ counted (the same or an older time) is ignored; a reading that does not meet the condition ends the run; a reading with no value is skipped (it neither extends nor ends the run). The run lives in the monitor's saved dict, so it carries across checks and restarts. -(Rain and the temperature crossing use the back-off rule in backoff.py instead: a change that must stand for a cooldown.)""" +(Rain and the temperature crossing use the back-off rule in backoff.py instead: a change that must stand for a cooldown.) + +Once an alert (gusts, UV) has fired it waits to re-arm: rearm() lets it fire again after its condition has stopped holding for a while.""" CONFIRM_READINGS = 2 @@ -21,3 +23,14 @@ def run(saved: dict, key: str, hit: bool | None) -> int: if hit is not None: saved[key] = saved.get(key, 0) + 1 if hit else 0 return saved.get(key, 0) + + +def rearm(saved: dict, hit: bool, now: float, after: float): + """The alert in `saved` (saved["active"]: it has fired) can fire again once its condition has not held (`hit`) for `after` + seconds; while it holds, the calm hasn't begun.""" + if hit: + saved["calm_since"] = None + elif saved["active"]: + saved["calm_since"] = saved["calm_since"] or now + if now - saved["calm_since"] >= after: + saved.update(active=False, calm_since=None) diff --git a/lib/alerts/forecast.py b/lib/alerts/forecast.py index 958f6a7..8228780 100644 --- a/lib/alerts/forecast.py +++ b/lib/alerts/forecast.py @@ -13,6 +13,7 @@ import logging from ..forecast.source import day_label, describe_day +from .menu import kinds_of log = logging.getLogger(__name__) @@ -33,10 +34,10 @@ def revised(old: dict, new: dict) -> bool: class ForecastMonitor: - KINDS = ("forecast",) + KINDS = kinds_of("Forecast") def __init__(self, forecast, state, notify): - """notify: the Notifier (its to_chat sends to one chat).""" + """notify: the Notifier (chats=[chat] sends to that one).""" self.forecast, self.state, self.notify = forecast, state, notify async def check(self): @@ -53,5 +54,5 @@ async def check(self): if not changes: continue lines = [f"• {day_label(day, today)}: was {describe_day(old)}\n now {describe_day(new)}" for day, old, new in changes] - if await self.notify.to_chat(chat_id, "\U0001f504 The forecast has changed since I sent it:\n" + "\n".join(lines), kind="forecast"): - self.state.record_forecast(chat_id, [new for _, _, new in changes]) + if await self.notify("\U0001f504 The forecast has changed since I sent it:\n" + "\n".join(lines), kind="forecast", chats=[chat_id]): + self.state.record_forecast(chat_id, [new for _, _, new in changes], today) diff --git a/lib/alerts/irrigation.py b/lib/alerts/irrigation.py index ac3237c..cf59dbf 100644 --- a/lib/alerts/irrigation.py +++ b/lib/alerts/irrigation.py @@ -2,8 +2,9 @@ - a battery below 5% is told to the chats subscribed to "irrigation battery" (kind irrigation_battery) (at most once a day, until it is charged); - if more than 1 mm of rain fell in the last 24 hours, or Open-Meteo gives today or tomorrow a 50% or better chance of at least - 1 mm of rain, the controller is put on a 24 hour weather delay (it skips its own schedule) and the chats subscribed to "irrigation - pause" are told why. A longer delay already set is left alone (nothing changes, so nothing is sent). + 1 mm of rain (in a forecast fetched within the last day: an older one doesn't count), the controller is put on a 24 hour weather + delay (it skips its own schedule) and the chats subscribed to "irrigation pause" are told why. A delay someone set is left alone + (nothing changes, so nothing is sent); the bot's own from yesterday, seconds from lapsing at the check, is started again. The bot has no scheduler, so `run` sleeps until the next check time. If the controller can't be reached it tries again every 30 minutes until it works or the day ends. The date of the last completed check is saved, so a restart neither repeats a day nor skips one. @@ -14,8 +15,10 @@ from datetime import datetime, timedelta from ..ecowitt.rain import rain_amount -from ..timeutil import now_local +from ..irrigation.tuya import is_paused +from ..timeutil import local_epoch, now_local, seconds_until, when from .forecast import RAIN_FROM_PCT +from .menu import kinds_of log = logging.getLogger(__name__) @@ -23,10 +26,11 @@ MEASURED_MM = 1.0 # more than this in the last 24 hours counts as rain RETRY_SECONDS = 30 * 60 MEASURED_HOURS = 24 +FORECAST_TRUSTED_SECONDS = 24 * 3600 # an older forecast (Open-Meteo unreachable for a day) doesn't decide class IrrigationMonitor: - KINDS = ("irrigation_battery", "irrigation_pause") + KINDS = kinds_of("Irrigation") def __init__(self, device, state, notify, tz, hour: int = 18, station=None, forecast=None): """station: the Ecowitt source (its recent() gives the measured rain); forecast: the Forecast source. Either may be None.""" @@ -41,10 +45,7 @@ def due(self, now: datetime) -> bool: return now.hour >= self.hour and self._memory().get("checked") != now.date().isoformat() def seconds_until_next(self, now: datetime) -> float: - target = now.replace(hour=self.hour, minute=0, second=0, microsecond=0) - if target <= now: - target += timedelta(days=1) - return max(1.0, (target - now).total_seconds()) + return max(1.0, seconds_until(self.hour, 0, self.tz, now)) async def run(self): """Forever: check when due, else sleep until the next check time.""" @@ -53,8 +54,6 @@ async def run(self): if self.due(now): try: await self.check(now) - except asyncio.CancelledError: - raise except Exception as e: log.warning("Irrigation check failed (%s: %s); trying again in %d minutes", type(e).__name__, e, RETRY_SECONDS // 60) await asyncio.sleep(min(RETRY_SECONDS, self.seconds_until_next(now))) @@ -74,13 +73,17 @@ async def check(self, now: datetime | None = None): self.state.save() await self.notify(f"🪫 The irrigation battery is at {battery}%. Replace or recharge it soon, or watering will stop.", kind="irrigation_battery") - reason = await self.rain_reason() + reason = await self.rain_reason(now) if reason: current = status.get("weather_delay") - if current in (None, "cancel"): + own = current == "24h" and memory.get("paused") == (now.date() - timedelta(days=1)).isoformat() + if not is_paused(status) or own: + if own: # the bot's pause from yesterday: ended first, so the new one runs a full day + await self.device.delay("cancel") await self.device.delay("24h") - log.info("Irrigation: delayed 24 hours (%s)", reason) - await self.notify(f"☔ Irrigation paused for 24 hours: {reason}.", kind="irrigation_pause") + memory["paused"] = today + log.info("Irrigation: delayed 24 hours%s (%s)", " more" if own else "", reason) + await self.notify(f"☔ Irrigation paused for {'another ' if own else ''}24 hours: {reason}.", kind="irrigation_pause") else: log.info("Irrigation: %s, but the delay is already %s", reason, current) else: @@ -88,8 +91,8 @@ async def check(self, now: datetime | None = None): memory["checked"] = today self.state.save() - async def rain_reason(self) -> str | None: - """Why the irrigation should be delayed (what rained or will), or None.""" + async def rain_reason(self, now: datetime) -> str | None: + """Why the irrigation should be delayed (what rained or will), or None. `now` is local.""" if self.station is not None: try: rows = await self.station.recent(MEASURED_HOURS) @@ -99,7 +102,11 @@ async def rain_reason(self) -> str | None: fell = None if fell is not None and fell > MEASURED_MM: return f"{fell:g} mm of rain in the last 24 hours" - if self.forecast is not None: + if self.forecast is not None and self.forecast.days: + if local_epoch(now, self.tz) - self.forecast.fetched_at > FORECAST_TRUSTED_SECONDS: + log.info("Irrigation: the forecast is from %s, over a day old: only the measured rain decides", + when(int(self.forecast.fetched_at), self.tz)) + return None today = self.forecast.now().date() for day in self.forecast.upcoming(2): chance = day.get("rain_1mm_pct") diff --git a/lib/alerts/menu.py b/lib/alerts/menu.py index 5614a1c..50be8a7 100644 --- a/lib/alerts/menu.py +++ b/lib/alerts/menu.py @@ -18,10 +18,12 @@ from ..buttons import opener, toggle_button -# kind -> the name on the button, in the order listed -LABELS = {"rain": "rain", "gusts": "gusts", "uv": "UV", "temps": "temperature crossing", - "particulates": "particulates", "pollen_and_asthma": "pollen & asthma", "forecast": "forecast changes", - "irrigation_battery": "irrigation battery", "irrigation_pause": "irrigation pause"} +# kind -> (the name on its button, the source that sends it: a source's name, or Irrigation for the controller), in the order listed +TYPES = {"rain": ("rain", "Ecowitt"), "gusts": ("gusts", "Ecowitt"), "uv": ("UV", "Ecowitt"), "temps": ("temperature crossing", "Ecowitt"), + "particulates": ("particulates", "AirGradient"), "pollen_and_asthma": ("pollen & asthma", "Pollen"), + "forecast": ("forecast changes", "Forecast"), + "irrigation_battery": ("irrigation battery", "Irrigation"), "irrigation_pause": ("irrigation pause", "Irrigation")} +LABELS = {kind: label for kind, (label, _) in TYPES.items()} ALL = "all" TITLE = "🔔 Alerts in this chat" PER_ROW = 2 @@ -29,17 +31,15 @@ MENUS = {ALERTS: "🔔 Alerts", SOUND: "🔊 Sound"} # the /alerts buttons -def available_kinds(sources: set[str]) -> list[str]: - """The alert types the bot can send, from the names of its sources.""" - wanted = (["rain", "gusts", "uv", "temps"] if "Ecowitt" in sources else []) + ( - ["particulates"] if "AirGradient" in sources else []) + (["pollen_and_asthma"] if "Pollen" in sources else []) + ( - ["forecast"] if "Forecast" in sources else []) + (["irrigation_battery", "irrigation_pause"] if "Irrigation" in sources else []) - return wanted +def kinds_of(source: str) -> tuple[str, ...]: + """The alert types a source (by name) sends: a monitor's KINDS.""" + return tuple(kind for kind, (_, sender) in TYPES.items() if sender == source) -def kinds_for(source_names, irrigation: bool) -> list[str]: - """The alert types the bot sends, from the names of its sources and whether an irrigation controller is configured.""" - return available_kinds(set(source_names) | ({"Irrigation"} if irrigation else set())) +def available_kinds(sources, irrigation: bool = False) -> list[str]: + """The alert types the bot sends: those of its sources (by name), and the irrigation ones with a controller configured.""" + names = {*sources, *(["Irrigation"] if irrigation else [])} + return [kind for kind, (_, source) in TYPES.items() if source in names] @dataclass(frozen=True) diff --git a/lib/alerts/notify.py b/lib/alerts/notify.py index 712339b..cc769f2 100644 --- a/lib/alerts/notify.py +++ b/lib/alerts/notify.py @@ -1,78 +1,110 @@ """Who gets alerts, and sending them. Alerts go to every chat the bot knows about (Telegram can't list a bot's chats, so they are -remembered as messages arrive) unless it unsubscribed (per alert type, with the buttons under each alert or /alerts). Chats and +remembered as messages arrive) unless it unsubscribed (per alert type, with the buttons under each alert or /alerts). A chat that +blocked or removed the bot is marked gone: it gets nothing, and has its settings back when it next writes. Chats and monitor state live in a small JSON file, so restarts neither forget chats nor repeat alerts. Alerts are sent silently, with the settings buttons and, when there is one, a small italic footer with a link label. """ +import asyncio import json import logging import os import time -from datetime import date +from collections.abc import Awaitable, Callable, Collection +from datetime import date, timedelta -from telegram import MessageEntity -from telegram.error import BadRequest, ChatMigrated, Forbidden, TelegramError +from telegram.error import BadRequest, ChatMigrated, Forbidden, NetworkError, RetryAfter, TelegramError, TimedOut +from ..entities import with_link from ..logs import QUOTES from .menu import ALL, LABELS, alert_keyboard log = logging.getLogger(__name__) +RETRY_SECONDS = 3 # before sending an alert again after a dropped connection +FLOOD_WAIT_MAX = 60 # Telegram's flood limit asks to wait: a longer wait is not waited out (that chat misses the alert) + + +def _load(path) -> dict: + """The saved state, {} when there is none. A file that can't be read is moved aside, as the next save would replace it, + so its chats and settings can still be recovered by hand.""" + try: + with open(path) as f: + return json.load(f) + except FileNotFoundError: + return {} + except (OSError, ValueError) as e: + aside = f"{path}.damaged-{time.strftime('%Y%m%d-%H%M%S')}" + os.replace(path, aside) + log.error("Couldn't read %s (%s): moved it to %s and starting with no known chats", path, e, aside) + return {} + + +def _flood_wait(e: RetryAfter) -> float: + """The seconds Telegram asks to wait (python-telegram-bot gives a number, or a timedelta once PTB_TIMEDELTA is set).""" + wait = e.retry_after + return wait.total_seconds() if isinstance(wait, timedelta) else wait + class AlertState: """Known chats (with alerts on/off) and monitor state, saved to a JSON file.""" def __init__(self, path): self.path = path - data: dict = {} - try: - with open(path) as f: - data = json.load(f) - except FileNotFoundError: - pass - except Exception: - log.exception("Couldn't read %s - starting with no known chats", path) + data = _load(path) self.chats: dict[int, dict] = {int(k): v for k, v in data.get("chats", {}).items()} - for entry in self.chats.values(): # the old on/off flag, and a chat with every type muted, are "unsubscribed from all" - if entry.pop("alerts", True) is False or set(entry.get("muted", [])) >= set(LABELS): - entry.pop("muted", None) - entry.update(optin=True, on=[]) self.monitor: dict = data.get("monitor", {}) self.charts: dict[str, str] = data.get("charts", {}) # the question behind each chart sent, for its period buttons def save(self): + """Written in full to a new file that then replaces the old one, on disk first, so a crash leaves one or the other whole.""" tmp = f"{self.path}.tmp" with open(tmp, "w") as f: json.dump({"chats": {str(k): v for k, v in self.chats.items()}, "monitor": self.monitor, "charts": self.charts}, f, indent=1) + f.flush() + os.fsync(f.fileno()) os.replace(tmp, self.path) def add_chat(self, chat_id: int, title: str): + """A message from this chat (or the bot added to it): known from now on, and back if it was gone.""" entry = self.chats.get(chat_id) if entry is None: self.chats[chat_id] = {"title": title} log.info("Alerts: now sending to %s (id %s)", title, chat_id) self.save() - elif entry.get("title") != title: + elif entry.get("title") != title or entry.get("gone"): + if entry.pop("gone", None): + log.info("Alerts: sending to %s (id %s) again", title, chat_id) entry["title"] = title self.save() - def remove_chat(self, chat_id: int, reason: str): - if self.chats.pop(chat_id, None) is not None: + def mark_gone(self, chat_id: int, reason: str): + """The bot can't post to this chat any more (it was blocked or removed): nothing is sent to it, and its settings are kept + for when it writes again.""" + entry = self.chats.get(chat_id) + if entry is not None and not entry.get("gone"): + entry["gone"] = True log.info("Alerts: no longer sending to chat %s (%s)", chat_id, reason) self.save() - def set_alerts(self, chat_id: int, title: str, on: bool): + def present(self) -> list[int]: + """The chats the bot is in: known, and not gone.""" + return [c for c, entry in self.chats.items() if not entry.get("gone")] + + def set_alerts(self, chat_id: int, on: bool): """/alerts on and /alerts off: every type.""" - self.chats.setdefault(chat_id, {"title": title}) self.set_kind(chat_id, ALL, on) def migrate_chat(self, old: int, new: int): - """A group became a supergroup: its settings move to the new chat id.""" + """A group became a supergroup: its settings, and the forecast it was sent, move to the new chat id.""" entry = self.chats.pop(old, None) if entry is not None: self.chats.setdefault(new, entry) + sent = self.monitor.get("forecast_sent", {}) + if str(old) in sent: + sent.setdefault(str(new), sent.pop(str(old))) log.info("Alerts: chat %s became %s", old, new) self.save() @@ -84,9 +116,10 @@ def muted(self, chat_id: int) -> set[str]: return set(LABELS) - set(entry.get("on", [])) return set(entry.get("muted", [])) - def set_kind(self, chat_id: int, kind: str, on: bool): + def set_kind(self, chat_id: int, kind: str, on: bool, offered: Collection[str] = LABELS): """Subscribe (on) or unsubscribe one alert type, or all of them (kind = "all"). Unsubscribing from all also keeps - future types off for the chat; subscribing to all turns them on again.""" + future types off for the chat, and so does unsubscribing one by one from every type `offered` (the types the bot + sends); subscribing to all turns them on again.""" if chat_id not in self.chats: return entry = self.chats[chat_id] @@ -105,10 +138,13 @@ def set_kind(self, chat_id: int, kind: str, on: bool): return muted = self.muted(chat_id) muted = muted - kinds if on else muted | kinds - if muted: - self.chats[chat_id]["muted"] = [k for k in LABELS if k in muted] + if not on and muted >= set(offered): # the last type muted: the same as unsubscribing from all + entry.pop("muted", None) + entry.update(optin=True, on=[]) + elif muted: + entry["muted"] = [k for k in LABELS if k in muted] else: - self.chats[chat_id].pop("muted", None) + entry.pop("muted", None) self.save() def loud(self, chat_id: int) -> set[str]: @@ -136,14 +172,13 @@ def set_keyboard(self, chat_id: int, version: str): self.chats[chat_id]["keyboard"] = version self.save() - def record_forecast(self, chat_id: int, days: list[dict]): - """Remember the forecast days this chat was just sent (a newer send replaces the baseline for the same day); days - already past are dropped. days is not empty.""" + def record_forecast(self, chat_id: int, days: list[dict], today: date): + """Remember the forecast days this chat was just sent (a newer send replaces the baseline for the same day, the other + days keep theirs); days before today are dropped.""" sent = self.monitor.setdefault("forecast_sent", {}).setdefault(str(chat_id), {}) for d in days: sent[d["date"].isoformat()] = {**d, "date": d["date"].isoformat()} - first = min(d["date"] for d in days).isoformat() # what was sent starts today: older days are past - for day in [k for k in sent if k < first]: + for day in [k for k in sent if k < today.isoformat()]: del sent[day] self.save() @@ -157,66 +192,37 @@ def forecast_chats(self) -> list[int]: def subscribed(self, chat_id: int, kind: str | None = None) -> bool: """Does this chat get this alert type (or, with no kind, any at all)?""" - if chat_id not in self.chats: + if chat_id not in self.chats or self.chats[chat_id].get("gone"): return False muted = self.muted(chat_id) return kind not in muted if kind else bool(set(LABELS) - muted) - def alerts_on(self, chat_id: int) -> bool: - return self.subscribed(chat_id) - def alert_chats(self, kind: str | None = None) -> list[int]: """The chats that get alerts of this type (any type, if none is given).""" return [c for c in self.chats if self.subscribed(c, kind)] -def _utf16_len(text: str) -> int: - """Telegram measures entity positions in UTF-16 code units (emoji count as 2).""" - return len(text.encode("utf-16-le")) // 2 - - -def with_footer(text: str, link: tuple[str, str] | None = None, extra: str | None = None) -> tuple[str, list]: - """Add a small italic footer line directly under the text: the link's label (clickable), - then any extra text, joined by " · ". Plain text stays plain (no HTML to escape); - only the footer is formatted. link = (label, url).""" - text = text.rstrip() - parts = ([link[0]] if link else []) + ([extra] if extra else []) - if not parts: - return text, [] - start = _utf16_len(text) + 1 - footer = " · ".join(parts) - entities = [MessageEntity(MessageEntity.ITALIC, start, _utf16_len(footer))] - if link: - entities.append(MessageEntity(MessageEntity.TEXT_LINK, start, _utf16_len(link[0]), url=link[1])) - return f"{text}\n{footer}", entities - - class Notifier: def __init__(self, bot, state: AlertState): self.bot, self.state = bot, state - async def __call__(self, text: str, link: tuple[str, str] | None = None, kind: str | None = None, photo: bytes | str | None = None): - """Send an alert (silently) to every chat subscribed to its type; forget chats the bot can no - longer post to. link = (label, url) adds a clickable label to the footer. Each carries the settings buttons. - photo: a picture the text is the caption of (the text alone if Telegram refuses the picture): a PNG, or the file id of one - Telegram holds. It is uploaded once: the chats after the first are sent Telegram's copy.""" - text, entities = with_footer(text, link) + async def __call__(self, text: str, link: tuple[str, str] | None = None, kind: str | None = None, + chart: Callable[[], Awaitable[bytes | None]] | None = None, chats: list[int] | None = None) -> int: + """Send an alert (silently) to the chats subscribed to its type: of `chats` (say, the one sent something that has since + changed), else of every chat. Chats the bot can no longer post to are marked gone. Returns how many it was sent to. + link = (label, url) adds a clickable label to the footer; each carries the settings buttons. chart draws the alert's + picture (None: the text alone), and is called only when some chat gets the alert. The text is the picture's caption + (the text alone if Telegram refuses it); it is uploaded once and the chats after the first are sent Telegram's copy.""" + targets = self.state.alert_chats(kind) if chats is None else [c for c in chats if self.state.subscribed(c, kind)] + text, entities = with_link(text, link) + photo = await chart() if chart and targets else None started, sent = time.monotonic(), 0 - for chat_id in self.state.alert_chats(kind): + for chat_id in targets: ok, message = await self._send(chat_id, text, entities, kind, photo) sent += ok if photo and (sizes := getattr(message, "photo", None)): photo = sizes[-1].file_id # the largest size: the picture as sent log.info("Alert sent to %d chat(s) in %.1fs: %s", sent, time.monotonic() - started, text, extra=QUOTES) - - async def to_chat(self, chat_id: int, text: str, link: tuple[str, str] | None = None, kind: str | None = None) -> bool: - """An alert for one chat (the one that was sent something that has since changed), if it is subscribed to the type. - True if it was sent.""" - if not self.state.subscribed(chat_id, kind): - return False - text, entities = with_footer(text, link) - sent, _ = await self._send(chat_id, text, entities, kind) - log.info("Alert %s chat %s: %s", "sent to" if sent else "NOT sent to", chat_id, text, extra=QUOTES) return sent async def _post(self, chat_id: int, text: str, entities, kind: str | None, photo: bytes | str | None): @@ -227,25 +233,35 @@ async def _post(self, chat_id: int, text: str, entities, kind: str | None, photo try: return await self.bot.send_photo(chat_id, photo, caption=text, caption_entities=entities, **common) except BadRequest as e: - if "not found" in str(e).lower(): + if "chat not found" in str(e).lower(): raise log.warning("Alert picture to %s failed (%s); sending the text alone", chat_id, e) return await self.bot.send_message(chat_id, text, entities=entities, disable_web_page_preview=True, **common) - async def _send(self, chat_id: int, text: str, entities, kind: str | None = None, photo: bytes | str | None = None) -> tuple[bool, object]: - """Send an alert (see _post); forget a chat the bot can no longer post to. (whether it was sent, the message sent)""" + async def _send(self, chat_id: int, text: str, entities, kind: str | None = None, photo: bytes | str | None = None, + again: bool = True) -> tuple[bool, object]: + """Send an alert (see _post) to one chat; a chat the bot can no longer post to is marked gone. Telegram's flood limit (when it asks + for a short wait) and a dropped connection get one more try; a timeout doesn't, as the message may have arrived. Whatever + fails for one chat is logged, so the others still get the alert. (whether it was sent, the message sent)""" try: - return True, await self._post(chat_id, text, entities, kind, photo) - except ChatMigrated as e: # the group became a supergroup: follow it, and send again there - self.state.migrate_chat(chat_id, e.new_chat_id) - return await self._send(e.new_chat_id, text, entities, kind, photo) if e.new_chat_id != chat_id else (False, None) - except Forbidden as e: # kicked from the group, or blocked in a private chat - self.state.remove_chat(chat_id, f"can't post: {e}") - except BadRequest as e: - if "not found" in str(e).lower(): - self.state.remove_chat(chat_id, f"chat gone: {e}") - else: - log.warning("Alert to %s failed: %s", chat_id, e) - except TelegramError as e: - log.warning("Alert to %s failed: %s", chat_id, e) + try: + return True, await self._post(chat_id, text, entities, kind, photo) + except ChatMigrated as e: # the group became a supergroup: follow it, and send again there + self.state.migrate_chat(chat_id, e.new_chat_id) + return await self._send(e.new_chat_id, text, entities, kind, photo) if e.new_chat_id != chat_id else (False, None) + except Forbidden as e: # kicked from the group, or blocked in a private chat + self.state.mark_gone(chat_id, f"can't post: {e}") + except BadRequest as e: + if "chat not found" not in str(e).lower(): # not "message thread not found", say: that chat is still there + raise + self.state.mark_gone(chat_id, f"chat gone: {e}") + except (RetryAfter, NetworkError) as e: # BadRequest, a NetworkError too, is caught above + wait = _flood_wait(e) if isinstance(e, RetryAfter) else RETRY_SECONDS + if not again or isinstance(e, TimedOut) or wait > FLOOD_WAIT_MAX: + raise + log.info("Alert to %s: %s; trying again in %gs", chat_id, e, wait) + await asyncio.sleep(wait) + return await self._send(chat_id, text, entities, kind, photo, again=False) + except Exception as e: # Telegram refused it, or the state file couldn't be saved + log.warning("Alert to %s failed: %s", chat_id, e, exc_info=not isinstance(e, TelegramError)) return False, None diff --git a/lib/alerts/particulates.py b/lib/alerts/particulates.py index e75f7f7..67c704d 100644 --- a/lib/alerts/particulates.py +++ b/lib/alerts/particulates.py @@ -11,6 +11,7 @@ from datetime import datetime from . import confirm, pictures +from .menu import kinds_of log = logging.getLogger(__name__) @@ -21,16 +22,16 @@ class ParticulatesMonitor: - KINDS = ("particulates",) + KINDS = kinds_of("AirGradient") def __init__(self, air, state, notify): self.air, self.state, self.notify = air, state, notify - async def check(self, reading: dict | None = None, now: datetime | None = None): - reading = reading if reading is not None else await self.air.current() + async def check(self, latest=None, now: datetime | None = None): + """latest: the sensor's latest reading (AirGradient.latest(): the reading, when it was taken, what it has been sustained at).""" + reading, when, held = latest if latest is not None else await self.air.latest() now = now or datetime.now(self.air.tz) m = self.state.monitor.setdefault("particulates", {"unsafe": False, "above": 0, "below": 0}) - when = reading.get("_time_utc") if when and (now - when).total_seconds() > STALE_SECONDS: log.info("Particulates: latest reading is stale (%s); skipping", when) return @@ -40,7 +41,7 @@ async def check(self, reading: dict | None = None, now: datetime | None = None): return if when is not None and not confirm.new_reading(m, when.timestamp()): # the sensor has said nothing new since the last check return - held = reading.get("_sustained") or {} # the lowest of the last readings (ANTISPIKE_REQUIRE_CONSECUTIVE): a spike alone can't make the air unhealthy + # held: the lowest of the last readings (ANTISPIKE_REQUIRE_CONSECUTIVE), so a spike alone can't make the air unhealthy high25, high10 = held.get("pm2_5", pm25), held.get("pm10", pm10) bad = (high25 is not None and high25 >= MASK_PM25) or (high10 is not None and high10 >= MASK_PM10) good = (pm25 is None or pm25 <= SAFE_PM25) and (pm10 is None or pm10 <= SAFE_PM10) @@ -48,14 +49,12 @@ async def check(self, reading: dict | None = None, now: datetime | None = None): confirm.run(m, "below", good) if not m["unsafe"] and m["above"] >= confirm.CONFIRM_READINGS: m["unsafe"] = True - photo = await self._picture() await self.notify("\U0001f637 Unhealthy air outside — wear a P2/N95 mask today. " + self._levels(reading), - link=self.air.link, kind="particulates", **({"photo": photo} if photo else {})) + link=self.air.link, kind="particulates", chart=self._picture) elif m["unsafe"] and m["below"] >= confirm.CONFIRM_READINGS: m["unsafe"] = False - photo = await self._picture() await self.notify("✅ Outdoor air is safe again. " + self._levels(reading), link=self.air.link, kind="particulates", - **({"photo": photo} if photo else {})) + chart=self._picture) self.state.save() async def _picture(self) -> bytes | None: diff --git a/lib/alerts/pictures.py b/lib/alerts/pictures.py index c745467..b24db9d 100644 --- a/lib/alerts/pictures.py +++ b/lib/alerts/pictures.py @@ -16,9 +16,9 @@ async def chart(source, text: str, tz) -> bytes | None: """The PNG for the request `text`, made from `source`'s tool the way the bot would answer it; None when it cannot be made (the alert then goes out as text). `source` is the station or the air sensor: the tool is found in its `tools` by the name the request resolves to.""" try: - read = intent.read(text, now_local(tz), True, True) + read = intent.read(text, now_local(tz)) tool, args = read.fast[0], read.fast[1] - turn = Turn(chart_asked=read.chart_asked, chart_field=read.chart_field, chart_fields=read.chart_fields, text=text) + turn = Turn.of(read, text) handler = next(t for t in source.tools if t.name == tool).handler await handler(args, turn) return await asyncio.to_thread(render, turn.charts[0], tz) diff --git a/lib/alerts/pollen_and_asthma.py b/lib/alerts/pollen_and_asthma.py index ab5f81a..eb62d11 100644 --- a/lib/alerts/pollen_and_asthma.py +++ b/lib/alerts/pollen_and_asthma.py @@ -10,6 +10,8 @@ import time from ..pollen.parse import LEVEL_EMOJI, LEVELS +from ..timeutil import day_text +from .menu import kinds_of log = logging.getLogger(__name__) @@ -21,7 +23,7 @@ class PollenAndAsthmaMonitor: - KINDS = ("pollen_and_asthma",) + KINDS = kinds_of("Pollen") def __init__(self, pollen, state, notify): self.pollen, self.state, self.notify = pollen, state, notify @@ -38,14 +40,12 @@ async def check(self): reading = cur[key] if reading is None: # not on the page (the asthma forecast only runs 1 Oct - 31 Dec): keep what we know continue - rank, day = LEVELS.index(reading["level"]), str(today) - if rank < WARN_FROM: + if LEVELS.index(reading["level"]) < WARN_FROM: m.pop(key, None) continue - last = m.get(key) - if last is None or last["rank"] < rank or last["day"] != day: - m[key] = {"rank": rank, "day": day} + if (m.get(key) or {}).get("day") != str(today): # once a day while it stays High (the top of the scale) + m[key] = {"day": str(today)} for_day = reading.get("date") - when = f" (forecast for {for_day:%a %d %b})" if for_day and for_day != today else "" + when = f" (forecast for {day_text(for_day, year=False)})" if for_day and for_day != today else "" await self.notify(f"{LEVEL_EMOJI[reading['level']]} {LABELS[key]} is {reading['level']}{when}.{ADVICE[key]}", link=LINK, kind="pollen_and_asthma") self.state.save() diff --git a/lib/alerts/weather.py b/lib/alerts/weather.py index 64fa084..4624957 100644 --- a/lib/alerts/weather.py +++ b/lib/alerts/weather.py @@ -1,9 +1,9 @@ """Weather alerts from the Ecowitt station, checked after every keep-warm refresh (from the same 5-minute readings; the rain check also reads the gauge's live value, one small request). - - Rain: "started" (one tip of the gauge is enough) and "stopped" (dry for ALERT_COOLDOWN_MINUTES), with how much fell. - Not during quiet hours (RAIN_QUIET_HOURS, default midnight to 6am): those are kept quiet and one summary of the overnight - rain is sent after they end. + - Rain: "started" (one tip of the gauge is enough) and "stopped" (dry for ALERT_COOLDOWN_MINUTES), with how much fell over + the whole spell. Not during quiet hours (RAIN_QUIET_HOURS, default midnight to 6am, and they may run across midnight): + those are kept quiet and one summary of the rain that fell in them is sent after they end. - Strong gusts: one alert when a gust goes over 40 km/h, and no more until the gusts have stayed at or under it for an hour, so a blustery afternoon is one message, not twenty. - Strong sun: one alert when the UV index is 10 or more in two readings in a row (confirm.py), and no more until it has stayed @@ -19,9 +19,10 @@ from ..ecowitt.glance import UVI_ALERT from . import confirm, pictures +from .menu import kinds_of from .backoff import due, remember from ..ecowitt.rain import Rows, duration, rain_amount, wet_flags -from ..timeutil import to_local +from ..timeutil import at_hour, clock_text, to_local log = logging.getLogger(__name__) @@ -41,7 +42,7 @@ def side(r: dict) -> str | None: class WeatherMonitor: - KINDS = ("rain", "gusts", "uv", "temps") + KINDS = kinds_of("Ecowitt") def __init__(self, station, state, notify, cooldown_seconds: int = ALERT_COOLDOWN_SECONDS, quiet: tuple[int, int] | None = None): self.station, self.state, self.notify = station, state, notify @@ -55,6 +56,12 @@ def _quiet(self, ts: int) -> bool: hour = to_local(ts, self.tz).hour return start <= hour < end if start < end else hour >= start or hour < end + def _quiet_began(self, ts: int) -> int: + """When the quiet hours that `ts` falls in began (epoch): the evening before, for hours that run across midnight.""" + local = to_local(ts, self.tz) + day = local.date() - timedelta(days=1 if local.hour < self.quiet[0] else 0) + return at_hour(day, self.quiet[0], self.tz) + @property def tz(self): return self.station.tz @@ -94,14 +101,12 @@ async def check(self): async def _rain(self, rows: Rows): m = self.state.monitor.setdefault("rain", {}) - if "raining" in m: # saved by an earlier version - m["alerted_state"] = "raining" if m.pop("raining") else "dry" m.setdefault("alerted_state", "dry") wet = wet_flags(rows) latest_ts = wet[-1][0] quiet = self._quiet(latest_ts) if quiet: - self._note_night(rows, wet) + self._note_night(wet) wet_times = [ts for ts, w, _ in wet if w] last_wet = wet_times[-1] if wet_times else None raining = last_wet is not None and latest_ts - last_wet < self.cooldown # dry means no rain for the whole cooldown @@ -121,27 +126,23 @@ async def _rain(self, rows: Rows): log.info("Alerts: rain stopped in quiet hours; the morning summary covers it") else: since = m.get("since") or wet_times[0] - fell = rain_amount(rows, since, last_wet) + fell = await self._fell(rows, since, last_wet) amount = f"{fell:.1f} mm fell" if fell else "Only a trace fell" took = duration(last_wet + 300 - since) await self.notify(f"\U0001f324️ The rain has stopped. {amount} over {took}.", kind="rain") m["since"] = None remember(m, state, latest_ts, alerted=not quiet) - def _note_night(self, rows: Rows, wet: list): - """In quiet hours: when it first and last rained since midnight, and the day's total so far (the gauge's running total).""" - latest = rows[-1][0] - today = to_local(latest, self.tz).date() + def _note_night(self, wet: list): + """In quiet hours: when it first and last rained since they began. The note lives until the summary after them takes it.""" + began = self._quiet_began(wet[-1][0]) night = self.state.monitor.get("night") - if not night or night["date"] != today.isoformat(): - night = self.state.monitor["night"] = {"date": today.isoformat(), "first": None, "last": None, "mm": 0.0} + if not night or night.get("began") != began: # a new night (a note left from an earlier one is replaced) + night = self.state.monitor["night"] = {"began": began, "first": None, "last": None} for ts, w, _ in wet: - if w and to_local(ts, self.tz).date() == today and self._quiet(ts): + if w and ts >= began: night["first"] = night["first"] or ts night["last"] = ts - daily = rows[-1][1].get("rainfall.daily") - if daily is not None: - night["mm"] = max(night["mm"], daily) async def _night_summary(self, rows: Rows): """After quiet hours end: one message about the rain that fell in them (only if any did).""" @@ -150,10 +151,11 @@ async def _night_summary(self, rows: Rows): if not night or self._quiet(latest): return self.state.monitor.pop("night") - if night["date"] != to_local(latest, self.tz).date().isoformat() or not night["first"]: - return # an old note (the bot was down) or a dry night - clock = lambda ts: f"{to_local(ts, self.tz):%-I:%M%p}".replace("AM", "am").replace("PM", "pm") - amount = f"{night['mm']:.1f} mm" if night["mm"] >= 0.1 else "a trace" + if not night.get("first") or latest - night.get("began", 0) >= 86400: + return # a dry night, or an old note (the bot was down) + clock = lambda ts: clock_text(to_local(ts, self.tz)) + fell = await self._fell(rows, night["first"], night["last"]) + amount = f"{fell:.1f} mm" if fell else "a trace" still = " It's still raining." if self.state.monitor.get("rain", {}).get("alerted_state") == "raining" else "" await self.notify(f"\U0001f327️ Overnight rain: {amount}, from about {clock(night['first'])} to {clock(night['last'] + 300)}.{still}", kind="rain") @@ -163,25 +165,19 @@ async def _gusts(self, rows: Rows): latest_ts = rows[-1][0] seen, m["checked"] = m.get("checked", latest_ts), latest_ts over = [(g, ts) for ts, r in rows if ts > seen and (g := r.get("wind.wind_gust")) is not None and g > GUST_ALERT_KMH] - if over: - m["calm_since"] = None - if not m["active"]: - m["active"] = True - gust, ts = max(over) - photo = await self._picture("Wind chart 1d") - await self.notify(f"\U0001f4a8 Strong gusts: {gust:.0f} km/h at {to_local(ts, self.tz):%-I:%M%p}".replace("AM", "am").replace("PM", "pm") - + f" (alerts above {GUST_ALERT_KMH} km/h).", kind="gusts", **({"photo": photo} if photo else {})) - elif m["active"]: - m["calm_since"] = m["calm_since"] or latest_ts - if latest_ts - m["calm_since"] >= GUST_REARM_SECONDS: - m.update(active=False, calm_since=None) + confirm.rearm(m, bool(over), latest_ts, GUST_REARM_SECONDS) + if over and not m["active"]: + m["active"] = True + gust, ts = max(over) + await self.notify(f"\U0001f4a8 Strong gusts: {gust:.0f} km/h at {clock_text(to_local(ts, self.tz))}" + f" (alerts above {GUST_ALERT_KMH} km/h).", kind="gusts", chart=lambda: self._picture("Wind chart 1d")) async def _uv(self, rows: Rows): """One alert per spell at or above UVI_ALERT, once it holds for two readings in a row (confirm.py); only readings newer than the last one counted are looked at, so a restart doesn't re-announce an old spell.""" m = self.state.monitor.setdefault("uv", {"active": False, "calm_since": None}) latest_ts = rows[-1][0] - m.setdefault("stamp", m.pop("checked", latest_ts)) # the first look only sets the mark + m.setdefault("stamp", latest_ts) # the first look only sets the mark high = False for ts, r in rows: if not confirm.new_reading(m, ts): @@ -192,19 +188,13 @@ async def _uv(self, rows: Rows): if hit and (streak == 1 or u > m["peak"][0]): # the run's highest reading is the one told m["peak"] = [u, ts] high = high or bool(hit) - if high: - m["calm_since"] = None - elif m["active"]: - m["calm_since"] = m["calm_since"] or latest_ts - if latest_ts - m["calm_since"] >= UV_REARM_SECONDS: - m.update(active=False, calm_since=None) + confirm.rearm(m, high, latest_ts, UV_REARM_SECONDS) if m.get("streak", 0) >= confirm.CONFIRM_READINGS and not m["active"]: m["active"] = True uv, ts = m["peak"] - photo = await self._picture("UV chart 1d") - await self.notify(f"\U0001f9f4 UV index {uv:g} at {to_local(ts, self.tz):%-I:%M%p}".replace("AM", "am").replace("PM", "pm") - + f": very high. Sunscreen, a hat and shade (alerts at {UVI_ALERT} and above).", kind="uv", - **({"photo": photo} if photo else {})) + await self.notify(f"\U0001f9f4 UV index {uv:g} at {clock_text(to_local(ts, self.tz))}" + f": very high. Sunscreen, a hat and shade (alerts at {UVI_ALERT} and above).", kind="uv", + chart=lambda: self._picture("UV chart 1d")) async def _cross(self, rows: Rows): ts, r = rows[-1] @@ -215,18 +205,23 @@ async def _cross(self, rows: Rows): if c is None: self.state.monitor["cross"] = {"alerted_state": current, "alerted_time": ts} return - if "side" in c: # saved by an earlier version - c["alerted_state"], c["alerted_time"] = c.pop("side"), c.pop("since") if due(c, current, ts, self.cooldown): o, i = r["outdoor.temperature"], r["indoor.temperature"] held = ts - c["alerted_time"] # how long the side we last told about has stood was = f"{int(held // 86400)} days" if held >= 2 * 86400 else duration(held) icon = "\U0001f321️" if current == "warmer" else "\u2744\ufe0f" # a thermometer for warmer, a snowflake for cooler - photo = await self._picture("Temperature chart 1d") await self.notify(f"{icon} It's now {abs(o - i):.1f}°C {current} outside than inside " - f"({o:.1f}°C vs {i:.1f}°C). It had been {'cooler' if current == 'warmer' else 'warmer'} for {was}.", kind="temps", **({"photo": photo} if photo else {})) + f"({o:.1f}°C vs {i:.1f}°C). It had been {'cooler' if current == 'warmer' else 'warmer'} for {was}.", kind="temps", + chart=lambda: self._picture("Temperature chart 1d")) remember(c, current, ts) + async def _fell(self, rows: Rows, since: int, until: int) -> float | None: + """The rain that fell from `since` to `until`. A spell that began before `rows` (the last 3 hours) is read back to its start: + its first tip shows only against the reading before it.""" + if since <= rows[0][0]: + rows = await self.station.recent((rows[-1][0] - since) / 3600 + 0.25) + return rain_amount(rows, since, until) + async def _picture(self, text: str) -> bytes | None: """The chart a typed `text` ("Wind chart 1d") would give, for an alert; None when it cannot be made.""" return await pictures.chart(self.station, text, self.tz) diff --git a/lib/analysis/pairs.py b/lib/analysis/pairs.py index dc25c28..df59e70 100644 --- a/lib/analysis/pairs.py +++ b/lib/analysis/pairs.py @@ -12,7 +12,7 @@ from ..airgradient.metrics import ZONES, zone from ..rain import SPELL_MM, rain_spells -from ..timeutil import MIN_DAY_SLOTS, SLOT, local_date, to_local +from ..timeutil import clock_text, day_text, local_date, MIN_DAY_SLOTS, SLOT, to_local from .scan import rank LOOKBACK = 6 # slots: the change is measured over the previous 3 hours @@ -94,11 +94,13 @@ def analyse(driver: dict[int, float], rain: dict[int, float], threshold: float, c = change_before(driver, t) if c is not None: groups["falling" if c <= -threshold else "rising" if c >= threshold else "steady"].append(t) + counted = [t for ts in groups.values() for t in ts] # the slots with a change before them: the shares are of these + counted_mm = sum(rain[t] for t in counted) table = {} for name, ts in groups.items(): mm = sum(rain[t] for t in ts) - time_share = len(ts) / max(sum(map(len, groups.values())), 1) - rain_share = mm / total if total else 0.0 + time_share = len(ts) / max(len(counted), 1) + rain_share = mm / counted_mm if counted_mm else 0.0 table[name] = {"slots": len(ts), "share_of_time": f"{100 * time_share:.0f}%", "rain_mm": round(mm, 1), "share_of_rain": f"{100 * rain_share:.0f}%", "rain_vs_fair_share": f"{rain_share / time_share:.1f}x" if time_share else None, @@ -117,7 +119,7 @@ def analyse(driver: dict[int, float], rain: dict[int, float], threshold: float, "lowest": min(driver[t] for t in s if t in driver)} for s in rain_spells(rain, wet)] fell = sum(1 for e in events if e["change"] is not None and e["change"] <= -threshold) rose = sum(1 for e in events if e["change"] is not None and e["change"] >= threshold) - clock = lambda ts: (lambda dt: f"{dt:%a} {dt.day} {dt:%b} {dt.strftime('%-I:%M%p').lower()}")(to_local(ts, tz)) + clock = lambda ts: f"{day_text(to_local(ts, tz), year=False)} {clock_text(to_local(ts, tz))}" summary, call = _summary(table, level, events, threshold, unit, clock) top = sorted(events, key=lambda e: -e["mm"])[:5] return {"slots": len(slots), "rain_mm": round(total, 1), "wet_slots": len(wet), "verdict": call, "findings": summary, "by_change_before": table, @@ -162,8 +164,8 @@ def analyse_air(values: dict[int, float], rain: dict[int, float], limits: tuple[ if len(before) >= MIN_WASH_SLOTS and len(after) >= MIN_WASH_SLOTS: events.append({"start": s[0], "mm": round(sum(rain[t] for t in s), 1), "before": float(np.mean(before)), "after": float(np.mean(after))}) - cleaned = sum(1 for e in events if e["after"] <= CLEANED * e["before"]) - worse = sum(1 for e in events if e["after"] * CLEANED >= e["before"]) + cleaned = sum(1 for e in events if e["after"] < e["before"] and e["after"] <= CLEANED * e["before"]) # unchanged (as zero to + worse = sum(1 for e in events if e["after"] > e["before"] and e["after"] * CLEANED >= e["before"]) # zero) is neither ratio = wet_level / dry_level if wet_level is not None and dry_level else None days: dict = {} for t in slots: @@ -181,7 +183,7 @@ def analyse_air(values: dict[int, float], rain: dict[int, float], limits: tuple[ findings = [] if wet_level is not None and dry_level is not None: findings.append(f"Average {label} {wet_level:.1f} in wet half-hours ({word(wet_level)}) against {dry_level:.1f} when dry " - f"({word(dry_level)}): {ratio:.2f}x") + f"({word(dry_level)})" + (f": {ratio:.2f}x" if ratio is not None else "")) if events: findings.append(f"Over {len(events)} rain events (1 mm or more) with readings around them: {cleaned} left the air cleaner " f"in the 6 hours after than the 6 before, {worse} worse and {len(events) - cleaned - worse} about the same") diff --git a/lib/analysis/scan.py b/lib/analysis/scan.py index b65123b..11a6658 100644 --- a/lib/analysis/scan.py +++ b/lib/analysis/scan.py @@ -9,10 +9,11 @@ skewed, and a rank correlation is not thrown by them.""" import warnings +from datetime import tzinfo import numpy as np -from ..timeutil import MIN_DAY_SLOTS, SLOT +from ..timeutil import MIN_DAY_SLOTS, SLOT, naive_local PER_DAY = 48 LAGS = range(13) # the weather leads the air by 0 to 6 hours @@ -61,13 +62,22 @@ def benjamini_hochberg(pvalues: list[float], q: float = FDR) -> list[bool]: return keep -def to_grid(series: dict[int, float], origin: int, days: int) -> np.ndarray: - """A series as an array of 30-minute slots from `origin`, NaN where there is no reading.""" +def clock_slots(origin: int, days: int, tz: tzinfo) -> np.ndarray: + """Each 30-minute slot from `origin` (a midnight in tz) as a slot of the local clock, 48 a day: across a daylight-saving change + a time of day keeps its place (the skipped hour's slots stay empty, the repeated hour's are filled twice).""" + start = naive_local(origin, tz) + return np.array([int((naive_local(origin + k * SLOT, tz) - start).total_seconds()) // SLOT for k in range(days * PER_DAY + 2)]) + + +def to_grid(series: dict[int, float], origin: int, days: int, clock: np.ndarray) -> np.ndarray: + """A series as an array of the local clock's 30-minute slots from `origin` (clock: clock_slots), NaN where there is no reading.""" out = np.full(days * PER_DAY, np.nan) - for t, v in series.items(): - k = (t - origin) // SLOT - if 0 <= k < len(out): - out[k] = v + times = (np.fromiter(series, np.int64, len(series)) - origin) // SLOT + values = np.fromiter(series.values(), float, len(series)) + inside = (times >= 0) & (times < len(clock)) + slots = clock[times[inside]] + kept = slots < len(out) + out[slots[kept]] = values[inside][kept] return out @@ -86,8 +96,9 @@ def centred_ranks(x: np.ndarray) -> tuple[np.ndarray, np.ndarray]: def _ranked(grid: np.ndarray) -> tuple[np.ndarray, np.ndarray, np.ndarray]: - """(anomaly ranks centred, mask, their squares): what a series contributes to every pair it is in.""" - x, m = centred_ranks(anomalies(grid)) + """(anomaly ranks centred, mask, their squares): what a series contributes to every pair it is in. Single precision: + the correlations are judged to two decimals, and the shuffles take half the memory.""" + x, m = (a.astype(np.float32) for a in centred_ranks(anomalies(grid))) return x, m, x * x @@ -123,6 +134,14 @@ def within_day(weather: tuple, air: tuple, shuffled: tuple) -> tuple[float, int, return float(observed[best]), best, (1 + int((strongest >= abs(observed[best])).sum())) / (len(strongest) + 1) +def _within_day_tests(weather: np.ndarray, air: dict[str, tuple], permutations: np.ndarray) -> dict[str, tuple | None]: + """within_day of one weather series against every air series (`air`: their _ranked results). The weather's days shuffled + are the bulk of a scan's memory (100 MB and more for a year), so they are made for one weather series at a time.""" + ranked = _ranked(weather) + xs, ms = _shuffled(ranked[0], permutations), _shuffled(ranked[1], permutations) + return {name: within_day(ranked, a, (xs, xs * xs, ms)) for name, a in air.items()} + + def daily_means(grid: np.ndarray) -> np.ndarray: """One mean per day (NaN for a day with under 12 hours of readings).""" counts = (~np.isnan(grid)).reshape(-1, PER_DAY).sum(axis=1) @@ -177,42 +196,39 @@ def spread(secs: np.ndarray) -> float: def scan(air: dict[str, dict[int, float]], weather: dict[str, dict[int, float]], direction: dict[int, float] | None, - speed: dict[int, float] | None, origin: int, days: int, seed: int = 1, shuffles: int = PERMUTATIONS) -> dict: + speed: dict[int, float] | None, origin: int, days: int, tz: tzinfo, seed: int = 1, shuffles: int = PERMUTATIONS) -> dict: """The whole scan. `air` and `weather` map a name to {epoch: value} at 30-minute slots; `origin` is the epoch of the - first slot and `days` the number of days. Returns {"tests": [...], "directions": [...], "note"?: ...} where each test - is {air, weather, r, lag_hours, r_daily, days, p, survives}.""" - grids = {k: to_grid(v, origin, days) for k, v in air.items()} - wgrids = {k: to_grid(v, origin, days) for k, v in weather.items()} + first slot (a midnight in `tz`) and `days` the number of days. Returns {"tests": [...], "directions": [...], "note"?: ...} + where each test is {air, weather, r, lag_hours, r_daily, days, p, survives}.""" + clock = clock_slots(origin, days, tz) # series with no readings (a metric the sensor doesn't have) are left out + grids = {k: to_grid(v, origin, days, clock) for k, v in air.items() if v} + wgrids = {k: to_grid(v, origin, days, clock) for k, v in weather.items() if v} overlap = max((int(((~np.isnan(g)).reshape(-1, PER_DAY).sum(axis=1) >= MIN_DAY_SLOTS).sum()) for g in grids.values()), default=0) if overlap < MIN_DAYS: return {"tests": [], "directions": [], "note": f"Only {overlap} days have enough air-quality readings; at least {MIN_DAYS} are needed."} rng = np.random.default_rng(seed) permutations = np.array([rng.permutation(days) for _ in range(shuffles)]) - # what each series contributes to every pair, worked out once - air_ranked = {k: _ranked(g) for k, g in grids.items()} - air_daily = {k: daily_means(g) for k, g in grids.items()} - weather_prepared = {} - for k, g in wgrids.items(): - ranked = _ranked(g) - xs, ms = _shuffled(ranked[0], permutations), _shuffled(ranked[1], permutations) - weather_prepared[k] = (ranked, (xs, xs * xs, ms), daily_means(g)) + air_ranked = {k: _ranked(g) for k, g in grids.items()} # what each air series contributes to every pair, worked out once + inside = {w: _within_day_tests(g, air_ranked, permutations) for w, g in wgrids.items()} + air_daily, weather_daily = ({k: daily_means(g) for k, g in gs.items()} for gs in (grids, wgrids)) tests = [] for a_name in grids: - for w_name, (ranked, shuffled, w_daily) in weather_prepared.items(): - inside = within_day(ranked, air_ranked[a_name], shuffled) - r_daily, n_days, p_daily = day_to_day(w_daily, air_daily[a_name], rng, DAILY_PERMUTATIONS * shuffles // PERMUTATIONS) - if inside is None and p_daily is None: + for w_name in wgrids: + r_daily, n_days, p_daily = day_to_day(weather_daily[w_name], air_daily[a_name], rng, DAILY_PERMUTATIONS * shuffles // PERMUTATIONS) + if inside[w_name][a_name] is None and p_daily is None: continue - r, lag, p = inside or (None, 0, 1.0) + r, lag, p = inside[w_name][a_name] or (None, 0, 1.0) best_p = min(p, p_daily if p_daily is not None else 1.0) * 2 # two views of one pair: allow for both tests.append({"air": a_name, "weather": w_name, "r": r, "lag_hours": lag * SLOT / 3600, "r_daily": r_daily, "days": n_days, "p": min(1.0, best_p)}) directions = [] if direction and speed: - d, s = to_grid(direction, origin, days), to_grid(speed, origin, days) + d, s = to_grid(direction, origin, days, clock), to_grid(speed, origin, days, clock) for a_name, a in grids.items(): if (found := direction_effect(a, d, s, permutations)) is not None: directions.append({"air": a_name, **found}) + if not tests and not directions: # "nothing stands out" would claim a comparison that wasn't made + return {"tests": [], "directions": [], "note": "No weather readings overlap the air-quality readings enough to compare."} keep = benjamini_hochberg([t["p"] for t in tests] + [d["p"] for d in directions]) for t, k in zip([*tests, *directions], keep): t["survives"] = k diff --git a/lib/bot.py b/lib/bot.py index 583c27f..b89ae27 100644 --- a/lib/bot.py +++ b/lib/bot.py @@ -14,6 +14,7 @@ from collections import defaultdict from dataclasses import dataclass, field from datetime import datetime, UTC +from typing import NamedTuple from telegram import InputMediaPhoto, Message, ReplyKeyboardRemove, Update from telegram.constants import ChatAction, ChatType @@ -23,15 +24,17 @@ from . import intent, periods, prompt, report, report_menu, templates from .admins import Admins from .alerts import menu -from .alerts import AlertState, with_footer +from .alerts import AlertState from .irrigation import IrrigationError, Watchdog from .irrigation import menu as irrigation_menu from .charts import render as render_chart from .tools import Turn from .config import Config +from .entities import with_link +from .sources import Source from .timeutil import now_local from .llm import Agent, shorten_old, strip_tool_turns, trim_history -from .logs import QUOTES +from .logs import QUOTES, fit log = logging.getLogger(__name__) @@ -60,6 +63,18 @@ class ChatState: lock: asyncio.Lock = field(default_factory=asyncio.Lock) +@dataclass +class Answer: + """What a question is answered with, worked out before it is sent.""" + reply: str | None = None # its text; None until something gives one + photos: list = field(default_factory=list) + titles: list[str] = field(default_factory=list) + silent: bool = False # a chart made in code: it goes with just its title, its text kept for the history + ready: periods.Drawn | None = None # a chart drawn moments ago, sent as it is + made: float | None = None # seconds the tool took to make the chart in code, for the timing line + ok: bool = True # False: the reply says something went wrong + + async def watchdog(label: str, seconds: float): """Log every task's stack if the question is still running after `seconds`: a hang says where it waits.""" await asyncio.sleep(seconds) @@ -71,9 +86,9 @@ async def watchdog(label: str, seconds: float): log.warning("Still working after %ds on %s; tasks waiting:\n %s", seconds, label, "\n ".join(stacks) or "(none)") -def _short(text: str, limit: int = 200) -> str: - text = " ".join(text.split()) - return text if len(text) <= limit else text[:limit] + "..." +def _short(text: str, limit: int) -> str: + """The text on one line, cut to `limit` characters (logs.fit): a question or button's data as the log names it.""" + return fit(" ".join(text.split()), limit) def chart_times(times: dict, made: float | None, pngs: list[bytes], sending: float) -> str: @@ -138,14 +153,11 @@ def connection_problem(err) -> bool: def polling_error(error: TelegramError): """Errors while polling for new messages. The library retries by itself, so a dropped - connection is a one-line warning, not a traceback. Must not raise.""" - try: - if connection_problem(error): - log.warning("Telegram connection problem while polling (%s: %s) - retrying", type(error).__name__, error) - else: - log.error("Telegram polling error (%s: %s)", type(error).__name__, error) - except Exception: - pass + connection is a one-line warning, not a traceback.""" + if connection_problem(error): + log.warning("Telegram connection problem while polling (%s: %s) - retrying", type(error).__name__, error) + else: + log.error("Telegram polling error (%s: %s)", type(error).__name__, error) TYPING_MAX_SECONDS = 600 # a safety net: the indicator never outlives a stuck question @@ -197,13 +209,18 @@ async def run(self, stop: asyncio.Event): await asyncio.wait_for(stop.wait(), DRAFT_MIN_GAP) +class Delivered(NamedTuple): + carried: bool # the keyboard (markup) went with the reply + photo: Message | None = None # the single chart, as sent + + async def deliver(msg: Message, text: str, photos: list[bytes], link: tuple[str, str] | None = None, markup=None, - titles: list[str] | None = None, period_row=None, sent_photo: list | None = None, bold_headings: bool = False) -> bool: + titles: list[str] | None = None, period_row=None, bold_headings: bool = False) -> Delivered: """Send the answer with any charts. A chart carries only its title as its caption; any answer text is its own message, sent first. link = (label, url) adds a small italic link line at the end of the text. markup (the button keyboard) rides on - the last text, or on the single photo when there is no text; returns whether it was sent (a group of photos can't carry one). - period_row (the period buttons) rides on a single photo instead, and then the keyboard waits; the photo's Message goes into sent_photo. - bold_headings: the text is the report (or a pollen / forecast lookup), whose heading names are set in bold.""" + the last text, or on the single photo when there is no text (a group of photos can't carry one). period_row (the period + buttons) rides on a single photo instead, and then the keyboard waits. bold_headings: the text is the report (or a pollen / + forecast lookup), whose heading names are set in bold.""" text = text.strip() if photos: text = strip_chart_talk(text) @@ -213,7 +230,7 @@ async def deliver(msg: Message, text: str, photos: list[bytes], link: tuple[str, chunks = split_message(text) for i, chunk in enumerate(chunks): last = i == len(chunks) - 1 - body, ents = with_footer(chunk, link) if last else (chunk, []) + body, ents = with_link(chunk, link) if last else (chunk, []) ents = [*(report.heading_entities(chunk) if bold_headings else []), *ents] await msg.reply_text(body, entities=ents or None, disable_web_page_preview=True, **({"reply_markup": markup} if markup and last and not photos else {})) @@ -224,22 +241,21 @@ async def deliver(msg: Message, text: str, photos: list[bytes], link: tuple[str, if len(photos) == 1: buttons = period_row or markup sent = await msg.reply_photo(photos[0], caption=caption(0), **({"reply_markup": buttons} if buttons else {})) - if sent_photo is not None and sent is not None: - sent_photo.append(sent) - return bool(markup) and not period_row + return Delivered(bool(markup) and not period_row, sent) await msg.reply_media_group([InputMediaPhoto(p, caption=caption(i)) for i, p in enumerate(photos)]) except TelegramError: log.exception("Couldn't send the chart") if not text: await msg.reply_text("Sorry, I couldn't send that chart. Please try again in a moment.") - return carried + return Delivered(carried) class Bot: - def __init__(self, cfg: Config, agent: Agent, sources: list, state: AlertState | None, irrigation=None): - """irrigation: the Tuya client of the irrigation controller, when one is configured (its button replaces Wind).""" + def __init__(self, cfg: Config, agent: Agent, sources: list[Source], state: AlertState, irrigation=None): + """state: the known chats and their settings. irrigation: the Tuya client of the irrigation controller, when one is + configured (its button replaces Wind).""" self.cfg, self.agent, self.sources, self.state, self.irrigation = cfg, agent, sources, state, irrigation - self.watchdog = Watchdog(irrigation, state) if irrigation and state else None # closes the valve after a run, however it ends + self.watchdog = Watchdog(irrigation, state) if irrigation else None # closes the valve after a run, however it ends self.keyboard_version = templates.version(irrigation is not None) self.admin_only = bool(getattr(cfg, "admin_only", False)) # ADMIN_ONLY: only the admins of the groups the bot is in may use it self.admins = Admins(state, getattr(cfg, "admin_chat_ids", ())) @@ -256,6 +272,12 @@ def __init__(self, cfg: Config, agent: Agent, sources: list, state: AlertState | self.background: set[asyncio.Task] = set() # tasks running on their own, kept so they aren't dropped self.charted = periods.Charted(state.charts if state else None, state.save if state else None) # the question behind each chart sent (kept across restarts) + async def close(self): + """Cancel the tasks still running on their own (the uploads of a chart's other periods).""" + for task in self.background: + task.cancel() + await asyncio.gather(*self.background, return_exceptions=True) + def register(self, app): app.add_handler(CommandHandler(["start", "help"], self.on_start)) app.add_handler(CommandHandler("reset", self.on_reset)) @@ -341,8 +363,7 @@ async def on_button(self, update: Update, context: ContextTypes.DEFAULT_TYPE): try: await handler(update, context) except ChatMigrated as e: # a press on an alert from before its group became a supergroup (the chat has a new id) - if self.state: - self.state.migrate_chat(query.message.chat.id, e.new_chat_id) + self.state.migrate_chat(query.message.chat.id, e.new_chat_id) with contextlib.suppress(TelegramError): await query.answer("This group was upgraded to a supergroup: please use the buttons on new alerts") @@ -354,40 +375,37 @@ def _title(update: Update) -> str: def remember_chat(self, update: Update): """Record the chat so alerts can be sent to it (Telegram can't list a bot's chats).""" - if self.state and update.effective_chat: + if update.effective_chat: self.state.add_chat(update.effective_chat.id, self._title(update)) async def on_membership(self, update: Update, context: ContextTypes.DEFAULT_TYPE): """The bot was added to or removed from a chat.""" change = update.my_chat_member - if not self.state or not change: + if not change: return status = change.new_chat_member.status if status in ("member", "administrator"): self.remember_chat(update) elif status in ("left", "kicked"): - self.state.remove_chat(change.chat.id, f"bot {status}") + self.state.mark_gone(change.chat.id, f"bot {status}") def _alert_kinds(self) -> list[str]: - return menu.kinds_for(self.by_name, bool(self.irrigation)) + return menu.available_kinds(self.by_name, bool(self.irrigation)) async def on_alerts(self, update: Update, context: ContextTypes.DEFAULT_TYPE): - """/alerts: the alert settings, as buttons (Subscribe | Unsubscribe). /alerts on and /alerts off still subscribe or + """/alerts: the alert settings, as buttons (🔔 Alerts and 🔊 Sound). /alerts on and /alerts off still subscribe or unsubscribe every type.""" msg, chat = update.effective_message, update.effective_chat in_group = chat.type != ChatType.PRIVATE if not in_group and not await self._allowed(update, context): # anyone in a group may look; a private chat is gated like the rest return - if not self.state: - await msg.reply_text("Alerts aren't available: no sensor is connected.") - return arg = context.args[0].lower() if context.args else "" self.remember_chat(update) if arg in ("on", "off"): if in_group and not (update.effective_user and await self._is_group_admin(context.bot, chat.id, update.effective_user.id)): await msg.reply_text("Only group admins can change alerts") # the menu below is still shown else: - self.state.set_alerts(chat.id, self._title(update), arg == "on") + self.state.set_alerts(chat.id, arg == "on") log.info("/alerts %s in %s", arg, describe_source(update)) muted, kinds = self.state.muted(chat.id), self._alert_kinds() loud = self.state.loud(chat.id) if chat.type == ChatType.PRIVATE else None # sound settings: private chats only @@ -395,8 +413,7 @@ async def on_alerts(self, update: Update, context: ContextTypes.DEFAULT_TYPE): async def _help_and_alerts(self, update: Update, context: ContextTypes.DEFAULT_TYPE): """What the bot can do, then the alert settings: the Help & Alerts button and /usage.""" - await update.effective_message.reply_text(templates.capabilities_text("Ecowitt" in self.by_name, "AirGradient" in self.by_name, - "Pollen" in self.by_name, "Forecast" in self.by_name)) + await update.effective_message.reply_text(templates.capabilities_text(self.by_name)) await self.on_alerts(update, context) async def on_usage(self, update: Update, context: ContextTypes.DEFAULT_TYPE): @@ -416,7 +433,7 @@ async def on_alert_button(self, update: Update, context: ContextTypes.DEFAULT_TY its sound on or off. In a group only admins may change them.""" query = update.callback_query press = menu.parse(query.data) - if press is None or not self.state: + if press is None: await query.answer() return message = query.message @@ -425,41 +442,41 @@ async def on_alert_button(self, update: Update, context: ContextTypes.DEFAULT_TY await query.answer("Only group admins can change alerts") return kinds = self._alert_kinds() - parent = press.parent if press.parent in kinds else None + parent = press.parent # set: the buttons under an alert of that type; None: the /alerts message + if parent and parent not in kinds: # an alert of a type the bot no longer sends + await query.answer(STALE_BUTTON) + return private = chat.type == ChatType.PRIVATE toast, opened = None, press.section if press.action != "close" else None if press.action == "sound" and private and press.kind in kinds and chat.id in self.state.chats: self.state.set_sound(chat.id, press.kind, press.on) toast = f"{self._alerts_named(press.kind)} {'now make' if press.on else 'no longer make'} a sound" elif press.action == "alert" and press.kind in kinds and chat.id in self.state.chats: - self.state.set_kind(chat.id, press.kind, press.on) + self.state.set_kind(chat.id, press.kind, press.on, kinds) toast = f"{self._alerts_named(press.kind)} {'on' if press.on else 'off'} in this chat" elif press.action not in ("open", "close"): await query.answer() return muted = self.state.muted(chat.id) loud = self.state.loud(chat.id) if private else None - markup = menu.alert_keyboard(parent, muted, loud, opened) if parent else menu.keyboard(muted, kinds, opened, loud) await query.answer(toast) - try: - if (message.text or "").startswith(menu.TITLE): # the /alerts message: keep its on/off summary current - await query.edit_message_text(menu.title(muted, kinds), reply_markup=markup) - else: # an alert: only its buttons change - await query.edit_message_reply_markup(reply_markup=markup) - except BadRequest as e: - if "not modified" not in str(e).lower(): - raise + if parent: # an alert: only its buttons change + await self._edit(query, None, menu.alert_keyboard(parent, muted, loud, opened)) + else: # the /alerts message: its on/off summary is kept current too + await self._edit(query, menu.title(muted, kinds), menu.keyboard(muted, kinds, opened, loud)) # ---------- irrigation ---------- - def _irrigation_alerts(self, chat_id: int, kind: str = "irrigation_battery") -> bool: - """Does this chat get the irrigation battery alerts (kind "irrigation_battery") or the pause alerts ("irrigation_pause")?""" - return bool(self.state and self.state.subscribed(chat_id, kind)) + def _irrigation_alerts(self, chat_id: int) -> tuple[bool, bool]: + """Does this chat get the irrigation battery alerts, and the pause alerts?""" + return tuple(self.state.subscribed(chat_id, kind) for kind, _ in irrigation_menu.ALERTS.values()) def _irrigation_keyboard(self, chat_id: int, opened: str | None = None, paused: bool = False, running: bool = False): - return irrigation_menu.keyboard(self._irrigation_alerts(chat_id), self._irrigation_alerts(chat_id, "irrigation_pause"), opened, paused, running) + return irrigation_menu.keyboard(*self._irrigation_alerts(chat_id), opened, paused, running) - def _irrigation_text(self, status: dict, chat_id: int) -> str: - return irrigation_menu.status_text(status, self._irrigation_alerts(chat_id), self._irrigation_alerts(chat_id, "irrigation_pause")) + def _irrigation_view(self, status: dict, chat_id: int, opened: str | None = None): + """(the status message, its buttons) for the controller's `status`.""" + return (irrigation_menu.status_text(status, *self._irrigation_alerts(chat_id)), + self._irrigation_keyboard(chat_id, opened, irrigation_menu.is_paused(status), bool(status.get("switch")))) async def send_irrigation(self, msg: Message): """The Irrigation button: the controller's status, with its buttons.""" @@ -469,14 +486,15 @@ async def send_irrigation(self, msg: Message): log.warning("Irrigation status failed: %s", e) await msg.reply_text("Sorry, I couldn't reach the irrigation controller.") return - await msg.reply_text(self._irrigation_text(status, msg.chat_id), reply_markup=self._irrigation_keyboard(msg.chat_id, None, irrigation_menu.is_paused(status), bool(status.get("switch")))) + text, markup = self._irrigation_view(status, msg.chat_id) + await msg.reply_text(text, reply_markup=markup) async def _irrigation_act(self, action: str, arg: str, chat_id: int) -> str | None: """Do what a button asks. Returns the toast to show, or None for a button that means nothing. Raises IrrigationError.""" if action == "water" and arg.isdigit() and int(arg) in irrigation_menu.WATER_MINUTES: - await self.irrigation.water(int(arg)) - if self.watchdog: + if self.watchdog: # first: a command whose reply is lost may still have opened the valve self.watchdog.arm(int(arg)) + await self.irrigation.water(int(arg)) return f"Watering for {arg} minutes" if action == "delay" and arg in irrigation_menu.DELAYS: await self.irrigation.delay(arg) @@ -486,9 +504,10 @@ async def _irrigation_act(self, action: str, arg: str, chat_id: int) -> str | No if self.watchdog: self.watchdog.disarm() return "Switched off" - if action in ("batt", "pause") and arg in ("on", "off") and self.state and chat_id in self.state.chats: - self.state.set_kind(chat_id, "irrigation_battery" if action == "batt" else "irrigation_pause", arg == "on") - return f"Irrigation {'battery' if action == 'batt' else 'pause'} alerts {arg} in this chat" + if action in irrigation_menu.ALERTS and arg in ("on", "off") and chat_id in self.state.chats: + kind, name = irrigation_menu.ALERTS[action] + self.state.set_kind(chat_id, kind, arg == "on", self._alert_kinds()) + return f"Irrigation {name} alerts {arg} in this chat" return None async def on_irrigation_button(self, update: Update, context: ContextTypes.DEFAULT_TYPE): @@ -504,8 +523,8 @@ async def on_irrigation_button(self, update: Update, context: ContextTypes.DEFAU opened = arg if action == "open" and arg in irrigation_menu.SECTIONS else None await query.answer() # the message says it: no need to ask the controller to open a list - await self._edit_irrigation(query, None, self._irrigation_keyboard(message.chat_id, opened, irrigation_menu.paused_in(message.text), - irrigation_menu.running_in(message.text))) + await self._edit(query, None, self._irrigation_keyboard(message.chat_id, opened, irrigation_menu.paused_in(message.text), + irrigation_menu.running_in(message.text))) return if action == "refresh": # read the controller again and redraw: nothing is sent to it, so no wait try: @@ -515,8 +534,7 @@ async def on_irrigation_button(self, update: Update, context: ContextTypes.DEFAU await query.answer("Couldn't reach the irrigation controller", show_alert=True) return await query.answer("Refreshed") - await self._edit_irrigation(query, self._irrigation_text(status, message.chat_id), - self._irrigation_keyboard(message.chat_id, None, irrigation_menu.is_paused(status), bool(status.get("switch")))) + await self._edit(query, *self._irrigation_view(status, message.chat_id)) return if action == "water" and arg.isdigit() and int(arg) in irrigation_menu.WATER_MINUTES: # not while the timer is paused try: @@ -528,8 +546,9 @@ async def on_irrigation_button(self, update: Update, context: ContextTypes.DEFAU if irrigation_menu.is_paused(status): log.info("Irrigation: water %s refused, paused for %s (chat %s)", arg, status["weather_delay"], message.chat_id) await query.answer(irrigation_menu.PAUSED_ERROR) - text, entities = irrigation_menu.with_error(self._irrigation_text(status, message.chat_id)) - await self._edit_irrigation(query, text, self._irrigation_keyboard(message.chat_id, None, True, bool(status.get("switch"))), entities) # nothing was sent: no wait + text, markup = self._irrigation_view(status, message.chat_id) + text, entities = irrigation_menu.with_error(text) + await self._edit(query, text, markup, entities) # nothing was sent: no wait return try: toast = await self._irrigation_act(action, arg, message.chat_id) @@ -548,11 +567,13 @@ async def on_irrigation_button(self, update: Update, context: ContextTypes.DEFAU except IrrigationError as e: log.warning("Irrigation status after %s %s failed: %s", action, arg, e) # the toast said what was done; the message stays as it was return - stay = "alerts" if action in ("batt", "pause") else None # the Alerts menu stays open after a toggle, to show the new label - await self._edit_irrigation(query, self._irrigation_text(status, message.chat_id), self._irrigation_keyboard(message.chat_id, stay, irrigation_menu.is_paused(status), bool(status.get("switch")))) + stay = "alerts" if action in irrigation_menu.ALERTS else None # the Alerts menu stays open after a toggle, to show the new label + await self._edit(query, *self._irrigation_view(status, message.chat_id, stay)) @staticmethod - async def _edit_irrigation(query, text: str | None, markup, entities=None): + async def _edit(query, text: str | None, markup, entities=None): + """Put new text (None: the text stays) and buttons into the message a button was pressed under. Telegram refusing an edit + that changes nothing is fine.""" try: if text is None: await query.edit_message_reply_markup(reply_markup=markup) @@ -566,11 +587,7 @@ async def on_report_button(self, update: Update, context: ContextTypes.DEFAULT_T """A press on Refresh under the report: read everything again and redraw the report in the same message. The chat's conversation is not touched (this is not a new question).""" query = update.callback_query - if query.data != report_menu.REFRESH: - await query.answer(STALE_BUTTON) - return - read = intent.read("Report", now_local(self.cfg.tz), "Ecowitt" in self.by_name, "AirGradient" in self.by_name, - "Pollen" in self.by_name, "Forecast" in self.by_name) + read = intent.read("Report", now_local(self.cfg.tz), self.by_name) try: text = await asyncio.wait_for(self._written_in_code(read, Turn(text="Report")), TURN_SECONDS) except Exception as e: # the message stays as it was @@ -578,7 +595,7 @@ async def on_report_button(self, update: Update, context: ContextTypes.DEFAULT_T await query.answer("Couldn't refresh, please try again", show_alert=True) return await query.answer() # no toast: the times in the headings show what changed - await self._edit_irrigation(query, text, report_menu.keyboard(), report.heading_entities(text)) + await self._edit(query, text, report_menu.keyboard(), report.heading_entities(text)) async def on_period_button(self, update: Update, context: ContextTypes.DEFAULT_TYPE): """A press on 1D / 7D / 1M / 3M / 1Y under a chart: ask the chart's question again for that period and put the new chart @@ -653,22 +670,20 @@ async def on_start(self, update: Update, context: ContextTypes.DEFAULT_TYPE): await update.effective_message.reply_text( HELP.format(user=update.effective_user.id, chat=update.effective_chat.id), reply_markup=templates.keyboard(self.irrigation is not None) if private else None) - if private and self.state: + if private: self.state.set_keyboard(update.effective_chat.id, self.keyboard_version) async def refresh_keyboards(self, tg_bot) -> int: """At startup: tell each private chat whose buttons are out of date that they changed, with the new keyboard (silently). A chat that hid them is left alone. Returns how many were sent.""" - if not self.state: - return 0 sent = 0 - for chat_id in [c for c in self.state.chats if c > 0 and self.state.keyboard(c) not in (self.keyboard_version, templates.HIDDEN)]: + for chat_id in [c for c in self.state.present() if c > 0 and self.state.keyboard(c) not in (self.keyboard_version, templates.HIDDEN)]: try: await tg_bot.send_message(chat_id, "Buttons updated.", reply_markup=templates.keyboard(self.irrigation is not None), disable_notification=True) self.state.set_keyboard(chat_id, self.keyboard_version) sent += 1 except Forbidden as e: # blocked the bot - self.state.remove_chat(chat_id, f"can't post: {e}") + self.state.mark_gone(chat_id, f"can't post: {e}") except TelegramError as e: log.warning("Buttons update to %s failed: %s", chat_id, e) if sent: @@ -678,8 +693,7 @@ async def refresh_keyboards(self, tg_bot) -> int: def _keyboard_stale(self, msg: Message) -> bool: """Does this private chat need the current buttons (it has never had them, or they have changed since)? Not when it hid them.""" - return bool(self.state and msg.chat.type == ChatType.PRIVATE - and self.state.keyboard(msg.chat_id) not in (self.keyboard_version, templates.HIDDEN)) + return msg.chat.type == ChatType.PRIVATE and self.state.keyboard(msg.chat_id) not in (self.keyboard_version, templates.HIDDEN) async def on_keyboard(self, update: Update, context: ContextTypes.DEFAULT_TYPE): """/keyboard shows the buttons again, /keyboard off hides them (private chats only).""" @@ -690,12 +704,10 @@ async def on_keyboard(self, update: Update, context: ContextTypes.DEFAULT_TYPE): await msg.reply_text("The buttons are only in private chats.") elif context.args and context.args[0].lower() == "off": await msg.reply_text("Buttons hidden. /keyboard shows them again.", reply_markup=ReplyKeyboardRemove()) - if self.state: - self.state.set_keyboard(msg.chat_id, templates.HIDDEN) + self.state.set_keyboard(msg.chat_id, templates.HIDDEN) else: await msg.reply_text("Here are the buttons.", reply_markup=templates.keyboard(self.irrigation is not None)) - if self.state: - self.state.set_keyboard(msg.chat_id, self.keyboard_version) + self.state.set_keyboard(msg.chat_id, self.keyboard_version) async def on_error(self, update: object, context: ContextTypes.DEFAULT_TYPE): """Errors raised inside handlers: network blips get one line, anything else a traceback.""" @@ -763,10 +775,10 @@ def _content(self, msg: Message, text: str, bot_id: int) -> str: async def _written_in_code(self, read, turn: Turn) -> str: """The report ("weather now") from the fast path's tool calls, made together; lib/report.py lays it out.""" calls = [read.fast[:2], *read.more] - results = await asyncio.gather(*(self.agent.tools.call(name, json.dumps(args), turn) for name, args in calls)) + results = await self.agent.tools.call_all(calls, turn) by_tool = {name: result for (name, _), result in zip(calls, results)} if read.report: - return report.report(by_tool) + return report.report(by_tool, now_local(self.cfg.tz).date()) return report.weather_now(by_tool["weather_now"]) or "The weather station isn't answering right now." async def _warn(self, msg: Message, tokens: int): @@ -780,7 +792,7 @@ async def _chart_in_code(self, read, turn: Turn) -> str | None: """A chart asked for plainly: the tool draws it and the caption is written here. None (the model then takes the question) when there is nothing to caption.""" calls = [read.fast[:2], *read.more] - results = await asyncio.gather(*(self.agent.tools.call(name, json.dumps(args), turn) for name, args in calls)) + results = await self.agent.tools.call_all(calls, turn) caption = report.chart_caption(read.fast[0], read.fast[1], results[0], turn, self.cfg.tz) if caption is None: turn.charts.clear() @@ -791,11 +803,10 @@ async def _lookup_in_code(self, read, turn: Turn) -> str | None: """A plain lookup (a reading, the air, the pollen, the forecast, what the bot can do) answered from its one tool result; None (the model then takes the question) when it can't be.""" if read.lookup == "about": - return templates.capabilities_text("Ecowitt" in self.by_name, "AirGradient" in self.by_name, - "Pollen" in self.by_name, "Forecast" in self.by_name) + return templates.capabilities_text(self.by_name) name, args = read.fast[:2] result = await self.agent.tools.call(name, json.dumps(args), turn) - return report.lookup(read.lookup, result, read.lookup_arg, read.sides) + return report.lookup(read.lookup, result, read.lookup_arg, read.sides, now_local(self.cfg.tz).date()) async def _ask_model(self, working: list[dict], system: str, read, turn: Turn, draft, on_heavy=None, conv_id=None) -> str: """The model's answer. If it has not answered in TURN_SECONDS, ask again once with one reasoning step less (medium > low > @@ -805,8 +816,8 @@ async def _ask_model(self, working: list[dict], system: str, read, turn: Turn, d while True: try: return await asyncio.wait_for( - self.agent.run(working, system, effort, first_call=first, require_tool=read.needs_data, - no_tools=read.about_the_bot, turn=turn, on_heavy=on_heavy, conv_id=conv_id, + self.agent.run(working, system, effort, first_call=first, tool_choice=read.tool_choice, turn=turn, + on_heavy=on_heavy, conv_id=conv_id, **({"on_text": draft.update} if draft else {})), budget) except TimeoutError: lower = intent.lower_effort(effort) @@ -819,13 +830,11 @@ async def _ask_model(self, working: list[dict], system: str, read, turn: Turn, d async def _draw_ahead(self, text: str, now) -> tuple[str, bytes, str, str] | None: """(image key, PNG, title, write-up) of the chart `text` asks for when it is one made in code with its own words as its key; None otherwise. Raises when it can't be drawn. Nobody is waiting: its requests are not urgent and its logs are quiet.""" - read = intent.read(text, now, "Ecowitt" in self.by_name, "AirGradient" in self.by_name, "Pollen" in self.by_name, - "Forecast" in self.by_name) + read = intent.read(text, now, self.by_name) key = self.images.canonical(text, now) - if not (read.fast and read.chart_in_code and key): # not a chart made in code ("Report"), or a source that isn't there + if not (read.chart_in_code and key): # not a chart made in code ("Report"), or a source that isn't there return None - turn = Turn(chart_asked=read.chart_asked, chart_field=read.chart_field, chart_fields=read.chart_fields, - average_asked=read.average_asked, readings=read.readings, per_day=read.per_day, text=text, ahead=True) + turn = Turn.of(read, text, ahead=True) caption = await asyncio.wait_for(self._chart_in_code(read, turn), TURN_SECONDS) if caption is None or len(turn.charts) != 1: raise ValueError("no chart came of it") @@ -923,6 +932,7 @@ async def respond(self, update: Update, context: ContextTypes.DEFAULT_TYPE, text self.idle.set() async def _respond(self, update: Update, context: ContextTypes.DEFAULT_TYPE, text: str, redraw: Message | None = None): + """Answer `text` (redraw: a chart's message, when a period button asks for that chart again in place).""" msg = update.effective_message text = text.strip() if not text: @@ -932,112 +942,118 @@ async def _respond(self, update: Update, context: ContextTypes.DEFAULT_TYPE, tex keep_typing(context.bot, msg.chat_id, self._thread(msg), stop_typing)) await asyncio.sleep(0) now = now_local(self.cfg.tz) - read = intent.read(text, now, "Ecowitt" in self.by_name, "AirGradient" in self.by_name, "Pollen" in self.by_name, - "Forecast" in self.by_name) + read = intent.read(text, now, self.by_name) log.info("%s %s%s", describe_source(update), f"(reasoning: {read.effort}) " if read.effort != intent.EFFORT_DEFAULT else "", text, extra=QUOTES) for source in self.sources: # fetch recent readings while the model thinks - if source.wants(text) and not (read.fast and read.chart_in_code): # a chart made in code reads just what it needs, first in line + if source.wants(text) and not read.chart_in_code: # a chart made in code reads just what it needs, first in line source.poke() if read.fast: log.info("Fast path: %s", read.fast[2]) chat = self.chats[self._key(msg)] - thread_id = self._thread(msg) started = time.monotonic() async with chat.lock: - draft = (Draft(context.bot, msg.chat_id, thread_id) + draft = (Draft(context.bot, msg.chat_id, self._thread(msg)) if redraw is None and msg.chat.type == ChatType.PRIVATE and hasattr(context.bot, "send_message_draft") else None) drafting = asyncio.create_task(draft.run(stop_typing)) if draft else None # a private chat also gets the "Thinking..." draft stuck = asyncio.create_task(watchdog(_short(text, 40), WATCHDOG_SECONDS)) working = [*chat.history, {"role": "user", "content": self._content(msg, text, context.bot.id)}] new_from = len(working) - ok, photos, titles, silent = True, [], [], False - made = None # how long the tool took when the chart is made in code, for the timing line - turn = Turn(chart_asked=read.chart_asked, chart_field=read.chart_field, chart_fields=read.chart_fields, - average_asked=read.average_asked, readings=read.readings, per_day=read.per_day, text=text) - ready = self.images.ready(text, now) if read.fast and read.chart_in_code and redraw is None else None + turn = Turn.of(read, text) try: - reply = None - if ready: # drawn moments ago (ahead, or for someone's same question): sent as it is, its text kept for the history - reply, photos, titles, silent = ready.caption, [ready.photo], [ready.title], True - elif read.fast and read.chart_in_code: # drawn by the tool; the chart goes out with just its title (the text is kept for the history) - began = time.monotonic() - reply = await asyncio.wait_for(self._chart_in_code(read, turn), TURN_SECONDS) - silent = reply is not None - made = time.monotonic() - began if silent else None - elif read.lookup: - reply = await asyncio.wait_for(self._lookup_in_code(read, turn), TURN_SECONDS) - if reply is None: - turn.forecast_shown.clear() - elif read.fast and (read.report or read.weather_now): # written in code from the tools' results, no model - reply = await asyncio.wait_for(self._written_in_code(read, turn), TURN_SECONDS) - if reply is not None: - working.append({"role": "assistant", "content": reply}) - else: - before = [str(m["content"]) for m in chat.history if m["role"] == "user"][-2:] - found = intent.topics(str(working[new_from - 1]["content"]), before) - system = prompt.build(datetime.now(self.cfg.tz), [s.describe() for s in self.sources], read.hints, - read.about_the_bot, read.rain_caption, found) - reply = await self._ask_model(working, system, read, turn, draft, lambda tokens: self._warn(msg, tokens), - f"ecobot-{msg.chat_id}") + answer = await self._answer(msg, read, turn, working, draft, now, redraw) chat.history = shorten_old(trim_history(strip_tool_turns(working))) for spec in turn.charts[:MAX_CHARTS]: # drawn while "typing..." is still showing try: - photos.append(await asyncio.to_thread(render_chart, spec, self.cfg.tz, turn.times)) - titles.append(spec.caption(now.date())) + answer.photos.append(await asyncio.to_thread(render_chart, spec, self.cfg.tz, turn.times)) + answer.titles.append(spec.caption(now.date())) except Exception: log.exception("Chart failed; sending the answer without it") except TimeoutError: log.error("Gave up after %ds on: %s", TURN_SECONDS, text, extra=QUOTES) - ok, reply = False, "Sorry, that took too long. Please try again in a moment." + answer = Answer("Sorry, that took too long. Please try again in a moment.", ok=False) except Exception as e: log.exception("Agent error") # The details (which can include provider error bodies) go to the log, not the chat - ok, reply = False, f"Sorry, something went wrong on my side ({type(e).__name__}). Please try again in a moment." + answer = Answer(f"Sorry, something went wrong on my side ({type(e).__name__}). Please try again in a moment.", ok=False) finally: stuck.cancel() stop_typing.set() await asyncio.gather(*[t for t in (typing, drafting) if t]) # wait for any in-flight "typing" or draft so none is sent after the reply - - caption = reply if silent else None # a chart made in code: its text, kept with the chart for the next to ask the same - if silent and photos: + await self._send(msg, context.bot, read, turn, answer, working[new_from:], now, redraw, started) + + async def _answer(self, msg: Message, read, turn: Turn, working: list[dict], draft, now: datetime, redraw: Message | None) -> Answer: + """The answer, from the first way that gives one: a chart drawn moments ago, a chart made in code, a lookup, the report + written in code, else the model. `working` is the conversation, the question last; the answer joins it.""" + if read.chart_in_code and redraw is None and (ready := self.images.ready(turn.text, now)): + answer = Answer(ready.caption, [ready.photo], [ready.title], silent=True, ready=ready) # drawn ahead, or for someone's same question + elif read.chart_in_code: # drawn by the tool; the chart goes out with just its title (the text is kept for the history) + began = time.monotonic() + reply = await asyncio.wait_for(self._chart_in_code(read, turn), TURN_SECONDS) + answer = Answer(reply, silent=reply is not None, made=time.monotonic() - began if reply is not None else None) + elif read.lookup: + answer = Answer(await asyncio.wait_for(self._lookup_in_code(read, turn), TURN_SECONDS)) + if answer.reply is None: + turn.forecast_shown.clear() + elif read.fast and (read.report or read.weather_now): # written in code from the tools' results, no model + answer = Answer(await asyncio.wait_for(self._written_in_code(read, turn), TURN_SECONDS)) + else: + answer = Answer() + if answer.reply is not None: + working.append({"role": "assistant", "content": answer.reply}) + return answer + before = [str(m["content"]) for m in working[:-1] if m["role"] == "user"][-2:] + found = intent.topics(str(working[-1]["content"]), before) + system = prompt.build(datetime.now(self.cfg.tz), {s.name: s.describe() for s in self.sources}, read.hints, + read.about_the_bot, read.rain_caption, found) + answer.reply = await self._ask_model(working, system, read, turn, draft, lambda tokens: self._warn(msg, tokens), + f"ecobot-{msg.chat_id}") + return answer + + async def _send(self, msg: Message, bot, read, turn: Turn, answer: Answer, added: list[dict], now: datetime, + redraw: Message | None, started: float): + """Send the answer, a single chart with the period buttons under it (in place of `redraw` when a button asked for it), the + keyboard if the chat's is out of date; then keep what follows from it: the chart for its period buttons and the next to + ask, the question behind it, the forecast days shown. `added`: the messages the answer added to the conversation.""" + text, reply, photos, titles = turn.text, answer.reply, answer.photos, answer.titles + caption = reply if answer.silent else None # a chart made in code: its text, kept with the chart for the next to ask the same + if answer.silent and photos: reply = "" - if ok and not reply.strip() and not photos: # a chart with no text that could not be drawn + if answer.ok and not reply.strip() and not photos: # a chart with no text that could not be drawn reply = "Sorry, I couldn't draw that chart. Please try again in a moment." - used = [m for m in working[new_from:] if m["role"] == "tool"] - log.info("%s %s in %.1fs, tools %d, charts %d, %d chars: %s", "Replied" if ok else "Error reply", msg.chat_id, + used = [m for m in added if m["role"] == "tool"] + log.info("%s %s in %.1fs, tools %d, charts %d, %d chars: %s", "Replied" if answer.ok else "Error reply", msg.chat_id, time.monotonic() - started, len(used), len(photos), len(reply), reply, extra=QUOTES) - used_air = any(tc["function"]["name"] == "air_quality" - for m in working[new_from:] for tc in m.get("tool_calls") or []) + used_air = any(tc["function"]["name"] == "air_quality" for m in added for tc in m.get("tool_calls") or []) air = self.by_name.get("AirGradient") markup = templates.keyboard(self.irrigation is not None) if self._keyboard_stale(msg) else None - refresh = ok and read.report and not photos and 0 < len(reply) <= TG_LIMIT and not markup # the report, in one message, to be refreshed in place - row =periods.keyboard(intent.period_days(text, now)) if len(photos) == 1 else None # period buttons under a single chart - sent_photo, drawn = [], list(photos) - kept = self.images.key(text, intent.period_days(text, now), now) if row and ok and not reply.strip() else None - if kept and not ready: # a chart and nothing else: keep it for the period buttons (and, its words being its key, for the next to ask them) + refresh = answer.ok and read.report and not photos and 0 < len(reply) <= TG_LIMIT and not markup # the report, in one message, to be refreshed in place + row = periods.keyboard(intent.period_days(text, now)) if len(photos) == 1 else None # period buttons under a single chart + drawn = list(photos) + kept = self.images.key(text, intent.period_days(text, now), now) if row and answer.ok and not reply.strip() else None + if kept and not answer.ready: # a chart and nothing else: keep it for the period buttons (and, its words being its key, for the next to ask them) self.images.put(kept, photos[0], titles[0], caption if kept == self.images.canonical(text, now) else None) try: began = time.monotonic() if redraw is not None and row and await self._redraw(redraw, reply, photos[0], titles[0], row, text, kept): reply, photos, row = "", [], None - carried_keyboard = await deliver(msg, reply, photos, link=air.link if air and used_air else None, - markup=markup or (report_menu.keyboard() if refresh else None), - titles=titles, period_row=row, sent_photo=sent_photo, - bold_headings=ok and bool(read.report or read.lookup in ("pollen", "forecast"))) and bool(markup) # the refresh button is not the chat's keyboard - if ready: - log.debug("Chart ms: drawn %.0fs ago, send %.0f%s", self.images.age(ready), (time.monotonic() - began) * 1000, - " (Telegram's copy)" if isinstance(drawn[0], str) else "") + sent = await deliver(msg, reply, photos, link=air.link if air and used_air else None, + markup=markup or (report_menu.keyboard() if refresh else None), titles=titles, period_row=row, + bold_headings=answer.ok and bool(read.report or read.lookup in ("pollen", "forecast"))) + carried_keyboard = sent.carried and bool(markup) # the refresh button is not the chat's keyboard + if answer.ready: + log.debug("Chart ms: drawn %.0fs ago, send %.0f%s", self.images.age(answer.ready), (time.monotonic() - began) * 1000, + " (Telegram's copy)" if isinstance(drawn[0], str) else "") elif drawn: - log.debug("Chart ms: %s", chart_times(turn.times, made, drawn, time.monotonic() - began)) - if sent_photo: - self.charted.remember(msg.chat_id, sent_photo[0].message_id, text) - self.images.sent(kept, sent_photo[0]) - if kept and read.fast and read.chart_in_code: # a chart with period buttons, made in code: its other periods are uploaded ahead - self.preload_soon(context.bot, text) - if self.state and ok and turn.forecast_shown: # it was sent: a later revision of these days is worth telling this chat - self.state.record_forecast(msg.chat_id, turn.forecast_shown) + log.debug("Chart ms: %s", chart_times(turn.times, answer.made, drawn, time.monotonic() - began)) + if sent.photo: + self.charted.remember(msg.chat_id, sent.photo.message_id, text) + self.images.sent(kept, sent.photo) + if kept and read.chart_in_code: # a chart with period buttons, made in code: its other periods are uploaded ahead + self.preload_soon(bot, text) + if answer.ok and turn.forecast_shown: # it was sent: a later revision of these days is worth telling this chat + self.state.record_forecast(msg.chat_id, turn.forecast_shown, now_local(self.cfg.tz).date()) if carried_keyboard: log.info("Buttons: sent keyboard %s to chat %s (it had %s)", self.keyboard_version, msg.chat_id, self.state.keyboard(msg.chat_id)) self.state.set_keyboard(msg.chat_id, self.keyboard_version) diff --git a/lib/buttons.py b/lib/buttons.py index 5a23e5d..11c109a 100644 --- a/lib/buttons.py +++ b/lib/buttons.py @@ -1,4 +1,4 @@ -"""Shared look of the bot's inline buttons: a plain button, a menu opener (▸ closed, ▾ open) and a toggle (green while on, struck +"""Shared look of the bot's inline buttons: a plain button, a menu opener (▸ closed, ▾ open) and a toggle (blue while on, struck through while off). Every menu builds its buttons with these.""" from collections.abc import Callable @@ -23,6 +23,6 @@ def struck(text: str) -> str: def toggle_button(word: str, on: bool, data: Callable[[bool], str]) -> InlineKeyboardButton: - """A button that flips a setting: its word on a green button while the setting is on, the word struck through on a plain button while it + """A button that flips a setting: its word on a blue button while the setting is on, the word struck through on a plain button while it is off. `data(turn_on)`: the callback data of pressing it, which sets the setting to `turn_on` (the opposite of how it is now).""" - return button(word if on else struck(word), data(not on), KeyboardButtonStyle.SUCCESS if on else None) + return button(word if on else struck(word), data(not on), KeyboardButtonStyle.PRIMARY if on else None) diff --git a/lib/charts.py b/lib/charts.py index 934faa0..a8dc5ce 100644 --- a/lib/charts.py +++ b/lib/charts.py @@ -15,26 +15,24 @@ import threading import time from dataclasses import dataclass -from datetime import tzinfo +from datetime import UTC, datetime, tzinfo from statistics import fmean import matplotlib - -matplotlib.use("Agg") # no display on the server -import matplotlib.dates as mdates # noqa: E402 -import matplotlib.pyplot as plt # noqa: E402 -import numpy as np # noqa: E402 -from matplotlib import font_manager # noqa: E402 -from matplotlib.collections import PatchCollection, PolyCollection # noqa: E402 -from matplotlib.colors import to_rgb # noqa: E402 -from matplotlib.lines import Line2D # noqa: E402 -from matplotlib.patches import Polygon, Rectangle, Wedge # noqa: E402 -from matplotlib.ticker import FixedLocator, FuncFormatter, MaxNLocator # noqa: E402 -from matplotlib.transforms import ScaledTranslation # noqa: E402 -from PIL import Image # noqa: E402 - -from .specs import Chart, Compass, Line, Panel # noqa: E402 -from .timeutil import to_local # noqa: E402 +import matplotlib.dates as mdates +import numpy as np +from matplotlib import font_manager +from matplotlib.backends.backend_agg import FigureCanvasAgg +from matplotlib.collections import PatchCollection, PolyCollection +from matplotlib.colors import to_rgb +from matplotlib.figure import Figure +from matplotlib.lines import Line2D +from matplotlib.patches import Rectangle, Wedge +from matplotlib.ticker import FixedLocator, FuncFormatter, MaxNLocator +from matplotlib.transforms import ScaledTranslation +from PIL import Image + +from .specs import Chart, Compass, Line, Panel # Palette. Slate neutrals for the furniture; one hue per reading (weather and air quality, kept clear of the green/yellow/red # rating zones); the same reading indoors is its complementary hue (opposite on the colour wheel, as a painter pairs them). @@ -64,10 +62,19 @@ AX_RECT = [0.075, 0.13, 0.905, 0.64] # left, bottom, width, height (figure fraction) of a single chart PANEL_IN = 1.1 # each panel past two adds this much height (inches) HEAD_IN, FOOT_IN = 0.9, 0.47 # room above and below the panels (inches) +TITLE_IN, SUBTITLE_IN = 0.27, 0.52 # the headline's lines, centred this far below the top (the colour key beside a big chart's title too) +ROSE_RECT = [0.685, 0.12, 0.27, 0.62] # the wind rose beside a wind chart (figure fraction); the line's plot ends before it, at: +BESIDE_ROSE_WIDTH = 0.57 +BAR_FILL = 0.85 # a bar fills this share of its slot; the rest is the gap to the next LEADER_CLEARANCE = 0.05 # peak labels sit at least this share of the panel's width away from the peaks (shorter = closer) BARS_SHARE = 0.95 # rain behind a line is drawn over the panel's full height (the tallest bar nearly reaches the top), under the line +def _bar_width(seconds: float) -> float: + """A bar's drawn width on the date axis, for bars `seconds` apart.""" + return seconds / 86400 * BAR_FILL + + @dataclass(frozen=True) class Look: """How large the parts of a panel of lines are drawn. A chart of one panel is drawn BIG; the panels of a stack SMALL.""" @@ -89,13 +96,15 @@ class Look: in_headline: bool # the panel's name and unit are the figure's headline, and its colour key sits beside it -BIG = Look(30, 1.5, 8.5, 18, 1.2, 9, (0.35, 0.8), 6, 17, 0.26, 10, 1, True, 5, False, True) -SMALL = Look(20, 1.2, 7.5, 12, 0.8, 5, (0.25, 0.6), 4, 14, 0.3, 8.5, 2, False, 4, True, False) +BIG = Look(end_dot=30, end_rim=1.5, pill_font=8.5, pill_dot=18, pill_rim=1.2, pill_lift=9, pill_box=(0.35, 0.8), pill_z=6, pitch=17, + pad=0.26, legend_font=10, key_from=1, units=True, nbins=5, card=False, in_headline=True) +SMALL = Look(end_dot=20, end_rim=1.2, pill_font=7.5, pill_dot=12, pill_rim=0.8, pill_lift=5, pill_box=(0.25, 0.6), pill_z=4, pitch=14, + pad=0.3, legend_font=8.5, key_from=2, units=False, nbins=4, card=True, in_headline=False) def _look_for(chart: Chart) -> Look: """A chart of one panel of lines is drawn BIG; anything else (a stack, rain alone) SMALL.""" - return BIG if len(chart.panels) == 1 and chart.panels[0].lines else SMALL + return BIG if chart.single else SMALL def _setup_fonts() -> str: @@ -113,7 +122,7 @@ def _setup_fonts() -> str: installed = {f.name for f in font_manager.fontManager.ttflist} family = next((n for n in ("Inter", "Roboto", "Helvetica Neue", "Arial", "Liberation Sans") if n in installed), "DejaVu Sans") - plt.rcParams.update({"font.family": family, "axes.unicode_minus": False}) + matplotlib.rcParams.update({"font.family": family, "axes.unicode_minus": False}) return family @@ -153,6 +162,12 @@ def _box(colour: str, pad: float = 0.25, rounding: float = 0.6) -> dict: return {"boxstyle": f"round,pad={pad},rounding_size={rounding}", "fc": colour, "ec": "none"} +def _pill(ax, text: str, at: tuple[float, float], colour: str, look: "Look", **place): + """A value on a pill of its line's colour, in the chart's look; `place` puts it (xytext, textcoords, ha, va, a leader ...).""" + ax.annotate(text, at, fontsize=look.pill_font, fontweight="bold", color=_ink_on(colour), bbox=_box(colour, *look.pill_box), + zorder=look.pill_z + 1, **place) + + def _dot(ax, x: float, y: float, colour: str, size: float = 12, edge: float = 0.8, z: int = 4): """A small dot with a white rim, marking a point on a line (`size` is its area in pt², as scatter takes it). It is a one-point line's marker: drawn just as a one-point scatter is, at a fraction of the cost.""" @@ -160,15 +175,13 @@ def _dot(ax, x: float, y: float, colour: str, size: float = 12, edge: float = 0. markeredgewidth=edge, zorder=z)) -def _to_dt(tz: tzinfo): - """A function turning an epoch into a naive local datetime, for matplotlib's date axis.""" - return lambda t: to_local(t, tz).replace(tzinfo=None) +EPOCH_NUM = float(mdates.date2num(datetime(1970, 1, 1, tzinfo=UTC))) # matplotlib's date number of epoch 0 -def _nums(tz: tzinfo, epochs) -> np.ndarray: - """Epochs as matplotlib date numbers (local time).""" - to_dt = _to_dt(tz) - return mdates.date2num([to_dt(t) for t in epochs]) +def _nums(epochs) -> np.ndarray: + """Epochs as matplotlib date numbers: real time, so a day of 23 or 25 hours (daylight saving) is drawn that long. The ticks + and their labels are local (_time_axis).""" + return np.fromiter(epochs, dtype=float) / 86400 + EPOCH_NUM def _end_of(bx, width: float, default: float) -> float: @@ -176,16 +189,14 @@ def _end_of(bx, width: float, default: float) -> float: return float(bx.max() + width / 86400) if len(bx) else default -def _style_axis(ax, size: float = 7.5, pad: float = 5, grid: float = 0.8, below: bool = True): +def _style_axis(ax): """The plain look shared by every chart: no frame but the baseline, faint horizontal grid, small muted ticks.""" for side in ("top", "right", "left"): ax.spines[side].set_visible(False) ax.spines["bottom"].set_color(AXIS) - ax.tick_params(axis="both", length=0, labelsize=size, labelcolor=MUTED, pad=pad) - if grid: - ax.grid(axis="y", color=GRID, linewidth=grid) - if below: - ax.set_axisbelow(True) + ax.tick_params(axis="both", length=0, labelsize=7.5, labelcolor=MUTED, pad=5) + ax.grid(axis="y", color=GRID, linewidth=0.8) + ax.set_axisbelow(True) def _shade_zones(ax, zones, ybottom: float, ytop: float, alpha: float, zorder: float = 0): @@ -196,7 +207,7 @@ def _shade_zones(ax, zones, ybottom: float, ytop: float, alpha: float, zorder: f ax.axhspan(max(lo, ybottom), min(hi, ytop), color=colour, alpha=alpha, linewidth=0, zorder=zorder) -def _pad_limits(ax, lo: float, hi: float, top: float = 0.2, bottom: float = 0.16, floor: float | None = None): +def _pad_limits(ax, lo: float, hi: float, top: float, bottom: float, floor: float | None): """Room above and below the data; `floor`: the axis does not go below it when the data does not (no negative wind).""" span = max(hi - lo, 1.0) y0 = lo - span * bottom @@ -208,14 +219,14 @@ def _extent(lines: list[Line], extra: list[float] = ()) -> tuple[float, float]: return (min([min(s.low or s.y) for s in lines] + list(extra)), max([max(s.high or s.y) for s in lines] + list(extra))) -def _draw_lines(ax, lines: list[Line], tz: tzinfo, look: Look, first: int = 0, reading: str = "") -> list[tuple]: +def _draw_lines(ax, lines: list[Line], look: Look, first: int = 0, reading: str = "") -> list[tuple]: """Each line's low-to-high range behind it, then the line (crisp, no glow or fill) and a dot on its end: [(colour, xs, ys)]. `first` is the colour index of the first line (the fallback colours cycle across a stack's panels).""" drawn = [] dense = max(len(s.x) for s in lines) > 200 for i, s in enumerate(lines, first): colour = _colour(s, i, reading) - xs, ys = _nums(tz, s.x), np.asarray(s.y, dtype=float) + xs, ys = _nums(s.x), np.asarray(s.y, dtype=float) if s.low: ax.fill_between(xs, s.low, s.high, color=colour, alpha=0.2, linewidth=0, zorder=3) w = 1.3 if len(lines) > 2 or s.low else 1.5 if dense else 2.2 # thinner where there are many lines, a band or many points @@ -240,17 +251,17 @@ def _draw_bars(ax, bx, ys, width: float, axis_top: float, zorder: int, alpha: fl brighter cap, so each reads as a little column. `kw` goes to both (for rain behind a line, the transform).""" ys = np.asarray(ys, dtype=float) _columns(ax, bx, ys, width, facecolors=_mix(RAIN, 1 - alpha), zorder=zorder, **kw) - cap = axis_top * 0.014 - _columns(ax, bx, np.minimum(cap, ys), width, np.maximum(ys - cap, 0), facecolors=RAIN, alpha=0.95, zorder=zorder, **kw) + cap = np.sign(ys) * np.minimum(axis_top * 0.014, np.abs(ys)) # at the bar's far end, below zero as above it + _columns(ax, bx, cap, width, ys - cap, facecolors=RAIN, alpha=0.95, zorder=zorder, **kw) -def _bars_behind(ax, bars, tz: tzinfo) -> float: +def _bars_behind(ax, bars) -> float: """Bars (rain) behind the panel's lines and over its grid, on a scale of their own read on the right: the tallest reaches BARS_SHARE of the panel's height. They are drawn in the panel itself (x on its time axis, y as a share of its height), and the scale's two labels sit where a right-hand axis would put them. Returns where the last bar ends (matplotlib date number).""" - bx = _nums(tz, bars.x) + bx = _nums(bars.x) top = max([*bars.y, 1.0]) - _draw_bars(ax, bx, np.asarray(bars.y, dtype=float) * (BARS_SHARE / top), bars.width / 86400 * 0.85, 1.0, 1, + _draw_bars(ax, bx, np.asarray(bars.y, dtype=float) * (BARS_SHARE / top), _bar_width(bars.width), 1.0, 1, transform=ax.get_xaxis_transform()) ax.add_line(Line2D([0, 1], [BARS_SHARE, BARS_SHARE], transform=ax.transAxes, color=RAIN, linewidth=0.7, linestyle=(0, (1, 2)), alpha=0.8, zorder=1)) # the top of the rain scale, on its own @@ -272,7 +283,7 @@ def _axes_width(chart: Chart) -> float: def _headline(fig, title: str, subtitle: str, height: float, unit: str = ""): """Title and subtitle at the top left; a unit that is not a degree goes in the title (ticks stay plain numbers).""" title = f"{title} ({unit})" if unit and _deg(unit) != "°" else title - head = fig.text(AX_RECT[0], 1 - 0.27 / height, title, fontsize=13, fontweight=TITLE_WEIGHT, color=TEXT, va="center") + head = fig.text(AX_RECT[0], 1 - TITLE_IN / height, title, fontsize=13, fontweight=TITLE_WEIGHT, color=TEXT, va="center") room = AX_RECT[2] * fig.get_figwidth() * fig.dpi # the width of the plots below renderer = fig.canvas.get_renderer() names = title.split(", ") @@ -283,43 +294,46 @@ def _headline(fig, title: str, subtitle: str, height: float, unit: str = ""): while head.get_window_extent(renderer).width > room and kept > 2: # a long list of readings: the first few "and N more" kept -= 1 head.set_text(", ".join(names[:kept]) + f" and {len(names) - kept} more") - fig.text(AX_RECT[0], 1 - 0.52 / height, subtitle, fontsize=8.5, color=MUTED, va="center") + fig.text(AX_RECT[0], 1 - SUBTITLE_IN / height, subtitle, fontsize=8.5, color=MUTED, va="center") -def _time_axis(ax, span_days: float, bars=None, tz: tzinfo | None = None): +def _time_axis(ax, span_days: float, tz: tzinfo, bars=None): """Tick positions and labels for a date axis, chosen by the length of the period. Bars of whole months or years (a count per - month or year) are labelled under the middle of each bar instead.""" - if bars and tz and bars.per in ("year", "month") and bars.x and (bars.per == "year" or len(bars.x) <= 14): - centres = _nums(tz, bars.x) + bars.width / 86400 * 0.85 / 2 + month or year) are labelled under the middle of each bar instead. The ticks fall on local hours, days and months.""" + if bars and bars.per in ("year", "month") and bars.x and (bars.per == "year" or len(bars.x) <= 14): + centres = _nums(bars.x) + _bar_width(bars.width) / 2 fmt = "%Y" if bars.per == "year" else "%b\n'%y" ax.xaxis.set_major_locator(FixedLocator(list(centres))) - ax.xaxis.set_major_formatter(FuncFormatter(lambda v, _: mdates.num2date(v).strftime(fmt))) + ax.xaxis.set_major_formatter(FuncFormatter(lambda v, _: mdates.num2date(v, tz).strftime(fmt))) return if span_days <= 1.1: - ax.xaxis.set_major_locator(mdates.HourLocator(byhour=range(0, 24, 3))) + ax.xaxis.set_major_locator(mdates.HourLocator(byhour=range(0, 24, 3), tz=tz)) fmt = "%-I%p" elif span_days <= 4: - ax.xaxis.set_major_locator(mdates.HourLocator(byhour=[0, 12])) + ax.xaxis.set_major_locator(mdates.HourLocator(byhour=[0, 12], tz=tz)) fmt = "%a %-I%p" elif span_days <= 45: - ax.xaxis.set_major_locator(mdates.DayLocator(interval=max(1, math.ceil(span_days / 8)))) + ax.xaxis.set_major_locator(mdates.DayLocator(interval=max(1, math.ceil(span_days / 8)), tz=tz)) fmt = "%a\n%-d %b" elif span_days <= 400: - ax.xaxis.set_major_locator(mdates.MonthLocator()) + ax.xaxis.set_major_locator(mdates.MonthLocator(tz=tz)) fmt = "%b\n'%y" else: - ax.xaxis.set_major_locator(mdates.MonthLocator(bymonth=[1, 4, 7, 10])) + ax.xaxis.set_major_locator(mdates.MonthLocator(bymonth=[1, 4, 7, 10], tz=tz)) fmt = "%b\n'%y" ax.xaxis.set_major_formatter(FuncFormatter( - lambda v, _: mdates.num2date(v).strftime(fmt).replace("AM", "am").replace("PM", "pm"))) + lambda v, _: mdates.num2date(v, tz).strftime(fmt).replace("AM", "am").replace("PM", "pm"))) -def _legend_dots(ax_or_fig, colours: list[str], labels: list[str], **kw): - handles = [Line2D([], [], marker="o", linestyle="", markersize=6, color=c) for c in colours] - ax_or_fig.legend(handles, labels, frameon=False, labelcolor=TEXT, handletextpad=0.2, columnspacing=1.1, **kw) +def _legend(where, colours: list[str], labels: list[str], marker: str = "o", size: float = 6, ink: str = TEXT, spacing: float = 1.1, + **kw): + """A row of coloured markers (dots; squares for shares and the rose's speeds), each with its name, unframed: `where` is the + axes or the figure, `kw` where it goes and its font.""" + handles = [Line2D([], [], marker=marker, linestyle="", markersize=size, color=c) for c in colours] + where.legend(handles, labels, frameon=False, labelcolor=ink, handletextpad=0.2, columnspacing=spacing, **kw) -def _pills(ax, lines: list[Line], drawn: list[tuple], tz: tzinfo, x0: float, x1: float, look: Look, deg: str = ""): +def _pills(ax, lines: list[Line], drawn: list[tuple], x0: float, x1: float, look: Look, deg: str = ""): """A dot on the highest and the lowest point of each line with a pill of its value above or below it. The point is the top (bottom) of the line's band when it has one, else its true record at its actual time (a dotted stem joins it to a line that does not reach it). A low on the floor says nothing and is not labelled; pills that land close @@ -332,7 +346,7 @@ def _pills(ax, lines: list[Line], drawn: list[tuple], tz: tzinfo, x0: float, x1: if want in s.records and s.low is not None: # a banded line: the label sits on the top (bottom) of its band rx, ry = _extreme(s, want, xs) elif want in s.records: # the true record, at its actual time (may sit off an averaged line) - rx, ry = float(_nums(tz, [s.records[want][0]])[0]), float(s.records[want][1]) + rx, ry = float(_nums([s.records[want][0]])[0]), float(s.records[want][1]) rx = min(max(rx, x0), x1) else: # only some records given (wind: the strongest gust, no lowest): no label for the other continue @@ -358,9 +372,8 @@ def _pills(ax, lines: list[Line], drawn: list[tuple], tz: tzinfo, x0: float, x1: else: left[5], right[5] = "right", "left" for rx, ry, text, colour, above, ha in pills: - ax.annotate(text, (rx, ry), xytext=(0, look.pill_lift if above else -look.pill_lift), textcoords="offset points", ha=ha, - va="bottom" if above else "top", fontsize=look.pill_font, fontweight="bold", color=_ink_on(colour), - bbox=_box(colour, *look.pill_box), zorder=look.pill_z + 1) + _pill(ax, text, (rx, ry), colour, look, xytext=(0, look.pill_lift if above else -look.pill_lift), textcoords="offset points", + ha=ha, va="bottom" if above else "top") def _rose_radius(outer: np.ndarray) -> float: @@ -379,7 +392,7 @@ def _render_rose(fig, compass: Compass): way, stacked by wind speed. It sits beside the wind speed line. The wedges are one collection in the polar axes' own units (the centre at 0.5, 0.5 and the edge at radius 0.5): 48 polar bars cost several times as much to make and draw.""" counts = np.asarray(compass.rose, dtype=float) # 16 compass points x 3 speed steps - ax = fig.add_axes([0.685, 0.12, 0.27, 0.62], projection="polar") + ax = fig.add_axes(ROSE_RECT, projection="polar") outer = np.cumsum(counts / max(counts.sum(), 1) * 100, axis=1) # where each step ends, out from the centre radius = _rose_radius(outer) ax.set_ylim(0, radius) # the grid's rings follow it @@ -399,9 +412,8 @@ def _render_rose(fig, compass: Compass): ax.set_facecolor(BG) if compass.speeds: a, b = compass.speed_steps - fig.legend([Line2D([], [], marker="s", linestyle="", markersize=6, color=c) for c in WIND_STEPS], - [f"under {a}", f"{a}–{b}", f"{b}+ km/h"], loc="lower right", bbox_to_anchor=(0.985, 0.0), ncol=3, - frameon=False, fontsize=8.5, labelcolor=MUTED, handletextpad=0.2, columnspacing=0.9) + _legend(fig, WIND_STEPS, [f"under {a}", f"{a}–{b}", f"{b}+ km/h"], marker="s", ink=MUTED, spacing=0.9, loc="lower right", + bbox_to_anchor=(0.985, 0.0), ncol=3, fontsize=8.5) # ---------- several panels on one time axis ---------- @@ -427,7 +439,7 @@ def _mark_highs(ax, lines: list[Line], drawn: list[tuple], x0: float, x1: float, mx, my = _extreme(line, "high", xs) peaks.append((mx, my, colour)) peaks.sort(key=lambda p: -p[1]) # the highest peak gets the top pill - size, pitch = look.pill_font, look.pitch # the pills' font and row spacing, in points + pitch = look.pitch # the pills' row spacing, in points groups = [] # peaks at about the same time share a column, stacked for peak in peaks: # (highest first) near = next((g for g in groups if abs(g[0][0] - peak[0]) < 0.05 * (x1 - x0)), None) @@ -454,19 +466,17 @@ def _mark_highs(ax, lines: list[Line], drawn: list[tuple], x0: float, x1: float, if len(columns) == len(groups): for g, cx in zip(groups, columns): for i, (mx, my, colour) in enumerate(g): - _dot(ax, mx, my, colour) - ax.annotate(f"{round(my, 1):g}", (mx, my), xytext=(cx, y_hi - (10 + pitch * i) * per_pt), textcoords="data", ha="center", - va="center", fontsize=size, fontweight="bold", color=_ink_on(colour), zorder=5, arrowprops=arrow(colour), - bbox=_box(colour)) + _dot(ax, mx, my, colour, look.pill_dot, look.pill_rim, look.pill_z) + _pill(ax, f"{round(my, 1):g}", (mx, my), colour, look, xytext=(cx, y_hi - (10 + pitch * i) * per_pt), textcoords="data", + ha="center", va="center", arrowprops=arrow(colour)) return placed = [] # no empty column: the right margin, at the peaks' heights for mx, my, colour in peaks: ly = min(my, placed[-1] - pitch * per_pt) if placed else my placed.append(ly) - _dot(ax, mx, my, colour) - ax.annotate(f"{round(my, 1):g}", (mx, my), xytext=(1.03, ly), textcoords=("axes fraction", "data"), ha="left", va="center", - fontsize=size, fontweight="bold", color=_ink_on(colour), zorder=5, annotation_clip=False, arrowprops=arrow(colour), - bbox=_box(colour)) + _dot(ax, mx, my, colour, look.pill_dot, look.pill_rim, look.pill_z) + _pill(ax, f"{round(my, 1):g}", (mx, my), colour, look, xytext=(1.03, ly), textcoords=("axes fraction", "data"), ha="left", + va="center", annotation_clip=False, arrowprops=arrow(colour)) def _figure(value: float) -> str: @@ -493,50 +503,63 @@ def _end_labels(ax, drawn: list[tuple], show_floor: bool = False): bbox={"boxstyle": "round,pad=0.15", "fc": "none", "ec": "none"}) -def _draw_panel(ax, p: Panel, tz: tzinfo, first: int, x0: float, x1: float, look: Look): - """One panel: lines (with bands and zones, peaks labelled, rain behind them), rain on its own, or shares. Returns the - number of colours used, where the drawing ends on the x axis and the legend's entries (colour, name).""" +def _draw_panel(ax, p: Panel, first: int, x0: float, x1: float, look: Look): + """One panel: shares, bars on their own (rain, counts), or lines (with bands and zones, peaks labelled, rain behind them). + Returns the number of colours used, where the drawing ends on the x axis and the legend's entries (colour, name).""" if p.shares: - s = p.shares # the share of each bar's time in good / poor / very poor: traffic-light bars stacked to 100% - bx = _nums(tz, s.x) - base = np.zeros(len(bx)) - for column, colour in zip((s.good, s.poor, s.very_poor), ZONE_COLOURS): - _columns(ax, bx, column, s.width / 86400 * 0.85, base, facecolors=colour, zorder=3) - base += np.asarray(column, dtype=float) - ax.set_ylim(0, 100) - ax.legend([Line2D([], [], marker="s", linestyle="", markersize=5, color=c) for c in ZONE_COLOURS], - ["good", "poor", "very poor"], loc="lower right", bbox_to_anchor=(1.0, 1.0), frameon=False, fontsize=8.5, - labelcolor=TEXT, ncol=3, handletextpad=0.2, columnspacing=0.9, borderaxespad=0.1) - return 0, _end_of(bx, s.width, x1), [] - if not p.lines: # rain on its own - b = p.bars - bx = _nums(tz, b.x) - top = max([*b.y, 1.0]) * 1.15 - bottom = min([*b.y, 0.0]) * 1.3 # a figure can be below zero (a coldest night) - _draw_bars(ax, bx, b.y, b.width / 86400 * 0.85, top, 3, alpha=0.75) - if b.values: # a few bars of counts or figures: the number at the end of each - for x, y in zip(bx, b.y): - ax.text(x + b.width / 86400 * 0.85 / 2, y + (top - bottom) * (0.015 if y >= 0 else -0.015), f"{y:g}", ha="center", - va="bottom" if y >= 0 else "top", fontsize=8, color=TEXT, zorder=5) - ax.set_ylim(bottom, top) - return 0, _end_of(bx, b.width, x1), [] + return _draw_shares(ax, p.shares, x1) + if p.bars_only: + return _draw_bars_alone(ax, p.bars, x1) + return _draw_readings(ax, p, first, x0, x1, look) + + +def _draw_shares(ax, s, x1: float): + """The share of each bar's time in good / poor / very poor: traffic-light bars stacked to 100%.""" + bx = _nums(s.x) + base = np.zeros(len(bx)) + for column, colour in zip((s.good, s.poor, s.very_poor), ZONE_COLOURS): + _columns(ax, bx, column, _bar_width(s.width), base, facecolors=colour, zorder=3) + base += np.asarray(column, dtype=float) + ax.set_ylim(0, 100) + _legend(ax, ZONE_COLOURS, ["good", "poor", "very poor"], marker="s", size=5, spacing=0.9, loc="lower right", + bbox_to_anchor=(1.0, 1.0), fontsize=8.5, ncol=3, borderaxespad=0.1) + return 0, _end_of(bx, s.width, x1), [] + + +def _draw_bars_alone(ax, b, x1: float): + """Rain on its own, or a count or figure per month or year (a few bars of them get their number on the end).""" + bx = _nums(b.x) + top = max([*b.y, 1.0]) * 1.15 + bottom = min([*b.y, 0.0]) * 1.3 # a figure can be below zero (a coldest night) + _draw_bars(ax, bx, b.y, _bar_width(b.width), top, 3, alpha=0.75) + if b.values: + for x, y in zip(bx, b.y): + ax.text(x + _bar_width(b.width) / 2, y + (top - bottom) * (0.015 if y >= 0 else -0.015), f"{y:g}", ha="center", + va="bottom" if y >= 0 else "top", fontsize=8, color=TEXT, zorder=5) + ax.set_ylim(bottom, top) + return 0, _end_of(bx, b.width, x1), [] + + +def _draw_readings(ax, p: Panel, first: int, x0: float, x1: float, look: Look): + """The panel's lines, each with its band, the rating zones of a rated reading behind it, the peaks labelled (as pills on the + lines, or aside), rain behind them on a scale of its own, and the latest values beside them.""" lines = p.lines marked = [i for i, s in enumerate(lines) if s.records and p.peaks != "none"] pilled = bool(marked) and p.peaks == "pills" _pad_limits(ax, *_extent(lines, [float(r[1]) for s in lines for r in s.records.values()] if pilled else []), top=look.pad if pilled else 0.18 if marked else 0.12, bottom=look.pad if pilled else 0.12, - floor=0) + floor=None if _deg(p.unit) == "°" else 0) # a temperature can go below zero; nothing else drawn as a line can ybottom, ytop = ax.get_ylim() - drawn = _draw_lines(ax, lines, tz, look, first, p.reading) + drawn = _draw_lines(ax, lines, look, first, p.reading) if p.zones: # a rated reading: its good / poor / very poor zones behind the line (and over any rain behind it, as always) _shade_zones(ax, p.zones, ybottom, ytop, 0.07, 1.5 if p.bars else 0) ax.set_ylim(ybottom, ytop) if marked and p.peaks == "aside": _mark_highs(ax, [lines[i] for i in marked], [drawn[i] for i in marked], x0, x1, look) elif marked: - _pills(ax, [lines[i] for i in marked], [drawn[i] for i in marked], tz, x0, x1, look, _deg(p.unit)) + _pills(ax, [lines[i] for i in marked], [drawn[i] for i in marked], x0, x1, look, _deg(p.unit)) if p.bars: - x1 = max(x1, _bars_behind(ax, p.bars, tz)) + x1 = max(x1, _bars_behind(ax, p.bars)) if len(lines) > 1: _end_labels(ax, drawn) elif p.reading in END_LABELLED and drawn[0][2][-1] != 0: # a latest value of zero (night, calm) has nothing to say @@ -545,60 +568,56 @@ def _draw_panel(ax, p: Panel, tz: tzinfo, first: int, x0: float, x1: float, look return len(lines), x1, entries -def _render(chart: Chart, tz: tzinfo, times: dict | None = None) -> bytes: +def _build(chart: Chart, tz: tzinfo) -> Figure: """The panels top to bottom on one time axis, so the rain (or another reading) lines up with what the others were doing. A chart of one panel of lines is the same thing drawn BIG (and with its wind rose beside it, when it has one).""" - started = time.perf_counter() panels, n = chart.panels, len(chart.panels) look = _look_for(chart) height = H_IN if n <= 2 else H_IN + PANEL_IN * (n - 2) - fig = plt.figure(figsize=(W_IN, height), dpi=DPI, facecolor=BG) - try: - _headline(fig, panels[0].label if look.in_headline else chart.title, chart.subtitle, height, panels[0].unit if look.in_headline else "") - width = _axes_width(chart) - body = height - HEAD_IN - FOOT_IN - gap = 0.27 if n > 2 else 0.22 - each = (body - gap * (n - 1)) / n - xs = _nums(tz, {t for p in panels for t in p.xs}) # each time once: only the range is wanted - x0, x1 = float(xs.min()), float(xs.max()) - axes, used = [], 0 - for i, p in enumerate(panels): - bottom = (FOOT_IN + (n - 1 - i) * (each + gap)) / height - rect = [AX_RECT[0], bottom, width, each / height] - if look.card: # the faint tint behind the panel: a rectangle on the figure (an axes of its own costs far more) - fig.add_artist(Rectangle(rect[:2], rect[2], rect[3], transform=fig.transFigure, facecolor=CARD, edgecolor="none", - linewidth=0, zorder=-1)) - ax = fig.add_axes(rect, facecolor="none" if look.card else BG, sharex=axes[0] if axes else None) - axes.append(ax) - _style_axis(ax) - count, x1, entries = _draw_panel(ax, p, tz, used, x0, x1, look) - used += count - if len(entries) >= look.key_from: # the colour key: beside the headline for a big chart, above the panel in a stack - key = dict(loc="center right", bbox_to_anchor=(AX_RECT[0] + AX_RECT[2], 1 - 0.27 / height)) if look.in_headline else \ - dict(loc="lower right", bbox_to_anchor=(1.0, 1.0), borderaxespad=0.1) - _legend_dots(fig if look.in_headline else ax, [c for c, _ in entries], [label for _, label in entries], ncol=len(entries), - fontsize=look.legend_font, **key) - deg = _deg(p.unit) - tick_unit = deg if deg == "°" else "" - ax.yaxis.set_major_locator(MaxNLocator(nbins=3 if n > 2 else look.nbins, steps=[1, 2, 2.5, 5, 10], - integer=bool(p.bars and not p.lines and p.bars.unit == "days"))) - ax.yaxis.set_major_formatter(FuncFormatter(lambda v, _, u=tick_unit: f"{v:g}{u}")) - if not look.in_headline: - title = f"{p.label} ({p.unit})" if p.unit and not tick_unit else p.label - ax.set_title(title, loc="left", fontsize=8.5, fontweight=TITLE_WEIGHT, color=TEXT, pad=4, y=1.0) # y: placed, not searched for - if i < n - 1: - ax.xaxis.set_visible(False) # the time is read off the bottom panel (these ticks have no length and no grid) - margin = (x1 - x0) * 0.015 # room for the end dots - axes[0].set_xlim(x0 - margin, x1 + margin) - _time_axis(axes[-1], x1 - x0, panels[-1].bars if not panels[-1].lines else None, tz) - if chart.compass: # beside the wind line - pos = axes[0].get_position() - axes[0].set_position([pos.x0, pos.y0, 0.57, pos.height]) - _render_rose(fig, chart.compass) - except Exception: - plt.close(fig) - raise - return _png(fig, times, started) + fig = Figure(figsize=(W_IN, height), dpi=DPI, facecolor=BG) + FigureCanvasAgg(fig) # drawn on its own canvas: no pyplot, whose figures would need closing + _headline(fig, panels[0].label if look.in_headline else chart.title, chart.subtitle, height, panels[0].unit if look.in_headline else "") + width = _axes_width(chart) + body = height - HEAD_IN - FOOT_IN + gap = 0.27 if n > 2 else 0.22 + each = (body - gap * (n - 1)) / n + xs = _nums({t for p in panels for t in p.xs}) # each time once: only the range is wanted + x0, x1 = float(xs.min()), float(xs.max()) + axes, used = [], 0 + for i, p in enumerate(panels): + bottom = (FOOT_IN + (n - 1 - i) * (each + gap)) / height + rect = [AX_RECT[0], bottom, width, each / height] + if look.card: # the faint tint behind the panel: a rectangle on the figure (an axes of its own costs far more) + fig.add_artist(Rectangle(rect[:2], rect[2], rect[3], transform=fig.transFigure, facecolor=CARD, edgecolor="none", + linewidth=0, zorder=-1)) + ax = fig.add_axes(rect, facecolor="none" if look.card else BG, sharex=axes[0] if axes else None) + axes.append(ax) + _style_axis(ax) + count, x1, entries = _draw_panel(ax, p, used, x0, x1, look) + used += count + if len(entries) >= look.key_from: # the colour key: beside the headline for a big chart, above the panel in a stack + key = dict(loc="center right", bbox_to_anchor=(AX_RECT[0] + AX_RECT[2], 1 - TITLE_IN / height)) if look.in_headline else \ + dict(loc="lower right", bbox_to_anchor=(1.0, 1.0), borderaxespad=0.1) + _legend(fig if look.in_headline else ax, [c for c, _ in entries], [label for _, label in entries], ncol=len(entries), + fontsize=look.legend_font, **key) + deg = _deg(p.unit) + tick_unit = deg if deg == "°" else "" + ax.yaxis.set_major_locator(MaxNLocator(nbins=3 if n > 2 else look.nbins, steps=[1, 2, 2.5, 5, 10], + integer=p.bars_only and p.bars.unit == "days")) + ax.yaxis.set_major_formatter(FuncFormatter(lambda v, _, u=tick_unit: f"{v:g}{u}")) + if not look.in_headline: + title = f"{p.label} ({p.unit})" if p.unit and not tick_unit else p.label + ax.set_title(title, loc="left", fontsize=8.5, fontweight=TITLE_WEIGHT, color=TEXT, pad=4, y=1.0) # y: placed, not searched for + if i < n - 1: + ax.xaxis.set_visible(False) # the time is read off the bottom panel (these ticks have no length and no grid) + margin = (x1 - x0) * 0.015 # room for the end dots + axes[0].set_xlim(x0 - margin, x1 + margin) + _time_axis(axes[-1], x1 - x0, tz, panels[-1].bars if panels[-1].bars_only else None) + if chart.compass: # beside the wind line + pos = axes[0].get_position() + axes[0].set_position([pos.x0, pos.y0, BESIDE_ROSE_WIDTH, pos.height]) + _render_rose(fig, chart.compass) + return fig def _palette(rgb: np.ndarray, counts: np.ndarray, size: int = 256) -> tuple[list[int], np.ndarray]: @@ -632,37 +651,38 @@ def distance_to(k: int) -> np.ndarray: # every colour's squared distance to co return taken, nearest -def _png(fig, times: dict | None = None, started: float = 0.0) -> bytes: - """The figure as a PNG of at most 256 colours (see _palette): well under half the size of a full-colour one, and quicker to - write. A chart has a few thousand colours, nearly all of them the in-between shades on the edges of lines and letters. - times: the seconds since `started` (the clock when the figure was begun) are added to its "draw", and the PNG's to its "png".""" - try: - fig.canvas.draw() - drawn = time.perf_counter() - width, height = fig.canvas.get_width_height(physical=True) - pixels = np.frombuffer(fig.canvas.buffer_rgba(), "> 8 & 255, colours >> 16], axis=1) - taken, nearest = _palette(rgb, counts) - lookup = np.empty(1 << 24, np.uint8) # each colour's place in the palette - lookup[colours] = nearest - image = Image.frombuffer("P", (width, height), lookup[pixels], "raw", "P", 0, 1) - image.putpalette(rgb[taken].astype(np.uint8).tobytes()) - buf = io.BytesIO() - image.save(buf, format="png", dpi=(DPI, DPI)) - if times is not None: - times["draw"] = times.get("draw", 0.0) + drawn - started - times["png"] = times.get("png", 0.0) + time.perf_counter() - drawn - return buf.getvalue() - finally: - plt.close(fig) - - -_DRAWING = threading.Lock() # pyplot keeps global state (current figure, rcParams): one chart is drawn at a time +def _png(pixels: np.ndarray, width: int, height: int) -> bytes: + """The picture (each pixel as one number: red + green·256 + blue·65536) as a PNG of at most 256 colours (see _palette): well + under half the size of a full-colour one, and quicker to write. A chart has a few thousand colours, nearly all of them the + in-between shades on the edges of lines and letters.""" + colours, counts = np.unique(pixels, return_counts=True) + rgb = np.stack([colours & 255, colours >> 8 & 255, colours >> 16], axis=1) + taken, nearest = _palette(rgb, counts) + lookup = np.empty(1 << 24, np.uint8) # each colour's place in the palette: quicker than np.unique's return_inverse (a sort of every pixel) + lookup[colours] = nearest + image = Image.frombuffer("P", (width, height), lookup[pixels], "raw", "P", 0, 1) + image.putpalette(rgb[taken].astype(np.uint8).tobytes()) + buf = io.BytesIO() + image.save(buf, format="png", dpi=(DPI, DPI)) + return buf.getvalue() + + +# matplotlib's fonts and text layout are shared between threads: one chart at a time. Its PNG is written under the lock too: with +# Python's one-thread-at-a-time interpreter, writing it beside the next chart's drawing only slows both. +_DRAWING = threading.Lock() def render(chart: Chart, tz: tzinfo, times: dict | None = None) -> bytes: """PNG bytes for one chart (1280x720; a stack of more than two panels is taller). Safe to call from several threads. times (a Turn's): the seconds spent drawing the figure and writing the PNG are added to its "draw" and "png".""" + started = time.perf_counter() with _DRAWING: - return _render(chart, tz, times) + fig = _build(chart, tz) + fig.canvas.draw() + drawn = time.perf_counter() + width, height = fig.canvas.get_width_height(physical=True) + png = _png(np.frombuffer(fig.canvas.buffer_rgba(), " (group, field, label, unit) SCAN_WEATHER = {**{n: (WEATHER[n].group, WEATHER[n].field, WEATHER[n].label.lower(), WEATHER[n].unit) @@ -68,8 +67,7 @@ "type": "object", "properties": { "metric": {"type": "string", "enum": ["all", *SCAN_AIR], "description": "One air reading, or all (default)."}, - "start_date": {"type": "string", "description": "First day, 'YYYY-MM-DD'. Default: 90 days before the end."}, - "end_date": {"type": "string", "description": "Last day, 'YYYY-MM-DD' (up to yesterday). Default: yesterday."}, + **period_parameters(SCAN_DAYS), "chart": {"type": "boolean", "description": "Set true for a chart of the strongest relationship found."}, }, } @@ -84,24 +82,23 @@ "type": "object", "properties": { "metric": {"type": "string", "enum": list(LINK_METRICS), "description": "The air reading. Default pm2_5."}, - "start_date": {"type": "string", "description": "First day, 'YYYY-MM-DD'. Default: 60 days before the end."}, - "end_date": {"type": "string", "description": "Last day, 'YYYY-MM-DD' (up to yesterday). Default: yesterday."}, + **period_parameters(LINK_DAYS), "chart": {"type": "boolean", "description": "Set true for a chart: the reading (with its rating zones) over rain bars."}, }, } def _error(text: str) -> str: - return json.dumps({"error": text}, ensure_ascii=False) + return json_result({"error": text}) def rating_shares(values: dict[int, float], limits: tuple[float, float], tz: tzinfo, first: date, last: date, label: str) -> Shares: """Per bar (see bar_layout): the percentage of its 30-minute readings that were good, poor and very poor.""" - origin, width, per = bar_layout(tz, first, last) + starts, width, per = bar_layout(tz, first, last) counts: dict[int, list[int]] = {} for t, v in values.items(): - k = origin + (t - origin) // width * width - counts.setdefault(k, [0, 0, 0])[zone(v, limits)] += 1 + if (k := bar_of(starts, t)) is not None: + counts.setdefault(k, [0, 0, 0])[zone(v, limits)] += 1 xs = sorted(counts) total = {k: sum(counts[k]) for k in xs} good, poor, very_poor = ([round(100 * counts[k][i] / total[k], 1) for k in xs] for i in range(3)) @@ -121,22 +118,18 @@ def period(self, args: dict, default_days: int = DEFAULT_DAYS) -> tuple[date, da """(first, last) day, or an error text. The last day is at most yesterday: today is still settling.""" return parse_period(args, now_local(self.tz).date(), default_days) + def _span(self, first: date, last: date) -> tuple[int, int]: + return day_bounds(first, self.tz)[0], day_bounds(last, self.tz, last_second=True)[1] + def weather(self, group: str, field: str, first: date, last: date) -> dict[int, float]: """One reading per 30-minute slot from the weather station's cache.""" - return self.weather_band(group, field, first, last, bands=False)[0] - - def weather_band(self, group: str, field: str, first: date, last: date, bands: bool = True) -> tuple[dict, dict, dict]: - """(values, lows, highs) per 30-minute slot; lows and highs are Ecowitt's own where it gives them, else the cached 5-minute readings'.""" - lo, hi = day_bounds(first, self.tz)[0], day_bounds(last, self.tz, last_second=True)[1] - fields = [field, field + "_low", field + "_high"] if bands else [field] - values, lows, highs = [*self.eco.cache.slots(self.eco.mac, "30min", group, fields, lo, hi), {}, {}][:3] - if bands and derives_range(field) and not (lows or highs): # no range of its own: from the cached 5-minute readings - lows, highs = self.eco.cache.slot_ranges(self.eco.mac, group, field, values, lo, hi) - return values, lows, highs + return self.eco.cache.slots(self.eco.mac, "30min", group, [field], *self._span(first, last))[0] async def air_series(self, metrics: list[str], first: date, last: date) -> tuple[dict[str, tuple[dict, dict, dict]], dict]: """({metric: (values, lows, highs)}, notes): the sensor's readings averaged into 30-minute slots, with each slot's - range, from one load of the period.""" + range, from one load of the period (none when no metric is asked for).""" + if not metrics: + return {}, {} rows, hourly, skipped = await self.air.rows(datetime.combine(first, time()), datetime.combine(last, time(23, 59, 59))) out = {} for metric in metrics: @@ -153,11 +146,6 @@ async def air_series(self, metrics: list[str], first: date, last: date) -> tuple notes["resolution"] = f"{hourly} day(s) only exist as hourly averages" return out, notes - async def air_slots(self, metric: str, first: date, last: date) -> tuple[dict, dict, dict, dict]: - """(values, lows, highs, notes) for one air metric; see air_series.""" - series, notes = await self.air_series([metric], first, last) - return (*series[metric], notes) - async def plot_chart(self, args: dict, turn: Turn | None = None) -> str: turn = turn or Turn() panels = args.get("panels") @@ -166,21 +154,23 @@ async def plot_chart(self, args: dict, turn: Turn | None = None) -> str: if isinstance(period := self.period(args), str): return _error(period) first, last = period - built, summary, notes = [], [], {} - for p in panels: - name, style = p.get("series"), p.get("style") or ("bars" if p.get("series") == "rain" else "line") + asked = [(p.get("series"), p.get("style") or ("bars" if p.get("series") == "rain" else "line")) for p in panels] + for name, style in asked: if name not in SERIES or style not in STYLES: return _error(f"series must be one of {', '.join(SERIES)} and style one of {', '.join(STYLES)}") if (style == "bars") != (name == "rain") or (style == "rating" and name not in ALL_METRICS): return _error("'bars' is for rain only, and 'rating' for the air-quality series only; use 'line' otherwise") - panel, facts = await self._panel(name, style, first, last, notes) + air, notes = await self.air_series([name for name, _ in asked if name in ALL_METRICS], first, last) + built, summary = [], [] + for name, style in asked: + panel, facts = self._panel(name, style, first, last, air.get(name)) if panel is None: return _error(f"no cached readings of {name} for {first} to {last}") built.append(panel) summary.append(facts) out = {"period": f"{first} to {last}", "panels": summary, **({"notes": notes} if notes else {})} self._add_chart(out, built, first, last, turn) - return json.dumps(out, ensure_ascii=False, separators=(",", ":")) + return json_result(out) def _add_chart(self, out: dict, panels: list[Panel], first: date, last: date, turn: Turn): """Send the panels as one chart (rain behind the first line) and tell the model what its reply is for.""" @@ -196,10 +186,10 @@ async def air_link(self, args: dict, turn: Turn | None = None) -> str: if isinstance(period := self.period(args, LINK_DAYS), str): return _error(period) first, last = period - values, lows, highs, notes = await self.air_slots(metric, first, last) - air_panel, _ = await self._panel(metric, "line", first, last, {}, data=(values, lows, highs)) + series, notes = await self.air_series([metric], first, last) + air_panel, _ = self._panel(metric, "line", first, last, series[metric]) rain = rain_slots(self.weather("rainfall", "daily", first, last)) - result = analyse_air(values, rain, RATINGS[metric], self.tz, LABELS[metric]) + result = analyse_air(series[metric][0], rain, RATINGS[metric], self.tz, LABELS[metric]) out = {"period": f"{first} to {last}", "metric": f"{LABELS[metric]} ({CHART_UNITS[metric]})", "resolution": "30-minute slots (air readings averaged, rain from the weather station's cache)", "how_to_read": ("Open with `verdict`, then the evidence in `findings`. Rain and clean air share weather (wind, fronts), " @@ -210,7 +200,7 @@ async def air_link(self, args: dict, turn: Turn | None = None) -> str: bars = rain_bars(rain, self.tz, first, last) if result and air_panel and bars.x and wants_chart(args, turn, datetime.combine(first, time()), datetime.combine(last, time())): self._add_chart(out, [air_panel, panel_for("rain", bars=bars)], first, last, turn) - return json.dumps(out, ensure_ascii=False, separators=(",", ":")) + return json_result(out) async def air_scan(self, args: dict, turn: Turn | None = None) -> str: turn = turn or Turn() @@ -220,7 +210,11 @@ async def air_scan(self, args: dict, turn: Turn | None = None) -> str: if isinstance(period := self.period(args, SCAN_DAYS), str): return _error(period) first, last = period + trimmed = (last - first).days >= SCAN_MAX_DAYS + first = max(first, last - timedelta(days=SCAN_MAX_DAYS - 1)) air, notes = await self.air_series(list(SCAN_AIR) if metric == "all" else [metric], first, last) + if trimmed: + notes["period"] = f"Only the last {SCAN_MAX_DAYS} days can be scanned per question; this covers {first} to {last}" weather = {} for name, (group, field, _, _) in SCAN_WEATHER.items(): if values := self.weather(group, field, first, last): @@ -236,7 +230,7 @@ async def air_scan(self, args: dict, turn: Turn | None = None) -> str: weather_names = {**{k: (v[2], v[3]) for k, v in SCAN_WEATHER.items()}, "pressure_change": ("3-hour pressure change", "hPa"), "rain": ("rain", "mm")} origin, days = day_bounds(first, self.tz)[0], (last - first).days + 1 - result = await asyncio.to_thread(scan.scan, {m: v[0] for m, v in air.items()}, weather, direction, speed, origin, days) + result = await asyncio.to_thread(scan.scan, {m: v[0] for m, v in air.items()}, weather, direction, speed, origin, days, self.tz) summary = scan.summarise(result, air_names, weather_names) strongest = summary.pop("strongest") out = {"period": f"{first} to {last}", "resolution": "30-minute slots; each pair compared within the day and day to day", @@ -246,33 +240,26 @@ async def air_scan(self, args: dict, turn: Turn | None = None) -> str: if strongest and wants_chart(args, turn, datetime.combine(first, time()), datetime.combine(last, time())): drawn = [n for n in (strongest["air"], {"wind_speed": "wind"}.get(strongest["weather"], strongest["weather"])) if n in SERIES] if len(drawn) == 2: - panels = [(await self._panel(n, "bars" if n == "rain" else "line", first, last, {}, data=air.get(n)))[0] for n in drawn] + panels = [self._panel(n, "bars" if n == "rain" else "line", first, last, air.get(n))[0] for n in drawn] if all(panels): self._add_chart(out, panels, first, last, turn) - return json.dumps(out, ensure_ascii=False, separators=(",", ":")) + return json_result(out) - async def _panel(self, name: str, style: str, first: date, last: date, notes: dict, - data: tuple[dict, dict, dict] | None = None) -> tuple[Panel | None, dict]: - """(the panel, its figures for the caption); the panel is None when there is nothing to draw. `data` is an air series - already loaded (values, lows, highs).""" + def _panel(self, name: str, style: str, first: date, last: date, + air: tuple[dict, dict, dict] | None = None) -> tuple[Panel | None, dict]: + """(the panel, its figures for the caption); the panel is None when there is nothing to draw. A weather reading comes from + the station's cache, an air metric's readings in `air` (values, lows, highs: see air_series).""" if name in WEATHER: reading = WEATHER[name] - group, field, label, unit = reading.group, reading.field, reading.label, reading.unit - values, lows, highs = self.weather_band(group, field, first, last) if name == "rain": - bars = rain_bars(rain_slots(values), self.tz, first, last) + bars = rain_bars(rain_slots(self.weather(reading.group, reading.field, first, last)), self.tz, first, last) return (panel_for("rain", bars=bars) if bars.x else None, {"series": name, "total_mm": round(sum(bars.y), 1), "wet_bars": len(bars.y)}) - if reading.band_field: # a mean shaded up to another field (wind: the gusts) - gust, _, gust_high = self.weather_band(group, reading.band_field, first, last) - highs = {t: max(gust.get(t, 0.0), gust_high.get(t, 0.0)) for t in {*gust, *gust_high}} - line = driver_series(values, self.tz, first, last, label, lows, highs, banded_short=bool(reading.band_field)) - facts = {"series": name, **self._stats(values, unit)} + values, lows, highs = reading_slots(self.eco.cache, self.eco.mac, name, *self._span(first, last)) + line = driver_series(values, self.tz, first, last, reading.label, lows, highs, banded_short=bool(reading.band_field)) + facts = {"series": name, **self._stats(values, reading.unit)} return (panel_for(name, [line]) if line else None), facts - if data is None: - *data, more = await self.air_slots(name, first, last) - notes.update(more) - values, lows, highs = data + values, lows, highs = air or ({}, {}, {}) if len(values) < 2: return None, {} if style == "rating": diff --git a/lib/config.py b/lib/config.py index d048e5c..d341df6 100644 --- a/lib/config.py +++ b/lib/config.py @@ -1,12 +1,13 @@ -"""Settings, all from environment variables (systemd loads them from the unit's EnvironmentFile).""" +"""Settings, all from environment variables (systemd loads them from the unit's EnvironmentFile). Each is declared once, as +a field of Config: the variable it is read from, its default, and how its text is read.""" import logging import os import re -from dataclasses import dataclass +from dataclasses import dataclass, field, fields from datetime import datetime, tzinfo from pathlib import Path -from zoneinfo import ZoneInfo +from zoneinfo import ZoneInfo, ZoneInfoNotFoundError ROOT = Path(__file__).resolve().parent.parent log = logging.getLogger(__name__) @@ -21,16 +22,50 @@ def data_dir() -> Path: return Path(os.path.abspath(Path(value).expanduser())) if value else ROOT -def _tz() -> tzinfo: +def local_tz() -> tzinfo: + """The time zone when no weather station names its own (Ecowitt does, and it wins): TZ, else the system's, the zone + /etc/localtime links to. Only a zone that has no name is a fixed offset, which daylight saving would put out.""" name = os.getenv("TZ", "").lstrip(":") - return ZoneInfo(name) if name else datetime.now().astimezone().tzinfo + if name: + try: + return ZoneInfo(name) + except (ValueError, ZoneInfoNotFoundError): + log.warning("TZ: %r is not a time zone name; using the system's", name) + linked = os.path.realpath("/etc/localtime") + try: + return ZoneInfo(linked.split("/zoneinfo/", 1)[1]) + except (IndexError, ValueError, ZoneInfoNotFoundError): + return datetime.now().astimezone().tzinfo + + +ON, OFF = ("on", "yes", "true", "1"), ("off", "no", "false", "0") + + +# Each reader takes a setting's text (stripped; empty when unset or blank), its default and the variable's name. +def _flag(value: str, default: bool, name: str) -> bool: + """An on/off setting: on, yes, true or 1; off, no, false or 0; unset or empty is the default. Any other value keeps the + default too, with a warning, so a typo can't switch a safeguard such as ADMIN_ONLY off.""" + if value.lower() in ON: + return True + if value.lower() in OFF: + return False + if value: + log.warning("%s: %r is not on or off; keeping the default (%s)", name, value, "on" if default else "off") + return default -def _on(value: str) -> bool: - return value.lower() in ("on", "1", "true", "yes") +def _required(value: str, default: str, name: str) -> str: + """A setting the bot can't run without; it stops with a message naming it when it is unset or blank.""" + if value: + return value + raise SystemExit(f"{name} is not set: the bot can't start without it (see the README)") -def _hours(value: str, default: tuple[int, int] | None) -> tuple[int, int] | None: +def _url(value: str, default: str, name: str) -> str: + return (value or default).rstrip("/") + + +def _hours(value: str, default: tuple[int, int] | None, name: str) -> tuple[int, int] | None: """"0-6" -> (0, 6); "off" or "none" -> None; anything unreadable -> the default.""" if value.lower() in ("off", "none", "no", "false"): return None @@ -41,18 +76,13 @@ def _hours(value: str, default: tuple[int, int] | None) -> tuple[int, int] | Non return (start, end) if 0 <= start <= 23 and 0 <= end <= 24 and start != end else default -def _hour(value: str, default: int) -> int: +def _hour(value: str, default: int, name: str) -> int: """"18" -> 18; anything unreadable or outside 0 to 23 -> the default.""" number = _number(value) return int(number) if number is not None and 0 <= number <= 23 else default -def _flag(value: str, default: bool) -> bool: - """A setting that is on or off, with a default when it is unset or empty.""" - return _on(value) if value else default - - -def _ids(value: str) -> frozenset[int]: +def _ids(value: str, default: frozenset[int], name: str) -> frozenset[int]: """"123, -1001 456" -> {123, -1001, 456}. An entry that is not a whole number is ignored (and logged): it must not stop the bot.""" found, bad = set(), [] for part in filter(None, re.split(r"[,\s]+", value)): @@ -61,23 +91,31 @@ def _ids(value: str) -> frozenset[int]: except ValueError: bad.append(part) if bad: - log.warning("ADMIN_CHAT_IDS: ignoring %s (not whole numbers)", ", ".join(map(repr, bad))) - return frozenset(found) + log.warning("%s: ignoring %s (not whole numbers)", name, ", ".join(map(repr, bad))) + return frozenset(found) or default -def _whole(value: str, default: int) -> int: +def _whole(value: str, default: int, name: str) -> int: """"100" -> 100, "0" -> 0; anything but a whole number of 0 or more -> the default.""" return int(value) if value.isdecimal() else default -def _readings_in_a_row(value: str, default: int) -> int: +def _between(low: int, high: int): + """A reader of a whole number kept from `low` to `high` ("0" is `low`, like any number under it); unreadable -> the default.""" + def read(value: str, default: int, name: str) -> int: + number = _number(value) + return default if number is None else int(min(high, max(low, number))) + return read + + +def _readings_in_a_row(value: str, default: int, name: str) -> int: """"3" -> 3; "off", "0" or "1" -> 1 (no rule); anything unreadable -> the default.""" if value.lower() in ("off", "no", "false", "none"): return 1 return max(1, int(value)) if value.isdecimal() else default -def _chat_id(value: str, name: str) -> int | None: +def _chat_id(value: str, default: int | None, name: str) -> int | None: """A Telegram chat id (negative for a group or channel); None, with a warning, when it is not a whole number.""" if not value: return None @@ -95,42 +133,69 @@ def _number(value: str) -> float | None: return None +def _decimal(value: str, default: float | None, name: str) -> float | None: + return _number(value) if value else default + + +LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR") + + +def _level(value: str, default: str, name: str) -> str: + """One of LEVELS, whatever its case; another value keeps the default, with a warning.""" + if value.upper() in LEVELS: + return value.upper() + if value: + log.warning("%s: %r is not %s or %s; keeping %s", name, value, ", ".join(LEVELS[:-1]), LEVELS[-1], default) + return default + + +def _setting(name: str, default=None, read=None): + """A Config field read from the environment variable `name` by `read` (a reader above; none: the text, blank being the default).""" + return field(default=default, metadata={"env": name, "read": read}) + + +def _from_env(f) -> object: + text = os.getenv(f.metadata["env"], "").strip() + return f.metadata["read"](text, f.default, f.metadata["env"]) if f.metadata["read"] else text or f.default + + @dataclass(frozen=True) class Config: - telegram_token: str - xai_api_key: str - xai_base_url: str - xai_model: str tz: tzinfo - ecowitt_api_key: str - ecowitt_app_key: str - airgradient_token: str - airgradient_location: str - airgradient_dashboard: str - state_path: Path = ROOT / "bot_state.json" - cache_path: Path = ROOT / "ecowitt_cache.sqlite" - air_cache_path: Path = ROOT / "airgradient_cache.sqlite" - conditions_cache_path: Path = ROOT / "conditions_cache.sqlite" # the last pollen page and forecast, kept across restarts - place: str = "Melbourne" # PLACE: named after the Pollen & asthma and Forecast headings in the report - pollen: bool = False # POLLEN=on: Melbourne grass pollen and thunderstorm asthma risk (a scraped website) - pollen_district: str = "Central" # the Victorian forecast district whose thunderstorm asthma risk is used - forecast: bool = False # FORECAST=on: the daily forecast from Open-Meteo - forecast_lat: float | None = None # FORECAST_LAT / FORECAST_LON; the weather station's own location when unset - forecast_lon: float | None = None - rain_quiet_hours: tuple[int, int] | None = (0, 6) # RAIN_QUIET_HOURS="0-6" (local hours; "off" for none): no rain alerts, a summary after - chart_all_feels_like: bool = False # CHART_ALL_FEELS_LIKE=on: "weather all week" also draws the feels-like panel (off: it nearly repeats temperature) - chart_all_vpd: bool = False # CHART_ALL_VPD=on: "weather all week" also draws the vapour pressure deficit panel (off: it is temperature and humidity combined) - tuya_device_id: str = "" # TUYA_DEVICE_ID, TUYA_CLIENT_ID, TUYA_CLIENT_SECRET: all three switch the irrigation controller on - tuya_client_id: str = "" - tuya_client_secret: str = "" - tuya_base_url: str = "https://openapi.tuyaeu.com" # TUYA_BASE_URL: the data centre's address (EU by default) - irrigation_check_hour: int = 18 # IRRIGATION_CHECK_HOUR: the local hour of the daily check (0 to 23) - admin_only: bool = True # ADMIN_ONLY (on by default): the bot answers nobody but the admins of the groups it is in - admin_chat_ids: frozenset[int] = frozenset() # ADMIN_CHAT_IDS: more ids allowed besides those admins (a user id, or a group id for its anonymous admins) - alert_cooldown_minutes: int = 30 # ALERT_COOLDOWN_MINUTES: the rain and temperature-crossing alerts: dry this long before "the rain has stopped", and this long between alerts (5 to 150) - telegram_preload_chat: int | None = None # TELEGRAM_PRELOAD_CHAT: a chat the other periods of each chart are silently uploaded to ahead of the press (none: off) - antispike_require_consecutive: int = 2 # ANTISPIKE_REQUIRE_CONSECUTIVE: readings in a row a rise of the particles needs to be the air, not a glitch of the sensor (1: off) - log_width: int = 71 # LOG_WIDTH: the most characters of a long log line (a request, a chat message), cut with an ellipsis (…); 0 cuts none + state_path: Path + cache_path: Path + air_cache_path: Path + conditions_cache_path: Path # the last pollen page and forecast, kept across restarts + telegram_token: str = _setting("TELEGRAM_BOT_TOKEN", "", _required) + xai_api_key: str = _setting("XAI_API_KEY", "", _required) + xai_base_url: str = _setting("XAI_BASE_URL", "https://api.x.ai/v1") + xai_model: str = _setting("XAI_MODEL", "grok-4.3") + ecowitt_api_key: str = _setting("ECOWITT_API_KEY", "") + ecowitt_app_key: str = _setting("ECOWITT_APP_KEY", "") + airgradient_token: str = _setting("AIRGRADIENT_API_TOKEN", "") + airgradient_location: str = _setting("AIRGRADIENT_LOCATION_ID", "") + airgradient_dashboard: str = _setting("AIRGRADIENT_DASHBOARD_URL", "") + place: str = _setting("PLACE", "Melbourne") # named after the Pollen & asthma and Forecast headings in the report + pollen: bool = _setting("POLLEN", False, _flag) # Melbourne grass pollen and thunderstorm asthma risk (a scraped website) + pollen_district: str = _setting("POLLEN_DISTRICT", "Central") # the Victorian forecast district whose thunderstorm asthma risk is used + forecast: bool = _setting("FORECAST", False, _flag) # the daily forecast from Open-Meteo + forecast_lat: float | None = _setting("FORECAST_LAT", None, _decimal) # the weather station's own location when unset + forecast_lon: float | None = _setting("FORECAST_LON", None, _decimal) + rain_quiet_hours: tuple[int, int] | None = _setting("RAIN_QUIET_HOURS", (0, 6), _hours) # "0-6", local hours ("off" for none): no rain alerts, a summary after + chart_all_feels_like: bool = _setting("CHART_ALL_FEELS_LIKE", False, _flag) # "weather all week" also draws the feels-like panel (off: it nearly repeats temperature) + chart_all_vpd: bool = _setting("CHART_ALL_VPD", False, _flag) # "weather all week" also draws the vapour pressure deficit panel (off: it is temperature and humidity combined) + tuya_device_id: str = _setting("TUYA_DEVICE_ID", "") # all three switch the irrigation controller on + tuya_client_id: str = _setting("TUYA_CLIENT_ID", "") + tuya_client_secret: str = _setting("TUYA_CLIENT_SECRET", "") + tuya_base_url: str = _setting("TUYA_BASE_URL", "https://openapi.tuyaeu.com", _url) # the data centre's address (EU by default) + irrigation_check_hour: int = _setting("IRRIGATION_CHECK_HOUR", 18, _hour) # the local hour of the daily check (0 to 23) + admin_only: bool = _setting("ADMIN_ONLY", True, _flag) # the bot answers nobody but the admins of the groups it is in + admin_chat_ids: frozenset[int] = _setting("ADMIN_CHAT_IDS", frozenset(), _ids) # more ids allowed besides those admins (a user id, or a group id for its anonymous admins) + alert_cooldown_minutes: int = _setting("ALERT_COOLDOWN_MINUTES", 30, _between(5, 150)) # the rain and temperature-crossing alerts: dry this long before "the rain has stopped", and this long between alerts + telegram_preload_chat: int | None = _setting("TELEGRAM_PRELOAD_CHAT", None, _chat_id) # a chat the other periods of each chart are silently uploaded to ahead of the press (none: off) + antispike_require_consecutive: int = _setting("ANTISPIKE_REQUIRE_CONSECUTIVE", 2, _readings_in_a_row) # readings in a row a rise of the particles needs to be the air, not a glitch of the sensor (1: off) + log_width: int = _setting("LOG_WIDTH", 71, _whole) # the most characters of a long log line (a request, a chat message), cut with an ellipsis (…); 0 cuts none + log_level: str = _setting("LOG_LEVEL", "INFO", _level) # DEBUG adds the bot's own debug lines; the libraries keep theirs @property def ecowitt(self) -> bool: @@ -146,7 +211,8 @@ def airgradient(self) -> bool: @classmethod def from_env(cls) -> "Config": - env = lambda k, default="": os.getenv(k, default).strip() + """Every setting from its variable, the files in the data folder. Stops, saying why, when one the bot can't run without is + missing or the folder can't be written.""" data = data_dir() try: data.mkdir(parents=True, exist_ok=True) @@ -156,42 +222,9 @@ def from_env(cls) -> "Config": if not writable: raise SystemExit(f"{data} is not writable by this user: set DATA_DIR to a writable folder " "(under systemd with StateDirectory=ecobot that is /var/lib/ecobot)") - cfg = cls( - telegram_token=os.environ["TELEGRAM_BOT_TOKEN"], - xai_api_key=os.environ["XAI_API_KEY"], - xai_base_url=env("XAI_BASE_URL", "https://api.x.ai/v1"), - xai_model=env("XAI_MODEL", "grok-4.3"), - tz=_tz(), - ecowitt_api_key=env("ECOWITT_API_KEY"), - ecowitt_app_key=env("ECOWITT_APP_KEY"), - airgradient_token=env("AIRGRADIENT_API_TOKEN"), - airgradient_location=env("AIRGRADIENT_LOCATION_ID"), - airgradient_dashboard=env("AIRGRADIENT_DASHBOARD_URL"), - place=env("PLACE", "Melbourne"), - pollen=_on(env("POLLEN")), - pollen_district=env("POLLEN_DISTRICT", "Central"), - forecast=_on(env("FORECAST")), - forecast_lat=_number(env("FORECAST_LAT")), - forecast_lon=_number(env("FORECAST_LON")), - rain_quiet_hours=_hours(env("RAIN_QUIET_HOURS"), (0, 6)), - chart_all_feels_like=_on(env("CHART_ALL_FEELS_LIKE")), - chart_all_vpd=_on(env("CHART_ALL_VPD")), - tuya_device_id=env("TUYA_DEVICE_ID"), - tuya_client_id=env("TUYA_CLIENT_ID"), - tuya_client_secret=env("TUYA_CLIENT_SECRET"), - tuya_base_url=env("TUYA_BASE_URL", "https://openapi.tuyaeu.com").rstrip("/"), - irrigation_check_hour=_hour(env("IRRIGATION_CHECK_HOUR"), 18), - admin_only=_flag(env("ADMIN_ONLY"), True), - admin_chat_ids=_ids(env("ADMIN_CHAT_IDS")), - alert_cooldown_minutes=int(min(150, max(5, _number(env("ALERT_COOLDOWN_MINUTES")) or 30))), - log_width=_whole(env("LOG_WIDTH"), 71), - antispike_require_consecutive=_readings_in_a_row(env("ANTISPIKE_REQUIRE_CONSECUTIVE"), 2), - telegram_preload_chat=_chat_id(env("TELEGRAM_PRELOAD_CHAT"), "TELEGRAM_PRELOAD_CHAT"), - state_path=data / "bot_state.json", - cache_path=data / "ecowitt_cache.sqlite", - air_cache_path=data / "airgradient_cache.sqlite", - conditions_cache_path=data / "conditions_cache.sqlite", - ) + cfg = cls(tz=local_tz(), state_path=data / "bot_state.json", cache_path=data / "ecowitt_cache.sqlite", + air_cache_path=data / "airgradient_cache.sqlite", conditions_cache_path=data / "conditions_cache.sqlite", + **{f.name: _from_env(f) for f in fields(cls) if "env" in f.metadata}) if not (cfg.ecowitt or cfg.airgradient): raise SystemExit("Set ECOWITT_API_KEY + ECOWITT_APP_KEY and/or AIRGRADIENT_API_TOKEN + AIRGRADIENT_LOCATION_ID") return cfg diff --git a/lib/daytable.py b/lib/daytable.py index 55f833d..88c5457 100644 --- a/lib/daytable.py +++ b/lib/daytable.py @@ -8,14 +8,15 @@ import logging import operator import re -from dataclasses import dataclass, field +from dataclasses import dataclass from collections.abc import Callable -from datetime import date, datetime, tzinfo +from datetime import date, datetime, timedelta, tzinfo from statistics import fmean from .captions import COUNT_CHART_HINT -from .specs import Bars, Chart, Panel -from .timeutil import day_bounds +from .series import AIR +from .specs import Chart, Panel, calendar_bars +from .timeutil import day_text log = logging.getLogger(__name__) @@ -25,8 +26,7 @@ MAX_LIMIT = 20 PRETTY = {"temp": "temperature", "uv": "UV index", "dew_point": "dew point", "feels_like": "feels-like", "vpd": "vapour pressure deficit", "wind_gust": "wind gust", "wind_speed": "wind speed", "rain_rate": "rain rate", "indoor_temp": "indoor temperature", - "indoor_humidity": "indoor humidity", "solar": "solar radiation", "pm2_5": "PM2.5", "pm10": "PM10", "pm1": "PM1", - "co2": "CO₂", "voc_index": "VOC index", "nox_index": "NOx index"} + "indoor_humidity": "indoor humidity", "solar": "solar radiation", **{name: m.label for name, m in AIR.items()}} SUFFIXES = ("_max", "_min", "_avg") @@ -96,11 +96,63 @@ def validate(args: dict, fields: dict[str, str]) -> str | None: return f"stat must be one of {', '.join(STATS)}" if args.get("stat") not in (None, "count") and args.get("of") not in fields: return f"stat {args['stat']} needs `of`: one of {', '.join(fields)}" + if args.get("of") and args["of"] not in fields: + return f"of must be one of {', '.join(fields)}" if args.get("group_by") not in (None, "month", "year"): return "group_by must be month or year" return None +@dataclass(frozen=True) +class Asked: + """A days question, checked: the period (up to yesterday), the arguments with sort_by filled in, and the fields a listed + day shows, each with its unit.""" + first: date + last: date + args: dict + units: dict[str, str] + + @property + def days(self) -> list[date]: + return [self.first + timedelta(days=k) for k in range((self.last - self.first).days + 1)] + + @property + def shown(self) -> list[str]: + return list(self.units) + + @property + def where(self) -> list[dict]: + return self.args.get("where") or [] + + @property + def sort_by(self) -> str: + return self.args["sort_by"] + + +def prepare(args: dict, fields: dict[str, str], shown: tuple[str, ...], today: date) -> Asked | str: + """The question checked (an error text when it can't be used) and filled in. `fields` maps each field the tool offers to + its unit; `shown` are the ones every listed day shows, before those the arguments name. Days are ranked by sort_by, else + the first condition's field, else `of`, else the first of `shown`.""" + try: + first = date.fromisoformat(str(args["start_date"])[:10]) + last = min(date.fromisoformat(str(args["end_date"])[:10]), today - timedelta(days=1)) + except (KeyError, ValueError) as e: + return f"bad date ({e}); use 'YYYY-MM-DD'" + if first > last: + return "start_date must be before end_date (today isn't final, so the latest day is yesterday)" + if problem := validate(args, fields): + return problem + where = args.get("where") or [] + sort_by = args.get("sort_by") or (where[0]["field"] if where else args.get("of") or shown[0]) + named = dict.fromkeys([*shown, sort_by, *(c["field"] for c in where), *([args["of"]] if args.get("of") else [])]) + return Asked(first, last, {**args, "sort_by": sort_by}, {n: fields[n] for n in named}) + + +def lists_days(args: dict) -> bool: + """Does the answer list the days? Not when only the counts are asked for, nor a figure per month or year.""" + return not (args.get("count_only") or (args.get("stat") not in (None, "count") and args.get("group_by"))) + + def conditions_text(where: list[dict], units: dict[str, str]) -> str: """"UV index ≥ 10", "rain ≥ 1 mm and lowest humidity < 40 %": the conditions as a chart's title says them.""" parts = [] @@ -129,13 +181,12 @@ def aggregate(stat: str, numbers: list[float]) -> float | None: return round({"max": max, "min": min, "sum": sum, "avg": fmean}[stat](numbers), 1) -def bars_chart(figures: dict[str, float | None], by: str, title: str, subtitle: str, label: str, unit: str, tz: tzinfo) -> Chart: - """One bar per month or year (those with no figure left out), the number above each.""" - shown = {k: v for k, v in figures.items() if v is not None} - starts = [datetime.strptime(k, "%Y-%m" if by == "month" else "%Y").date() for k in shown] - bars = Bars(label, unit, [day_bounds(d, tz)[0] for d in starts], [float(v) for v in shown.values()], - 28 * 86400 if by == "month" else 365 * 86400, by, values=True) - return Chart(title, subtitle, [Panel(label, "" if unit == "days" else unit, bars=bars)]) +def bars_chart(figures: dict[str, float | None], by: str, title: str, subtitle: str, label: str, unit: str, tz: tzinfo, + period: tuple[date, date]) -> Chart: + """One bar per month or year (those with no figure left out), the number above each; `period` is the first and last day counted.""" + shown = {datetime.strptime(k, "%Y-%m" if by == "month" else "%Y").date(): v for k, v in figures.items() if v is not None} + return Chart(title, subtitle, [Panel(label, "" if unit == "days" else unit, bars=calendar_bars(label, unit, shown, by, tz))], + period=period) @dataclass @@ -146,26 +197,23 @@ class Analysis: rank: Callable holds: Callable row: Callable - rows: list = field(default_factory=list) -def analyse(values: dict[str, dict[date, float]], shown: list[str], units: dict[str, str], args: dict, checked: list[date], - first: date, last: date, tz: tzinfo, turn=None, row_extra: Callable[[date], dict] | None = None, - scope: str = "") -> Analysis: - """Find, rank, count and group the checked days. `values` has {day: figure} for every field in `shown`.""" - where = args.get("where") or [] - sort_by = args.get("sort_by") or (where[0]["field"] if where else shown[0]) +def analyse(values: dict[str, dict[date, float]], asked: Asked, checked: list[date], tz: tzinfo, turn=None, + row_extra: Callable[[date], dict] | None = None, scope: str = "") -> Analysis: + """Find, rank, count and group the checked days. `values` has {day: figure} for every field the question shows.""" + args, where, shown, units, sort_by = asked.args, asked.where, asked.shown, asked.units, asked.sort_by + first, last = asked.first, asked.last limit = max(1, min(int(args.get("limit") or 5), MAX_LIMIT)) - label = lambda d: f"{d:%a} {d.day} {d:%b %Y}" holds = lambda d, conds: all(d in values[c["field"]] and OPS[c["op"]](values[c["field"]][d], c["value"]) for c in conds) rank = lambda ds: sorted((d for d in ds if d in values[sort_by]), key=lambda d: values[sort_by][d], reverse=args.get("order", "desc") != "asc") - row = lambda d: {"date": label(d), **(row_extra(d) if row_extra else {}), + row = lambda d: {"date": day_text(d), **(row_extra(d) if row_extra else {}), **{n: round(values[n][d], 1) for n in shown if d in values[n]}} matches = [d for d in checked if holds(d, where)] ranked = rank(matches) - out = {"period": f"{label(first)} - {label(last)}", "days_checked": len(checked), "matching_days": len(matches), - "units": {n: units[n] for n in shown}} + out = {"period": f"{day_text(first)} - {day_text(last)}", "days_checked": len(checked), "matching_days": len(matches), + "units": dict(units)} if where: out["condition"] = conditions_text(where, units) stat, of, by = args.get("stat") or "count", args.get("of"), args.get("group_by") @@ -184,7 +232,7 @@ def analyse(values: dict[str, dict[date, float]], shown: list[str], units: dict[ buckets[key(d)] += 1 if by and buckets: out[f"by_{by}"] = buckets - if turn is not None: # a count or figure per month or year is a bar chart, not a list + if turn is not None and any(v is not None for v in buckets.values()): # a bar chart, not a list (unless no bucket has a figure) on = f" on days with {conditions_text(where, units)}" if where and stat != "count" else "" if stat == "count": title, name, unit = (f"Days with {conditions_text(where, units)}" if where else "Days checked"), "Days", "days" @@ -193,10 +241,9 @@ def analyse(values: dict[str, dict[date, float]], shown: list[str], units: dict[ title, name, unit = phrase(stat, of) + on, noun(of).capitalize(), units[of] detail = f"{len(matches):,} days counted" detail += f" · {scope}" if scope else "" - turn.charts.append(bars_chart(buckets, by, title, f"{label(first)} – {label(last)} · {detail} · per {by}", - name, unit, tz)) + turn.charts.append(bars_chart(buckets, by, title, f"{day_text(first)} – {day_text(last)} · {detail} · per {by}", + name, unit, tz, (first, last))) out["chart"] = COUNT_CHART_HINT - rows = [] if args.get("count_only") or stat != "count" and by else [row(d) for d in ranked[:limit]] - if not args.get("count_only") and not (stat != "count" and by): - out["days"] = rows - return Analysis(out, matches, ranked, rank, holds, row, rows) + if lists_days(args): + out["days"] = [row(d) for d in ranked[:limit]] + return Analysis(out, matches, ranked, rank, holds, row) diff --git a/lib/ecowitt/api.py b/lib/ecowitt/api.py index 8c9e6aa..56cab17 100644 --- a/lib/ecowitt/api.py +++ b/lib/ecowitt/api.py @@ -14,18 +14,39 @@ import asyncio import logging +import re from collections import deque -from datetime import datetime, timedelta +from datetime import UTC, date, datetime, time, timedelta, tzinfo +from typing import NamedTuple import httpx log = logging.getLogger(__name__) BASE = "https://api.ecowitt.net/api/v3/device" -RETENTION = {"5min": 90, "30min": 365, "4hour": 730, "1day": 1460} # days kept -MAX_SPAN = {"5min": timedelta(days=1), "30min": timedelta(days=7), - "4hour": timedelta(days=28), "1day": timedelta(days=365)} -CYCLE_SECONDS = {"5min": 300, "30min": 1800, "4hour": 14400, "1day": 86400} + + +class Cycle(NamedTuple): + """One of Ecowitt's resolutions.""" + seconds: int # the time one reading covers + span: timedelta # the most one request may ask for + kept: int # days Ecowitt keeps its readings + settle: int # seconds after a reading's time is over before its value is final + + +CYCLES = {"5min": Cycle(300, timedelta(days=1), 90, 3600), "30min": Cycle(1800, timedelta(days=7), 365, 7200), + "4hour": Cycle(14400, timedelta(days=28), 730, 8 * 3600), "1day": Cycle(86400, timedelta(days=365), 1460, 86400)} +# Readings every half hour or finer: they fit a local day exactly (Ecowitt's 4-hour and daily ones are cut on UTC hours), +# and they change at every refresh. +FINE_CYCLES = tuple(name for name, c in CYCLES.items() if c.seconds <= 1800) + + +def kept_from(cycle: str, today: date) -> date: + """The first day whose readings at this cycle Ecowitt is sure still to have: two days clear of its retention edge, so a + request can't race the deletion.""" + return today - timedelta(days=CYCLES[cycle].kept - 2) + + # What the bot keeps cached and warm (the extras feed the rain alerts and predictions). One list, so the # warm-up and the archive fetch the same thing and can reuse each other's data. GROUPS = ["outdoor", "indoor", "pressure", "wind", "rainfall", "rainfall_piezo", "solar_and_uvi"] @@ -35,10 +56,17 @@ UNITS = {"temp_unitid": "C", "pressure_unitid": "hPa", "wind_speed_unitid": "kmh", "rainfall_unitid": "mm"} UNIT_IDS = {"temp_unitid": 1, "pressure_unitid": 3, "wind_speed_unitid": 7, "rainfall_unitid": 12} + +def unit_text(text: str | None) -> str | None: + """A reading or a unit as Ecowitt writes it, written the bot's way: "12.3 ℃" -> "12.3 °C", "131 º" -> "131°", "1.8 mm/hr" -> + "1.8 mm/h". None for nothing.""" + return text.replace("℃", "°C").replace(" º", "°").replace("º", "°").replace("mm/hr", "mm/h").strip() if text else None + BUSY_RETRIES = 3 MIN_GAP_SECONDS = 1.0 # Ecowitt rejects requests that come too fast ("Operation too frequent") RETRY_ON = ("busy", "too frequent") FMT = "%Y-%m-%d %H:%M:%S" +KEYS_IN_URL = re.compile(r"\b(application_key|api_key)=[^&\s'\"]*") # the keys go in the query string # Fields Ecowitt reports in a unit we do not use: (field prefix, its unit) -> (the unit kept, the factor). The vapour pressure @@ -46,6 +74,15 @@ UNIT_FIXES = {("vpd", "inHg"): ("kPa", 3.38639)} +def daily_run(tz: tzinfo, first: date, last: date) -> str: + """When Ecowitt's daily records start and end, local time ("10am to 10am"): they are UTC days, so in Melbourne they run from + 10am, or from 11am in daylight saving; a period that has both says both.""" + days = [first + timedelta(days=n) for n in range(0, (last - first).days + 1, 30)] + [last] + hours = sorted({datetime.combine(d, time(), UTC).astimezone(tz).hour for d in days}) + run = [f"{h % 12 or 12}{'am' if h < 12 else 'pm'} to {h % 12 or 12}{'am' if h < 12 else 'pm'}" for h in hours] + return run[0] if len(run) == 1 else f"{run[0]} ({run[1]} in daylight saving)" + + def fix_units(data) -> dict: """The response with any field in UNIT_FIXES converted (readings and unit), in place.""" for fields in data.values() if isinstance(data, dict) else (): @@ -133,37 +170,41 @@ async def close(self): await self.client.aclose() async def _get(self, path: str, urgent: bool = False, **params) -> dict | list: - """The response's "data". Retries when Ecowitt says it is busy or we were too quick; - raises EcowittError. urgent: someone is waiting for it, so it goes before queued background requests.""" + """The response's "data"; raises EcowittError. Asked again, up to BUSY_RETRIES times and waiting longer each time, when + Ecowitt is busy, rate limited or erring (429, 5xx), not when it refuses the request or there is no answer at all. + urgent: someone is waiting for it, so it goes before queued background requests (and the turn is free while a request + waits to be asked again).""" loop = asyncio.get_running_loop() - await self._turn.acquire(urgent) - try: - for attempt in range(BUSY_RETRIES + 1): + for attempt in range(BUSY_RETRIES + 1): + await self._turn.acquire(urgent) + try: if (pause := self._last + MIN_GAP_SECONDS - loop.time()) > 0: await asyncio.sleep(pause) self._last = loop.time() self.requests += 1 try: res = await self.client.get(f"{BASE}/{path}", params={**self.keys, **params}) + except httpx.HTTPError as e: # no answer: the network, a timeout + msg, passing, again = str(e) or type(e).__name__, True, False + else: body = _body(res) - if body is None: # no Ecowitt answer: the network, or a gateway error page - res.raise_for_status() - raise ValueError("unreadable response") - if str(body.get("code")) == "0" and res.is_success: + if body is not None and str(body.get("code")) == "0" and res.is_success: return body.get("data") or {} - msg = str(body.get("msg") or f"HTTP {res.status_code}") - # Ecowitt answered and said no (any HTTP status): passing only if busy or rate limited - passing = res.status_code in (408, 429) or res.status_code >= 500 or any(k in msg.lower() for k in RETRY_ON) - except (httpx.HTTPError, ValueError) as e: - msg, passing = str(e) or type(e).__name__, True - if any(k in msg.lower() for k in RETRY_ON) and attempt < BUSY_RETRIES: - wait = 3 * 2 ** attempt - log.info("Ecowitt says %r (%s), retrying in %.1fs", msg, path, wait) - await asyncio.sleep(wait) - continue - raise EcowittError(msg, transient=passing) - finally: - self._turn.release() + if body is None: # a gateway's error page, or something unreadable + msg = f"HTTP {res.status_code}" if not res.is_success else "unreadable response" + else: # Ecowitt said no: passing only if it is busy, rate limiting or failing itself + msg = str(body.get("msg") or f"HTTP {res.status_code}") + again = res.status_code == 429 or res.status_code >= 500 or any(k in msg.lower() for k in RETRY_ON) + passing = again or body is None or res.status_code == 408 + finally: + self._turn.release() + msg = KEYS_IN_URL.sub(r"\1=…", msg) # an error can quote the URL; its text is shown to the model and logged + if again and attempt < BUSY_RETRIES: + wait = 3 * 2 ** attempt + log.info("Ecowitt says %r (%s), retrying in %.1fs", msg, path, wait) + await asyncio.sleep(wait) + continue + raise EcowittError(msg, transient=passing) async def devices(self) -> list[dict]: data = await self._get("list", limit=50) diff --git a/lib/ecowitt/archive.py b/lib/ecowitt/archive.py index 88fc674..a5e076e 100644 --- a/lib/ecowitt/archive.py +++ b/lib/ecowitt/archive.py @@ -11,17 +11,20 @@ import logging from collections.abc import Iterator from datetime import datetime, time, timedelta +from time import monotonic -from ..timeutil import now_local -from .api import MIN_GAP_SECONDS, RETENTION +from ..timeutil import naive_local, now_local, seconds_until +from ..warm import every +from .api import CYCLES, MIN_GAP_SECONDS, kept_from from .fetch import spans -from .store import BUCKET_SECONDS, horizon, subtract +from .store import horizon log = logging.getLogger(__name__) RUN_AT = time(1, 30) PACE_SECONDS = 2.0 # gap between requests, so questions aren't starved -COARSE_CYCLES = ("1day", "4hour", "30min") # 5-minute data goes day by day, first +COARSE_CYCLES = tuple(c for c in reversed(CYCLES) if c != "5min") # coarsest first; 5-minute data goes day by day, before them +OUTAGE_STOP = 3 # requests failing in a row (no answer, or a busy Ecowitt) that end a run: the rest wait for the next class Archive: @@ -43,16 +46,16 @@ def _work(self) -> Iterator[tuple[str, datetime, datetime]]: earliest = (created.date() - timedelta(days=1)) if created else None def first_day(cycle: str): - oldest = today - timedelta(days=RETENTION[cycle] - 2) # stay clear of the retention edge + oldest = kept_from(cycle, today) return max(oldest, earliest) if earliest else oldest def settled(cycle: str) -> datetime: """The end of the last whole bucket that is final at this resolution. Whole buckets only, so the edge holds still between runs instead of creeping forward with the clock (which would leave a few seconds uncovered, and refetched, every time).""" - size = BUCKET_SECONDS[cycle] + size = CYCLES[cycle].seconds ts = horizon(cycle) - size - return datetime.fromtimestamp(ts - ts % size - 1, tz).replace(tzinfo=None) + return naive_local(ts - ts % size - 1, tz) for n in range((today - first_day("5min")).days, 0, -1): # finished days only day = today - timedelta(days=n) @@ -66,33 +69,38 @@ def settled(cycle: str) -> datetime: async def run_once(self) -> tuple[int, int]: """Cache whatever is missing. Returns (ranges fetched, ranges failed). Each range is saved as soon as it arrives, so an interrupted run just carries on from there next time.""" - # What is cached, read once (a query per resolution and group) rather than once per range - cover = await asyncio.to_thread(lambda: {(c, g): self.station.cache.coverage(self.station.mac, c, g) - for c in ("5min", *COARSE_CYCLES) for g in self.groups}) - epoch = self.station.fetcher(self.groups).epoch - todo = [(cycle, start, end) for cycle, start, end in self._work() - if any(subtract((epoch(start), epoch(end)), cover[cycle, g]) for g in self.groups)] + started = monotonic() + work, epoch = list(self._work()), self.station.fetcher(self.groups).epoch + missing = await asyncio.to_thread(self.station.cache.uncovered, self.station.mac, self.groups, # one read of what is held + [(cycle, epoch(start), epoch(end)) for cycle, start, end in work]) + todo = [w for w, gaps in zip(work, missing) if any(gaps.values())] if todo: # each range takes the pause after it plus about a second for the request itself seconds = len(todo) * (max(self.pace, MIN_GAP_SECONDS) + 1) if seconds >= 60: # a short run only gets the line when it finishes log.info("Ecowitt: %d req to fetch (about %s)", len(todo), f"{seconds:.0f} s" if seconds < 90 else f"{seconds / 60:.0f} min") - fetched = failed = 0 + fetched = failed = in_a_row = 0 for i, (cycle, start, end) in enumerate(todo, 1): fetcher = self.station.fetcher(self.groups) await fetcher.get(cycle, start, end, load=False) # only to cache it - if fetcher.errors and not await self._drop_unsupported_groups(cycle, start, end): - failed += 1 + if not fetcher.errors or fetcher.rejected and await self._drop_unsupported_groups(cycle, start, end): + fetched, in_a_row = fetched + 1, 0 else: - fetched += 1 + failed += 1 + in_a_row = 0 if fetcher.rejected else in_a_row + 1 # a refusal is about this range; a timeout, about Ecowitt + if in_a_row == OUTAGE_STOP: + log.warning("Ecowitt: %d requests in a row failed (%s); the other %d wait for the next run", in_a_row, + fetcher.errors[-1], len(todo) - i) + break if i % 20 == 0 and i < len(todo): log.info("Ecowitt: %d of %d req done", i, len(todo)) await asyncio.sleep(self.pace) + log.info("Ecowitt: %d req, %sheld %s, %.0fs", fetched, f"{failed} failed, " if failed else "", self.held(), monotonic() - started) return fetched, failed async def _drop_unsupported_groups(self, cycle: str, start: datetime, end: datetime) -> bool: - """After a failed request, try each group alone. Groups that fail on their own are dropped + """After a request Ecowitt refused, try each group alone. Groups that fail on their own are dropped (the station probably doesn't have them). True if the range was then cached for the rest.""" if len(self.groups) < 2: return False @@ -117,22 +125,8 @@ def held(self) -> str: return f"{days} days (5m/30m/4h/1d)" def _seconds_until_next_run(self) -> float: - now = datetime.now(self.station.tz) - nxt = now.replace(hour=RUN_AT.hour, minute=RUN_AT.minute, second=0, microsecond=0) - if nxt <= now: - nxt += timedelta(days=1) - return (nxt - now).total_seconds() + return seconds_until(RUN_AT.hour, RUN_AT.minute, self.station.tz) async def loop(self): - """Backfill now, then again every night at RUN_AT, until cancelled.""" - while True: - try: - started = datetime.now() - fetched, failed = await self.run_once() - log.info("Ecowitt: %d req, %sheld %s, %.0fs", fetched, f"{failed} failed, " if failed else "", - self.held(), (datetime.now() - started).total_seconds()) - except asyncio.CancelledError: - raise - except Exception: - log.exception("Ecowitt archive run failed") - await asyncio.sleep(self._seconds_until_next_run()) + """Backfill now, then again every night at RUN_AT, until cancelled. A run that fails is logged; the next night's tries again.""" + await every(self._seconds_until_next_run, self.run_once) diff --git a/lib/ecowitt/days.py b/lib/ecowitt/days.py index 5ae6b0e..19e02dc 100644 --- a/lib/ecowitt/days.py +++ b/lib/ecowitt/days.py @@ -10,18 +10,19 @@ """ import asyncio -import json import logging import operator import re -from datetime import date, datetime, timedelta, tzinfo +from datetime import date, datetime, tzinfo from statistics import fmean -from ..daytable import analyse, grouped_as_asked, parameters, validate -from ..timeutil import day_bounds, local_date, now_local +from ..daytable import analyse, grouped_as_asked, lists_days, parameters, prepare +from ..series import WEATHER +from ..timeutil import day_bounds, day_text, local_date, now_local +from ..tools import json_result +from .api import FINE_CYCLES, daily_run from .calendar import PublicHolidays -from .extremes import collect -from .store import HistoryCache, subtract +from .store import HistoryCache log = logging.getLogger(__name__) @@ -32,20 +33,22 @@ "rain": ("rainfall", "daily", "high", "mm"), # the day's total: the rain counter's highest reading "wind_gust": ("wind", "wind_gust", "high", "km/h"), } -# every other reading the station has, a day's highest (_max) and lowest (_min): a day's peak UV, its driest air, its lowest pressure ... -for _name, _group, _field, _unit in ( - ("humidity", "outdoor", "humidity", "%"), ("pressure", "pressure", "relative", "hPa"), ("wind_speed", "wind", "wind_speed", "km/h"), - ("dew_point", "outdoor", "dew_point", "°C"), ("feels_like", "outdoor", "feels_like", "°C"), ("vpd", "outdoor", "vpd", "kPa"), - ("solar", "solar_and_uvi", "solar", "W/m²"), ("uv", "solar_and_uvi", "uvi", ""), ("rain_rate", "rainfall", "rain_rate", "mm/h"), - ("indoor_temp", "indoor", "temperature", "°C"), ("indoor_humidity", "indoor", "humidity", "%")): +# Every other reading (series), then three that aren't series readings: a day's highest (_max), its mean (_avg: "average humidity +# per month") and, where its low says something (not 0 every night or calm hour), its lowest (_min): a day's peak UV, its driest +# air, its lowest pressure ... +_OTHERS = [("wind_speed" if name == "wind" else name, r.group, r.field, r.unit, r.low) + for name, r in WEATHER.items() if name not in ("temperature", "rain")] +_OTHERS += [("rain_rate", "rainfall", "rain_rate", "mm/h", False), + ("indoor_temp", "indoor", "temperature", "°C", True), ("indoor_humidity", "indoor", "humidity", "%", True)] +for _name, _group, _field, _unit, _low in _OTHERS: FIELDS[f"{_name}_max"] = (_group, _field, "high", _unit) - FIELDS[f"{_name}_avg"] = (_group, _field, "mean", _unit) # a day's mean: "average humidity per month" - if _name not in ("solar", "uv", "rain_rate", "wind_speed"): # these have no interesting low: it is 0 every night or calm hour + FIELDS[f"{_name}_avg"] = (_group, _field, "mean", _unit) + if _low: FIELDS[f"{_name}_min"] = (_group, _field, "low", _unit) FIELDS["temp_avg"] = ("outdoor", "temperature", "mean", "°C") AVERAGED = {n for n in FIELDS if n.endswith(("_max", "_min")) and n not in ("temp_max", "temp_min")} # no range of their own at 30 minutes or daily -ALWAYS_SHOWN = ("temp_max", "temp_min", "rain") -SUB_DAILY = ("5min", "30min") +FIELD_UNITS = {n: f[3] for n, f in FIELDS.items()} +ALWAYS_SHOWN = ("temp_max", "temp_min", "rain") # every listed day shows them; days rank by the first when nothing else is named PARAMETERS = parameters(list(FIELDS), "rain >= 1 and temp_max > 30", { "only": {"type": "string", "enum": ["public_holiday", "weekend"], @@ -56,59 +59,44 @@ "compare readings on the same day or count days (\"the hottest day it also rained\", \"how many days over " "35°C\", \"how many days was UV 9 or more\", \"the wettest day\", \"the windiest cold day\"). Fields: temp_max / temp_min (outdoor °C), " "rain (mm total for the day; a rainy day is 1 mm or more, above 0 is only a trace), wind_gust (km/h, highest), and a day's " - "highest (_max) or lowest (_min) of every other reading: humidity, pressure, wind_speed, dew_point, feels_like, vpd, " - "indoor_temp, indoor_humidity (_max and _min); solar, uv, rain_rate (_max only), e.g. uv_max >= 9; and _avg, a day's mean (temp_avg, " + "highest (_max) or lowest (_min) of every other reading: humidity, pressure, dew_point, feels_like, vpd, " + "indoor_temp, indoor_humidity (_max and _min); wind_speed, solar, uv, rain_rate (_max only), e.g. uv_max >= 9; and _avg, a day's mean (temp_avg, " "humidity_avg, pressure_avg ...). For \"how many\" use count_only; group_by month or year breaks it down and draws a bar chart. " "For a total, average, highest or lowest per month or year use stat and of (\"rain per month\": stat sum, of rain; \"hottest day each " "year\": stat max, of temp_max; \"average humidity per month\": stat avg, of humidity_avg). Returns the total of matching " "days and the top ones. For a record's value and the time it happened (hottest, coldest, fastest gust), use weather_history instead. For ONE known day, give start_date = end_date = that day and no conditions: it returns that day's figures. Can be limited to local public holidays or weekends (only). Works from cached history, so any period up to the whole record is fast.") -def load_cached(cache: HistoryCache, mac: str, cycle: str, names: list[str], start: int, end: int) -> dict: - """The cached readings of the wanted fields, folded by collect() into {"group.field": {"pts": ...}}.""" - data: dict = {} +def day_figures(cache: HistoryCache, mac: str, cycle: str, names: list[str], days: set[date], tz: tzinfo, start: int, + end: int) -> dict[str, dict[date, float]]: + """{reading: {day: its figure}} for each of `days`, from the cached readings between start and end (epochs): the day's highest + high, lowest low or mean. Each field is read once, however many readings use it (temp_max, temp_min and temp_avg are one), + and folded into days as it is read, one at a time, so a long period of 5-minute readings is not held whole.""" + readers: dict[tuple[str, str], list[str]] = {} for name in names: - group, field, _, _ = FIELDS[name] - found = cache.load_fields(mac, cycle, group, [field, f"{field}_high", f"{field}_low"], start, end) - data.setdefault(group, {}).update(found) - store: dict = {} - collect(store, data, cycle) - return store - - -def per_day(store: dict, name: str, days: set[date], tz: tzinfo) -> dict[date, float]: - """One extreme per day (the highest high, or the lowest low) of the readings that fall in `days`.""" - group, field, kind, _ = FIELDS[name] - out: dict[date, float] = {} - if kind == "mean": # the day's mean of its readings - sums: dict[date, list[float]] = {} - for ts, rec in store.get(f"{group}.{field}", {}).get("pts", {}).items(): - if (day := local_date(ts, tz)) in days and rec.get("value") is not None: - sums.setdefault(day, []).append(rec["value"][0]) - return {d: fmean(v) for d, v in sums.items()} - for ts, rec in store.get(f"{group}.{field}", {}).get("pts", {}).items(): - day = local_date(ts, tz) - found = rec.get(kind) or rec.get("value") - if day in days and found is not None: - if day not in out or (found[0] > out[day] if kind == "high" else found[0] < out[day]): - out[day] = found[0] + readers.setdefault(FIELDS[name][:2], []).append(name) + out: dict[str, dict[date, float]] = {} + for (group, field), wanted in readers.items(): + value, high, low = cache.slots(mac, cycle, group, [field, f"{field}_high", f"{field}_low"], start, end) + for name in wanted: + kind = FIELDS[name][2] + points = value if kind == "mean" else {**value, **(high if kind == "high" else low)} # a slot's own high or low, if it has one + figures: dict[date, list[float]] = {} + for ts, v in points.items(): + if (day := local_date(ts, tz)) in days: + got = figures.setdefault(day, []) + if kind == "mean" or not got: + got.append(v) + elif v > got[0] if kind == "high" else v < got[0]: + got[0] = v + out[name] = {d: fmean(v) if kind == "mean" else v[0] for d, v in figures.items()} return out def find_days(cache: HistoryCache, mac: str, tz: tzinfo, args: dict, now: datetime, turn=None) -> dict: - today = now.date() - try: - first = date.fromisoformat(str(args["start_date"])[:10]) - last = min(date.fromisoformat(str(args["end_date"])[:10]), today - timedelta(days=1)) - except (KeyError, ValueError) as e: - return {"error": f"bad date ({e}); use 'YYYY-MM-DD'"} - if first > last: - return {"error": "start_date must be before end_date (today isn't final, so the latest day is yesterday)"} - where = args.get("where") or [] - if problem := validate(args, {n: f[3] for n, f in FIELDS.items()}): - return {"error": problem} - sort_by = args.get("sort_by") or (where[0]["field"] if where else args.get("of") or "temp_max") - args = {**args, "sort_by": sort_by} + asked = prepare(args, FIELD_UNITS, ALWAYS_SHOWN, now.date()) + if isinstance(asked, str): + return {"error": asked} only = args.get("only") if only not in (None, "public_holiday", "weekend"): return {"error": "only must be public_holiday or weekend"} @@ -117,39 +105,35 @@ def find_days(cache: HistoryCache, mac: str, tz: tzinfo, args: dict, now: dateti return {"error": calendar.problem()} kind = {None: lambda d: True, "weekend": lambda d: d.weekday() >= 5, "public_holiday": lambda d: calendar.name(d) is not None}[only] - shown = list(dict.fromkeys([*ALWAYS_SHOWN, sort_by, *(c["field"] for c in where), *([args["of"]] if args.get("of") else [])])) + first, last, shown, where, sort_by = asked.first, asked.last, asked.shown, asked.where, asked.sort_by groups = sorted({FIELDS[n][0] for n in shown}) # which cycle each day is answered from: exact sub-daily data where every needed group has it - days = [first + timedelta(days=k) for k in range((last - first).days + 1)] + days = asked.days + windows = [(cycle, d) for cycle in FINE_CYCLES for d in days] source: dict[date, str] = {} - for cycle in SUB_DAILY: - cover = {g: cache.coverage(mac, cycle, g) for g in groups} - for d in days: - lo, hi = day_bounds(d, tz, True) - if d not in source and all(not subtract((lo, hi), cover[g]) for g in groups): - source[d] = cycle + for (cycle, d), gaps in zip(windows, cache.uncovered(mac, groups, [(cycle, *day_bounds(d, tz, True)) for cycle, d in windows])): + if d not in source and not any(gaps.values()): + source[d] = cycle rest = {d for d in days if d not in source} values: dict[str, dict[date, float]] = {n: {} for n in shown} - for cycle in SUB_DAILY: + for cycle in FINE_CYCLES: mine = {d for d, c in source.items() if c == cycle} if mine: - store = load_cached(cache, mac, cycle, shown, day_bounds(min(mine), tz, True)[0], day_bounds(max(mine), tz, True)[1]) - for n in shown: - values[n].update(per_day(store, n, mine, tz)) + for n, figures in day_figures(cache, mac, cycle, shown, mine, tz, day_bounds(min(mine), tz, True)[0], + day_bounds(max(mine), tz, True)[1]).items(): + values[n].update(figures) if rest: # daily buckets: labelled with the local date their 10am start falls on - store = load_cached(cache, mac, "1day", shown, day_bounds(min(rest), tz, True)[0] - 86400, day_bounds(max(rest), tz, True)[1] + 86400) - for n in shown: - values[n].update(per_day(store, n, rest, tz)) + for n, figures in day_figures(cache, mac, "1day", shown, rest, tz, day_bounds(min(rest), tz, True)[0] - 86400, + day_bounds(max(rest), tz, True)[1] + 86400).items(): + values[n].update(figures) for d in rest: if any(d in values[n] for n in shown): source[d] = "daily" checked = [d for d in days if kind(d) and any(d in values[n] for n in shown)] - units = {n: FIELDS[n][3] for n in shown} - label = lambda d: f"{d:%a} {d.day} {d:%b %Y}" - got = analyse(values, shown, units, args, checked, first, last, tz, turn, + got = analyse(values, asked, checked, tz, turn, lambda d: {**({"holiday": calendar.name(d)} if only == "public_holiday" else {}), "source": "daily" if source.get(d) == "daily" else "exact"}, {"public_holiday": "public holidays only", "weekend": "weekends only"}.get(only, "")) @@ -171,15 +155,15 @@ def find_days(cache: HistoryCache, mac: str, tz: tzinfo, args: dict, now: dateti break exact_days = sorted(d for d in checked if source.get(d) != "daily") - if args.get("count_only") or (args.get("stat") not in (None, "count") and args.get("group_by")): + if not lists_days(args): trace = [] # just the figures if trace: out["trace_rain_days"] = [row(d) for d in trace] out["note_trace"] = (f"These days rank higher but had less than {rain_cond['value']:g} mm of rain (a trace), so they " "are not in \"days\". Mention the first as a short footnote.") if any(source.get(d) == "daily" for d in checked): - out["exact_from"] = label(exact_days[0]) if exact_days else None - out["note_daily"] = ("Days marked \"daily\" come from daily readings that run 10am to 10am, so their rain and dates " + out["exact_from"] = day_text(exact_days[0]) if exact_days else None + out["note_daily"] = (f"Days marked \"daily\" come from daily readings that run {daily_run(tz, first, last)}, so their rain and dates " "are approximate (a morning shower can be counted on the day before). Say so briefly if any " "listed day is marked daily, or if the period reaches back before exact_from.") if AVERAGED & set(shown) and any(source.get(d) != "5min" for d in checked): @@ -204,4 +188,4 @@ async def days_tool(cache: HistoryCache, mac: str, tz: tzinfo, args: dict, turn= args = grouped_as_asked(args, turn) now = now_local(tz) result = await asyncio.to_thread(find_days, cache, mac, tz, args, now, turn) - return json.dumps(result, ensure_ascii=False, separators=(",", ":")) + return json_result(result) diff --git a/lib/ecowitt/direction.py b/lib/ecowitt/direction.py index bffdd3c..30140fb 100644 --- a/lib/ecowitt/direction.py +++ b/lib/ecowitt/direction.py @@ -7,7 +7,7 @@ from datetime import tzinfo from statistics import fmean -from ..timeutil import local_date +from ..timeutil import day_text, local_date POINTS = ("N", "NNE", "NE", "ENE", "E", "ESE", "SE", "SSE", "S", "SSW", "SW", "WSW", "W", "WNW", "NW", "NNW") TOP = 3 # compass points listed, most common first @@ -63,7 +63,7 @@ def summarise(readings: list[tuple[int, float, bool]], tz: tzinfo, by_day: bool, days: dict = {} for ts, d, _ in readings: days.setdefault(local_date(ts, tz), []).append(d) - out["daily"] = {day.strftime("%a %d %b"): f"{Counter(point(d) for d in ds).most_common(1)[0][0]} " + out["daily"] = {day_text(day, year=False): f"{Counter(point(d) for d in ds).most_common(1)[0][0]} " f"({_share(Counter(point(d) for d in ds).most_common(1)[0][1], len(ds))})" for day, ds in sorted(days.items())} return out diff --git a/lib/ecowitt/extremes.py b/lib/ecowitt/extremes.py index 9ac6e21..e4f2f63 100644 --- a/lib/ecowitt/extremes.py +++ b/lib/ecowitt/extremes.py @@ -1,10 +1,10 @@ """Turning Ecowitt readings into per-series points and finding their lowest and highest, with when each happened.""" -from datetime import date, datetime, tzinfo, UTC +from datetime import datetime, tzinfo, UTC from typing import NamedTuple -from ..timeutil import local_date -from .api import CYCLE_SECONDS +from ..timeutil import clock_text, day_text +from .api import CYCLES from .fetch import series_in @@ -68,20 +68,12 @@ def high_of(rec: dict) -> float: return rec["high"][0] if "high" in rec else rec["value"][0] -def daily_readings(pts: dict, before: date | None = None, tz: tzinfo | None = None) -> list[tuple]: - """(ts, value, low, high, fine) for the 5- and 30-minute readings (those before `before`), for daily_summary.""" - return [(t, r["value"][0], low_of(r), high_of(r), r["cycle"] == "5min") for t, r in pts.items() - if r["cycle"] != "1day" and "value" in r and (before is None or local_date(t, tz) < before)] +def daily_readings(pts: dict) -> list[tuple]: + """(ts, value, low, high, fine) for the 5- and 30-minute readings, for daily_summary.""" + return [(t, r["value"][0], low_of(r), high_of(r), r["cycle"] == "5min") for t, r in pts.items() if r["cycle"] != "1day" and "value" in r] -def _clock(dt: datetime) -> str: - """12-hour clock, e.g. '7:05am', '3:30pm', '10am'.""" - text = dt.strftime("%I:%M%p").lstrip("0").lower() - return text.replace(":00", "") if dt.minute == 0 else text - - -def _day(dt: datetime) -> str: - return f"{dt:%a} {dt.day} {dt:%b %Y}" +_clock = lambda dt: clock_text(dt, whole_hours=True) # "7:05am", "3:30pm", "10am" def describe_time(ts: int, cycle: str, tz: tzinfo) -> tuple[str, str, str]: @@ -90,12 +82,12 @@ def describe_time(ts: int, cycle: str, tz: tzinfo) -> tuple[str, str, str]: the window it happened in, so the model can copy it as is. The date is separate (so an emoji can go before it) and empty when the window spans two days, since then the wording has both.""" start = datetime.fromtimestamp(ts, UTC).astimezone(tz) - raw = start.strftime("%a %d %b %Y %H:%M") + raw = f"{day_text(start)} {clock_text(start)}" if cycle == "5min": - return raw, f"at {_clock(start)}", _day(start) + return raw, f"at {_clock(start)}", day_text(start) if cycle == "30min": - return raw, f"around {_clock(start)}", _day(start) - end = datetime.fromtimestamp(ts + CYCLE_SECONDS[cycle], UTC).astimezone(tz) + return raw, f"around {_clock(start)}", day_text(start) + end = datetime.fromtimestamp(ts + CYCLES[cycle].seconds, UTC).astimezone(tz) if start.date() == end.date(): - return raw, f"sometime between {_clock(start)} and {_clock(end)}", _day(start) - return raw, f"sometime between {_clock(start)} {_day(start)} and {_clock(end)} {_day(end)}", "" + return raw, f"sometime between {_clock(start)} and {_clock(end)}", day_text(start) + return raw, f"sometime between {_clock(start)} {day_text(start)} and {_clock(end)} {day_text(end)}", "" diff --git a/lib/ecowitt/fetch.py b/lib/ecowitt/fetch.py index 648035c..ce8080c 100644 --- a/lib/ecowitt/fetch.py +++ b/lib/ecowitt/fetch.py @@ -5,11 +5,11 @@ import functools import logging import time -from datetime import datetime, timedelta, tzinfo, UTC +from datetime import date, datetime, timedelta, tzinfo -from ..timeutil import local_epoch, to_local -from .api import EcowittError, MAX_SPAN, RETENTION -from .store import HistoryCache, HotStore, merge as merge_intervals, subtract +from ..timeutil import local_epoch, naive_local +from .api import CYCLES, EcowittError, kept_from +from .store import HistoryCache, HotStore, merge as merge_intervals log = logging.getLogger(__name__) @@ -26,7 +26,7 @@ def series_in(data: dict): def spans(cycle: str, t: datetime, until: datetime): """(start, end) pieces of [t, until) that each fit Ecowitt's per-request limit for this cycle.""" while t < until: - e = min(t + MAX_SPAN[cycle] - timedelta(seconds=1), until) + e = min(t + CYCLES[cycle].span - timedelta(seconds=1), until) yield t, e t = e + timedelta(seconds=1) @@ -73,11 +73,12 @@ def epoch(self, local: datetime) -> int: return local_epoch(local, self.tz) def local(self, epoch: int) -> datetime: - return to_local(epoch, self.tz).replace(tzinfo=None) + return naive_local(epoch, self.tz) @timed async def get(self, cycle: str, start: datetime, end: datetime, refresh: bool = False, load: bool = True) -> dict: - """Readings for [start, end] (local time). refresh=True ignores the in-memory copy (used by + """Readings for [start, end] (local time), however long: what Ecowitt is asked for is split to its per-request limit. + refresh=True ignores the in-memory copy (used by the keep-warm refresh). load=False only makes sure the range is cached and in memory, and skips building the result nobody will read (the refresh and the archive). When the cache and memory hold it all (the usual case) it is read at once, even while a refresh is fetching; otherwise @@ -106,14 +107,16 @@ async def _read(self, cycle: str, s: int, e: int, load: bool, fetch: bool, refre continue if not fetch: return None - data = await self._fetch(cycle, self.local(gap_start), self.local(gap_end), need) - if data is None: - continue # failed: not recorded as fetched, so it's retried next time - await asyncio.to_thread(self.cache.store, self.mac, cycle, need, data, gap_start, gap_end) - for g in need: - self.hot.put(self.mac, cycle, g, gap_start, gap_end, data.get(g) or {}) - if load: - merge_data(fresh, data) + for a, b in spans(cycle, self.local(gap_start), self.local(gap_end)): # one request each, within Ecowitt's limit + data = await self._fetch(cycle, a, b, need) + if data is None: + continue # failed: not recorded as fetched, so it's retried next time + piece = (self.epoch(a), self.epoch(b)) + await asyncio.to_thread(self.cache.store, self.mac, cycle, need, data, *piece) + for g in need: + self.hot.put(self.mac, cycle, g, *piece, data.get(g) or {}) + if load: + merge_data(fresh, data) self.from_cache += not gaps self.from_memory += from_memory if not load: @@ -144,8 +147,7 @@ async def read(self, fn, *args): async def _missing(self, cycle: str, windows: list[tuple[int, int]]) -> list[dict[str, list[tuple[int, int]]]]: """For each window (epoch seconds), the parts of it the cache lacks, by group: one read of the coverage in one thread hop (the hop costs more than the query, and far more while a chart is being drawn).""" - held = await asyncio.to_thread(self.cache.coverage_of, self.mac, cycle, self.groups) if self.groups else {} - return [{g: subtract(w, held[g]) for g in self.groups} for w in windows] + return await asyncio.to_thread(self.cache.uncovered, self.mac, self.groups, [(cycle, s, e) for s, e in windows]) async def covered(self, cycle: str, start: int, end: int) -> bool: """True if the cache holds every group for [start, end] (epoch seconds).""" @@ -156,13 +158,13 @@ async def covered_each(self, cycle: str, windows: list[tuple[int, int]]) -> list """covered() for each of these windows, from one read of the coverage.""" return [bool(self.groups) and not any(missing.values()) for missing in await self._missing(cycle, windows)] - async def finer_cycle(self, ts: int, current: str, window_end: int, now_utc: datetime) -> str | None: + async def finer_cycle(self, ts: int, current: str, window_end: int, today: date) -> str | None: """Finest cycle finer than `current` available for this window: still kept by Ecowitt, or already in the cache (e.g. archived 5-minute data). None if neither.""" - age = (now_utc - datetime.fromtimestamp(ts, UTC)).days + day = self.local(ts).date() for cycle in ("5min", "30min", "4hour"): if cycle == current: return None - if age < RETENTION[cycle] - 1 or await self.covered(cycle, ts, window_end): + if day >= kept_from(cycle, today) or await self.covered(cycle, ts, window_end): return cycle return None diff --git a/lib/ecowitt/link.py b/lib/ecowitt/link.py index 4dcad96..c9d0b40 100644 --- a/lib/ecowitt/link.py +++ b/lib/ecowitt/link.py @@ -8,7 +8,6 @@ before and the rain after; and the biggest rain spells with the change that preceded each.""" import asyncio -import json import logging from datetime import date, datetime, time, tzinfo @@ -19,22 +18,19 @@ from ..panels import panel_for from ..series import WEATHER from ..specs import Chart, Line, stack -from ..timeutil import day_bounds, local_date, now_local, parse_period +from ..timeutil import day_bounds, day_text, local_date, now_local, parse_period, period_parameters +from ..tools import json_result from .store import HistoryCache log = logging.getLogger(__name__) DEFAULT_DAYS = 90 -# driver -> (group, field, unit, the change over 3 hours that counts as falling or rising) -DRIVERS = {"pressure": ("pressure", "relative", "hPa", 1.0), - "humidity": ("outdoor", "humidity", "%", 5.0), - "wind": ("wind", "wind_speed", "km/h", 5.0)} +DRIVERS = {"pressure": 1.0, "humidity": 5.0, "wind": 5.0} # a reading (series) -> the change over 3 hours that counts as falling or rising PARAMETERS = { "type": "object", "properties": { - "start_date": {"type": "string", "description": f"First day, 'YYYY-MM-DD'. Default: the last {DEFAULT_DAYS} days."}, - "end_date": {"type": "string", "description": "Last day, 'YYYY-MM-DD' (up to yesterday). Default: yesterday."}, + **period_parameters(DEFAULT_DAYS), "driver": {"type": "string", "enum": list(DRIVERS), "description": "The reading to relate to rain. Default pressure (relative)."}, "chart": {"type": "boolean", "description": "Set true for a chart: the reading as a line, rain as bars behind it."}, @@ -48,10 +44,25 @@ "whole record; never averaged to days.") +def reading_slots(cache: HistoryCache, mac: str, name: str, start: int, end: int) -> tuple[dict, dict, dict]: + """(values, lows, highs) per 30-minute slot of a reading (series) from the cache, as every chart of it shades them: the + band is Ecowitt's own lows and highs where it gives them, else the cached 5-minute readings' (store.slot_ranges). A + reading shaded up to another field (wind) has its band reach that field's highest (the gusts).""" + reading = WEATHER[name] + group, field = reading.group, reading.field + values, lows, highs = cache.slots(mac, "30min", group, [field, field + "_low", field + "_high"], start, end) + if reading.band_field: + gust, gust_high = cache.slots(mac, "30min", group, [reading.band_field, reading.band_field + "_high"], start, end) + highs = {t: max(gust.get(t, 0.0), gust_high.get(t, 0.0)) for t in {*gust, *gust_high}} + elif not (lows or highs): + lows, highs = cache.slot_ranges(mac, group, field, values, start, end) + return values, lows, highs + + def driver_series(driver: dict[int, float], tz: tzinfo, first: date, last: date, label: str, lows: dict[int, float] | None = None, highs: dict[int, float] | None = None, banded_short: bool = True) -> Line | None: - """The reading as a line (see lines.build_line); a day's mean has its range shaded, from Ecowitt's own 30-minute - lows and highs where the cache holds them.""" + """The reading as a line (see lines.build_line); a day's mean has its range shaded, from the slots' lows and highs + (reading_slots) where there are any.""" line = build_line(slot_readings(driver, lows, highs), tz, ((last - first).days + 1) * 86400) return line.unbanded((last - first).days + 1, keep=banded_short).spec(label) if line else None @@ -73,12 +84,13 @@ def link(cache: HistoryCache, mac: str, tz: tzinfo, args: dict, now: datetime) - name = args.get("driver") or "pressure" if name not in DRIVERS: return {"error": f"driver must be one of {', '.join(DRIVERS)}"}, None, None, None - group, field, unit, threshold = DRIVERS[name] + reading, threshold = WEATHER[name], DRIVERS[name] + group, field, unit = reading.group, reading.field, reading.unit lo, hi = day_bounds(first, tz)[0], day_bounds(last, tz, last_second=True)[1] - driver, lows, highs = cache.slots(mac, "30min", group, [field, field + "_low", field + "_high"], lo, hi) + driver, lows, highs = reading_slots(cache, mac, name, lo, hi) (daily,) = cache.slots(mac, "30min", "rainfall", ["daily"], lo, hi) rain = rain_slots(daily) - out = {"period": f"{first:%a} {first.day} {first:%b %Y} - {last:%a} {last.day} {last:%b %Y}", "driver": f"{name} ({unit})", + out = {"period": f"{day_text(first)} - {day_text(last)}", "driver": f"{name} ({unit})", "resolution": "30-minute readings (not averaged to days)", "how_to_read": ("by_change_before groups each 30-minute slot by how the reading had changed over the previous 3 " f"hours: falling = down {threshold:g} {unit} or more, rising = up {threshold:g} or more. share_of_rain " @@ -105,4 +117,4 @@ async def link_tool(cache: HistoryCache, mac: str, tz: tzinfo, args: dict, turn) if spec and wants_chart(args, turn, datetime.combine(first, time()), datetime.combine(last, time())): turn.charts.append(spec) out["chart"] = LINK_CHART_HINT - return json.dumps(out, ensure_ascii=False, separators=(",", ":")) + return json_result(out) diff --git a/lib/ecowitt/query.py b/lib/ecowitt/query.py index 16cd1d4..44de919 100644 --- a/lib/ecowitt/query.py +++ b/lib/ecowitt/query.py @@ -11,20 +11,21 @@ """ import asyncio -import json import logging -from datetime import date, datetime, time, timedelta, UTC +from datetime import date, datetime, time, timedelta from statistics import fmean +from typing import NamedTuple from ..captions import AVERAGE_CHART_HINT, CHART_HINT, STACK_CHART_HINT, DIRECTION_CHART_HINT, wants_chart -from ..lines import build_line +from ..lines import Sample, build_line, daily_summary from ..daytable import grouping_in from ..rain import rain_bars, rain_slots -from ..timeutil import day_bounds, daily_summary, local_date, now_local +from ..timeutil import day_bounds, day_text, local_date, now_local, parse_local, SLOT +from ..tools import json_result from ..series import WEATHER, derives_range, field_of, find, find_name from ..panels import panel_for from ..specs import Bars, Chart, Compass, Line, Panel, period_text, stack -from .api import CYCLE_SECONDS, RETENTION +from .api import CYCLES, daily_run, kept_from from .direction import SPEED_STEPS, rose as direction_rose, summarise as summarise_direction from .extremes import Ext, better, collect, daily_readings, describe_time, fold, high_of, low_of, series_extremes from .fetch import Fetcher, spans @@ -32,7 +33,7 @@ log = logging.getLogger(__name__) INTRADAY_DAYS = 8 # up to this many days: 5-minute readings where archived -FINE_DAYS = 31 # up to this many: 30-minute readings; longer periods use daily records +FINE_DAYS = 31 # up to this many: 30-minute readings and a breakdown by day; longer periods use daily records, by month DAILY_ALL_DAYS = 7 # longer periods send every day's figures only when day by day was asked for DIRECTION_DAYS = 365 # how far back wind direction is counted (from cached 5-minute readings) MAX_REFINE_WINDOWS = 4 # overall records @@ -62,6 +63,13 @@ def consecutive(days: list[date]) -> list[list[date]]: return runs +def chart_field(args: dict, turn) -> str | None: + """The one reading a chart is asked for (the person's words, else the model's chart_field) as the field it plots, however it + is written ("UV" and "uvi" -> "uvi", "wind gust" -> "wind_speed": wind is its speed shaded up to the gusts); None if none.""" + named = str(turn.chart_field or args.get("chart_field") or "").strip().lower().replace(" ", "_") + return field_of(named) if named else None + + def stack_names(args: dict, turn) -> list[str]: """The readings to put side by side (from the person's words, else the model's chart_fields), two or more.""" raw = turn.chart_fields or args.get("chart_fields") or [] @@ -69,6 +77,14 @@ def stack_names(args: dict, turn) -> list[str]: return names if len(names) >= 2 else [] +class Target(NamedTuple): + """An extreme that _refine looks for at a finer time: the period's record ("all") or a month's.""" + scope: str # "all" or "month" + key: str # "group.field" + want: str # "low" or "high" + month: str = "" # a month's record: its row in the breakdown + + # ---------- one history question ---------- class HistoryQuery: """Answers one question: fetch, find the extremes, refine them, and build the compact @@ -77,7 +93,6 @@ class HistoryQuery: def __init__(self, fetcher: Fetcher, args: dict, turn): self.f, self.tz, self.args, self.turn = fetcher, fetcher.tz, args, turn self.now = now_local(self.tz) - self.now_utc = datetime.now(UTC) self.store: dict = {} # "group.field" -> {"unit", "pts"}, filled by _fetch_period self.start = self.end = self.span = None # the period, set by run() self.held = False # the whole period is in the cache at 30 minutes, and fits the row budget @@ -90,6 +105,11 @@ def __init__(self, fetcher: Fetcher, args: dict, turn): self.compass: Compass | None = None # the wind rose (counts per compass point by speed), drawn beside the wind chart self.wind: Fetcher | None = None # the wind rose's own (5-minute readings), made when wind direction is counted + @property + def by_day(self) -> bool: + """The answer breaks the period down by day (up to FINE_DAYS), else by month.""" + return self.span <= timedelta(days=FINE_DAYS) + async def run(self) -> str: try: return await self._run() @@ -99,14 +119,14 @@ async def run(self) -> str: async def _run(self) -> str: today = self.now.date() try: - start = datetime.fromisoformat(str(self.args.get("start_date") or today).replace("T", " ")) - end = datetime.fromisoformat(str(self.args.get("end_date") or self.now).replace("T", " ")) + start = parse_local(self.args.get("start_date") or today, self.tz) + end = parse_local(self.args.get("end_date") or self.now, self.tz) except ValueError as e: return f"Error: bad date ({e}). Use 'YYYY-MM-DD HH:MM:SS'." - self.start = max(start.replace(tzinfo=None), datetime.combine(today - timedelta(days=RETENTION["1day"] - 2), time())) - self.end = min(end.replace(tzinfo=None), self.now) + self.start = max(start, datetime.combine(kept_from("1day", today), time())) + self.end = min(end, self.now) if self.start >= self.end: - return "Error: start_date must be before end_date and within the last 4 years." + return f"Error: start_date must be before end_date and within the last {CYCLES['1day'].kept // 365} years." self.span = self.end - self.start self.held = await self._held_locally() self.detailed = self.held or self.span <= timedelta(days=DETAILED_DAYS) @@ -121,7 +141,7 @@ async def _run(self) -> str: return "Error: no history data returned." + ( " Details: " + "; ".join(self.f.errors[:5]) if self.f.errors else "") self.overall = {k: ex for k, s in self.store.items() if len(ex := series_extremes(s)) == 2} - self.monthly = self._monthly_extremes() if self.span > timedelta(days=31) else {} + self.monthly = {} if self.by_day else self._monthly_extremes() refined = await self._refine() return self._answer(refined) @@ -161,19 +181,17 @@ async def _held_locally(self) -> bool: f = self.f if (self.span.days + 1) * 48 * len(f.groups) * FIELDS_PER_GROUP > MAX_ROWS: return False - return all(await f.covered_each("30min", [(f.epoch(t), f.epoch(e)) for t, e in spans("30min", self.start, self.end - timedelta(days=1))])) + return await f.covered("30min", f.epoch(self.start), f.epoch(self.end - timedelta(days=1))) - async def _chunks(self, cycle: str, t: datetime, until: datetime): - for a, b in spans(cycle, t, until): - collect(self.store, await self.f.get(cycle, a, b), cycle) + async def _read(self, cycle: str, t: datetime, until: datetime): + collect(self.store, await self.f.get(cycle, t, until), cycle) async def _fetch_period(self): """Coarse pass: the best data for the whole period at the fewest requests.""" f, start, end, today = self.f, self.start, self.end, self.now.date() - age_days = (today - start.date()).days - thirty_floor = datetime.combine(today - timedelta(days=RETENTION["30min"] - 2), time()) + thirty_floor = datetime.combine(kept_from("30min", today), time()) recent = datetime.combine(today - timedelta(days=6), time()) - if self.span <= timedelta(days=1) and (age_days < RETENTION["5min"] - 1 or await f.covered( + if self.span <= timedelta(days=1) and (start.date() >= kept_from("5min", today) or await f.covered( "5min", f.epoch(start), f.epoch(end))): collect(self.store, await f.get("5min", start, end), "5min") elif self.span <= timedelta(days=FINE_DAYS) and end > thirty_floor: @@ -183,14 +201,14 @@ async def _fetch_period(self): five = self.span <= timedelta(days=INTRADAY_DAYS) and await f.covered("5min", f.epoch(a), f.epoch(b)) collect(self.store, await f.get("5min" if five else "30min", a, b), "5min" if five else "30min") if start < thirty_floor: - await self._chunks("1day", start, min(end, thirty_floor - timedelta(seconds=1))) + await self._read("1day", start, min(end, thirty_floor - timedelta(seconds=1))) elif self.held: - await self._chunks("30min", start, end) # held in the cache: exact local days, true lows and highs, no requests + await self._read("30min", start, end) # held in the cache: exact local days, true lows and highs, no requests else: # Longer: Ecowitt's daily records (each has its own low and high, and it is a request or two, cached), # plus the newest days at 30 minutes so the line reaches today. Exact times come from refining the extremes. - await self._chunks("30min", max(start, recent), end) - await self._chunks("1day", start, min(end, recent - timedelta(seconds=1))) + await self._read("30min", max(start, recent), end) + await self._read("1day", start, min(end, recent - timedelta(seconds=1))) async def _summarise_direction(self): """Wind direction leaves the low/high pipeline: it is counted by compass point instead. Recent @@ -210,7 +228,8 @@ async def _summarise_direction(self): if a < b: bounds[day] = a, b held = await wind.covered_each("5min", [(wind.epoch(a), wind.epoch(b)) for a, b in bounds.values()]) - fine_days = {day for day, whole in zip(bounds, held) if whole or day == self.end.date()} # today's comes from memory or one request + fine_days = {day for day, whole in zip(bounds, held) # today's (or a settling yesterday's) comes from memory or one request + if whole or day == self.end.date() and (self.now.date() - day).days <= 1} fine: dict = {} for run in consecutive(sorted(fine_days)): # a run of days is one read collect(fine, await wind.get("5min", bounds[run[0]][0], bounds[run[-1]][1]), "5min") @@ -225,7 +244,7 @@ async def _summarise_direction(self): counted = [(t, d, exact) for t, (d, exact) in sorted(readings.items()) if first <= f.local(t)] calm = [x for x in counted if speeds.get(x[0], 1) <= 0] counted = [x for x in counted if speeds.get(x[0], 1) > 0] - result = summarise_direction(counted, self.tz, self.span <= timedelta(days=31), len(calm)) + result = summarise_direction(counted, self.tz, self.by_day, len(calm)) if not result: continue self.compass = Compass(direction_rose([(t, d, x, speeds.get(t)) for t, d, x in counted]), @@ -243,7 +262,7 @@ def _monthly_extremes(self) -> dict: for ts, rec in sorted(series["pts"].items()): if until is None or ts >= until: day = local_date(ts, self.tz) - month, until = day.strftime("%b %Y"), day_bounds(day, self.tz)[1] + month, until = self._label(day), day_bounds(day, self.tz)[1] fold(months.setdefault(month, {}), ts, rec) out[key] = months return out @@ -254,64 +273,61 @@ async def _refine(self) -> int: for periods of up to a few months, each month's temperature records. A second pass narrows a 30-minute slot to archived 5-minute data. Returns the number of windows fetched.""" f = self.f - slots: dict = {} # ("all", key, want) or ("month", key, month, want) -> Ext + slots: dict[Target, Ext] = {} for key in sorted(self.overall, key=lambda k: "temperature" not in k): for want in ("low", "high"): - slots[("all", key, want)] = self.overall[key][want] + slots[Target("all", key, want)] = self.overall[key][want] if self.monthly and self.detailed: for key, months in self.monthly.items(): if key.endswith(".temperature"): for month, d in reversed(list(months.items())): # newest first: where 5-minute data is for want, ext in d.items(): - slots[("month", key, month, want)] = ext + slots[Target("month", key, want, month)] = ext caps = {"all": MAX_REFINE_WINDOWS, "month": MAX_MONTH_REFINE_WINDOWS} refined = 0 for _ in range(2): - windows: dict = {} # (cycle, start_ts, end_ts) -> [slot id] + windows: dict = {} # (cycle, start_ts, end_ts) -> [Target] used = {"all": 0, "month": 0} finer_of: dict = {} # slots on the same bucket share an answer; a new pass looks again (data was fetched) - for sid, ext in slots.items(): - w_end = ext.ts + CYCLE_SECONDS[ext.cycle] - 1 + for target, ext in slots.items(): + w_end = ext.ts + CYCLES[ext.cycle].seconds - 1 if (ext.ts, ext.cycle) not in finer_of: - finer_of[ext.ts, ext.cycle] = await f.finer_cycle(ext.ts, ext.cycle, w_end, self.now_utc) + finer_of[ext.ts, ext.cycle] = await f.finer_cycle(ext.ts, ext.cycle, w_end, self.now.date()) finer = finer_of[ext.ts, ext.cycle] if not finer: continue w = (finer, ext.ts, w_end) if w not in windows: - if used[sid[0]] >= caps[sid[0]]: + if used[target.scope] >= caps[target.scope]: continue - used[sid[0]] += 1 + used[target.scope] += 1 windows[w] = [] - windows[w].append(sid) + windows[w].append(target) if not windows: break refined += len(windows) for (cycle, w_start, w_end), targets in windows.items(): fine: dict = {} - for a, b in spans(cycle, f.local(w_start), min(f.local(w_end), self.now)): # a UTC day is 25 local hours across a DST change - collect(fine, await f.get(cycle, a, b), cycle) - for sid in targets: - key, want = sid[1], sid[-1] - found = series_extremes(fine[key], w_start, w_end).get(want) if key in fine else None + collect(fine, await f.get(cycle, f.local(w_start), min(f.local(w_end), self.now)), cycle) + for target in targets: + found = series_extremes(fine[target.key], w_start, w_end).get(target.want) if target.key in fine else None if not found: continue - cur = slots[sid] - keep = cur if better(want, cur.value, found.value) else found # the more extreme value... - slots[sid] = Ext(keep.value, keep.raw, found.ts, found.cycle, keep.exact) # ...at the finer time - for sid, ext in slots.items(): - if sid[0] == "all": - self.overall[sid[1]][sid[2]] = ext + cur = slots[target] + keep = cur if better(target.want, cur.value, found.value) else found # the more extreme value... + slots[target] = Ext(keep.value, keep.raw, found.ts, found.cycle, keep.exact) # ...at the finer time + for target, ext in slots.items(): + if target.scope == "all": + self.overall[target.key][target.want] = ext else: - self.monthly[sid[1]][sid[2]][sid[3]] = ext + self.monthly[target.key][target.month][target.want] = ext return refined def _answer(self, refined: int) -> str: f = self.f wanted = self.args.get("include_derived") or [] wanted = {wanted} if isinstance(wanted, str) else set(wanted) - asked = [self.turn.chart_field or self.args.get("chart_field"), *(WEATHER[n].field for n in stack_names(self.args, self.turn))] - asked = [a and field_of(a) for a in asked] + asked = [chart_field(self.args, self.turn), *(WEATHER[n].field for n in stack_names(self.args, self.turn))] wanted |= {f for f in asked if f in DERIVED} # charting one of them brings it into the result if "app_temp" in wanted: wanted.add("app_tempin") # indoor's name for apparent temperature @@ -322,16 +338,17 @@ def _answer(self, refined: int) -> str: log.log(logging.DEBUG if self.turn.ahead else logging.INFO, "%s..%s: %d reads (%d cache, %d mem), %d req%s, %d refined", f"{self.start:%m-%d}", f"{self.end:%m-%d}", f.ranges, f.from_cache, f.from_memory, f.calls, f", {len(f.errors)} failed" if f.errors else "", refined) - out = {"period": f"{self.start:%a %d %b %Y} - {self.end:%a %d %b %Y}", "series": shown if shown else series_out} + out = {"period": f"{day_text(self.start)} - {day_text(self.end)}", "series": shown if shown else series_out} if self.monthly and not self.detailed: - out["monthly_note"] = ("Monthly figures for long periods come from daily data that runs 10am-10am, so they " - "have no dates, and a low early on the 1st may be counted in the previous month.") + out["monthly_note"] = (f"Monthly figures for long periods come from daily data that runs " + f"{daily_run(self.tz, self.start.date(), self.end.date())}, so they have no dates, and a low " + "early on the 1st may be counted in the previous month.") if wants_chart(self.args, self.turn, self.start, self.end): self._add_charts(out, {k: v for k, v in series_out.items() if k in self.store}) - if f.errors: - out["missing"] = f.errors[:10] + if errors := f.errors + (self.wind.errors if self.wind else []): + out["missing"] = errors[:10] out["warning"] = "Some data could not be fetched; the answer may be incomplete. Say so." - return json.dumps(out, ensure_ascii=False, separators=(",", ":")) + return json_result(out) def _in_focus(self, key: str) -> bool: """Is this series wanted by the question? Outdoor and indoor come with both temperature and humidity: when the words name @@ -350,15 +367,15 @@ def _result(self, key: str, ext: dict) -> dict: entry[f"{want}_time"], entry[f"{want}_when"], entry[f"{want}_date"] = describe_time(e.ts, e.cycle, tz) if not e.exact: entry[f"{want}_note"] = "from averaged data; the real value may be more extreme" - if self.span <= timedelta(days=31): + if self.by_day: days: dict = {} for ts, rec in self.store[key]["pts"].items(): if rec["cycle"] != "1day": # sub-daily points give correct local days fold(days.setdefault(local_date(ts, tz), {}), ts, rec) keep = None if len(days) <= DAILY_ALL_DAYS or self.turn.per_day else { # a long period: the days of its records, not all of them - local_date(ext[w].ts, tz).strftime("%a %d %b") for w in ("low", "high")} - entry["daily"] = {d.strftime("%a %d %b"): {w: e.raw for w, e in days[d].items()} for d in sorted(days) - if keep is None or d.strftime("%a %d %b") in keep} + self._label(local_date(ext[w].ts, tz)) for w in ("low", "high")} + entry["daily"] = {self._label(d): {w: e.raw for w, e in days[d].items()} for d in sorted(days) + if keep is None or self._label(d) in keep} else: entry["monthly"] = {} for month, d in self.monthly.get(key, {}).items(): @@ -397,6 +414,11 @@ def _add_charts(self, out: dict, plottable: dict): out["chart"] = (CHART_HINT + " For wind direction, give the most common direction, not a high and low." if spec and wind is not spec else DIRECTION_CHART_HINT) + def _label(self, day: date) -> str: + """The row a day belongs to in the breakdown, in the one form every table of it uses: the day ("Sat 3 Oct"), or its month + ("Oct 2026") when the period is broken down by month.""" + return day_text(day, year=False) if self.by_day else f"{day:%b %Y}" + def _daily_means(self, pts: dict) -> dict: """{local date: mean of that day's readings}: from 5- or 30-minute readings where held, else the daily bucket.""" means = {local_date(ts, self.tz): rec["value"][0] for ts, rec in pts.items() if rec.get("cycle") == "1day" and "value" in rec} @@ -404,15 +426,14 @@ def _daily_means(self, pts: dict) -> dict: return means def _add_averages(self, entry: dict, means: dict): - """The mean over the period, and per day (up to 31 days) or per month: the answer to "what was the average".""" + """The mean over the period, and per row of the breakdown (each day, or each month): the answer to "what was the average".""" if not means: return fmt = lambda v: f"{v:.1f}" entry["average"] = fmt(fmean(means.values())) + table = entry.get("daily" if self.by_day else "monthly") for d, v in means.items(): - table = entry.get("daily") if self.span <= timedelta(days=31) else entry.get("monthly") - label = d.strftime("%a %d %b") if self.span <= timedelta(days=31) else d.strftime("%b %Y") - if table is not None and label in table: + if table is not None and (label := self._label(d)) in table: table[label].setdefault("_sum", []).append(v) for table in (entry.get("daily"), entry.get("monthly")): for row in (table or {}).values(): @@ -424,9 +445,8 @@ def _chart_spec(self, series_out: dict, field: str | None = None) -> Chart | Non """Line chart: one line per group for the field asked about (`field`, else chart_field; temperature by default, else the first field), at the finest resolution fetched for the whole period (5- or 30-minute readings, or daily averages for long periods), plus the true record high and low with their times.""" - asked = field or self.turn.chart_field or self.args.get("chart_field") - wanted = str(asked or "temperature").strip().lower().replace(" ", "_") - wanted = field_of(wanted) # a reading's name ("uv", "pressure") or its field ("uvi") + asked = field_of(field) if field else chart_field(self.args, self.turn) + wanted = asked or "temperature" keys = [k for k in series_out if k.endswith("." + wanted)] if not keys and (wanted in DERIVED or asked and find_name(wanted)): # asked for on its own and not there: no chart, rather than a temperature one return None @@ -436,8 +456,9 @@ def _chart_spec(self, series_out: dict, field: str | None = None) -> Chart | Non keys = [k for k in series_out if k.endswith("." + field)] if keys[0] == "rainfall.daily" and self.rain_bars.x: # the day's counter is a running total: draw what fell, as columns return Chart("Rain", f"{period_text(self.start.date(), self.end.date())} · rain per {self.rain_bars.per}" - + (" · older days from daily records (10am to 10am)" if self.rain_from_daily else ""), - [panel_for("rain", bars=self.rain_bars)]) + + (f" · older days from daily records ({daily_run(self.tz, self.start.date(), self.end.date())})" + if self.rain_from_daily else ""), + [panel_for("rain", bars=self.rain_bars)], period=(self.start.date(), self.end.date())) field = keys[0].split(".", 1)[-1] unit = series_out[keys[0]]["unit"].replace("º", "°") lines, resolution, ranged = [], None, False @@ -464,7 +485,7 @@ def _chart_spec(self, series_out: dict, field: str | None = None) -> Chart | Non + (", shaded up to the gusts" if banded and ranged else ", range shaded" if ranged else "") + (" · records marked" if any(x.records for x in lines) and not banded else "")) panel = panel_for(name, lines) if (name := find_name(field)) else Panel(title, unit, lines) - return Chart(panel.label, subtitle, [panel]) + return Chart(panel.label, subtitle, [panel], period=(self.start.date(), self.end.date())) def _band_key(self, k: str) -> str | None: """The series this one is shaded up to, if it is a reading with one (wind: the gusts) and that series was fetched.""" @@ -473,7 +494,7 @@ def _band_key(self, k: str) -> str | None: key = f"{group}.{reading.band_field}" if reading and reading.band_field and field == reading.field else None return key if key in self.store else None - def _series_readings(self, k: str) -> list: + def _series_readings(self, k: str) -> list[Sample]: """One series as readings for lines.build_line. A reading with a band series (wind) is its mean shaded up to that series.""" band = self._band_key(k) gust = self.store[band]["pts"] if band else {} @@ -484,25 +505,29 @@ def _series_readings(self, k: str) -> list: low, high = (r["low"][0] if "low" in r else None), (r["high"][0] if "high" in r else None) if band: low, high = low_of(r), max(high_of(gust[t]) if t in gust and "value" in gust[t] else high_of(r), v) - out.append((t, v, low, high, CYCLE_SECONDS[r["cycle"]])) + out.append(Sample(t, v, low, high, CYCLES[r["cycle"]].seconds)) return out - def _series_entry(self, k: str, label: str | None = None) -> tuple[Line, str] | None: + def _series_entry(self, k: str) -> tuple[Line, str] | None: """One series as a chart line and how it was drawn; None if there is too little to draw.""" line = build_line(self._series_readings(k), self.tz, self.span.total_seconds(), smooth=k in SMOOTH_SERIES, force_daily=self.turn.average_asked, until=now_local(self.tz).date()) if line is None: return None group = k.split(".", 1)[0] - line.unbanded(self.span.days + 1, keep=self._band_key(k) is not None) # a short period: a clean line, except wind's gusts - return line.spec(label or group.replace("_", " ").capitalize(), indoor=group == "indoor"), line.name + line = line.unbanded(self.span.days + 1, keep=self._band_key(k) is not None) # a short period: a clean line, except wind's gusts + return line.spec(group.replace("_", " ").capitalize(), indoor=group == "indoor"), line.name async def _rain_bars(self) -> Bars: - """Rain for the chart, summed into bars. From the cached 30-minute readings; days older than those (about a year) come from + """Rain for the chart, summed into bars. From the cached 30-minute readings and, after the last of them, the query's own + (the newest hours are not in the cache yet: they are still settling); days older than those (about a year) come from the daily records, the day's total, which are approximate (10am to 10am). A figure per month or year asked for in the person's words (`by`) gets one bar for each, whatever the length of the period.""" lo, hi = self.f.epoch(self.start), self.f.epoch(self.end) (daily,) = await self.f.read(self.f.cache.slots, self.f.mac, "30min", "rainfall", ["daily"], lo, hi) + after = max(daily) + SLOT if daily else lo # the cache stops at the settle horizon: what is newer, this query read itself + daily.update({t: r["value"][0] for t, r in self.store.get("rainfall.daily", {}).get("pts", {}).items() + if t >= after and r["cycle"] in ("5min", "30min") and "value" in r}) rain = rain_slots(daily) held = {local_date(t, self.tz) for t in rain} (old,) = await self.f.read(self.f.cache.slots, self.f.mac, "1day", "rainfall", ["daily"], lo - 86400, hi + 86400) @@ -511,7 +536,7 @@ async def _rain_bars(self) -> Bars: if day not in held and self.start.date() <= day <= self.end.date(): rain[day_bounds(day, self.tz)[0] + 12 * 3600] = mm self.rain_from_daily = True - by = grouping_in(self.turn.text) if self.span > timedelta(days=31) else None + by = None if self.by_day else grouping_in(self.turn.text) return rain_bars(rain, self.tz, self.start.date(), self.end.date(), until=hi, by=by, since=lo) def _stack_spec(self, names: list[str]) -> Chart | None: diff --git a/lib/ecowitt/source.py b/lib/ecowitt/source.py index ff4c0e6..7301293 100644 --- a/lib/ecowitt/source.py +++ b/lib/ecowitt/source.py @@ -3,31 +3,30 @@ readings ready, readings() feeds the alert monitors.""" import asyncio -import json import logging import os import time from datetime import datetime, timedelta, UTC +from zoneinfo import ZoneInfo, ZoneInfoNotFoundError from ..config import Config -from ..intent import NOW_GROUPS -from ..timeutil import now_local -from ..tools import Tool, Turn +from ..sources import Source +from ..timeutil import local_epoch, naive_local, when +from ..tools import Tool, Turn, json_result from ..warm import FAST_REFRESH_SECONDS, Warmer -from .api import EcowittAPI, GROUPS, UNITS +from .api import FINE_CYCLES, EcowittAPI, GROUPS, UNITS, unit_text from .calendar import PublicHolidays from .link import DESCRIPTION as LINK_DESCRIPTION, PARAMETERS as LINK_PARAMETERS, link_tool from .days import DESCRIPTION as DAYS_DESCRIPTION, PARAMETERS as DAYS_PARAMETERS, days_tool from .glance import glance -from .fetch import Fetcher, spans -from ..series import WEATHER, find -from .query import HistoryQuery, stack_names +from .fetch import Fetcher, series_in +from ..series import READING_GROUPS, WEATHER, find +from .query import HistoryQuery, chart_field, stack_names from .store import HistoryCache, HotStore log = logging.getLogger(__name__) DEFAULT_GROUPS = "outdoor,indoor" -FAST_CYCLES = ("5min", "30min") # kept warm by refetching every cycle; the 4-hour and daily tails only when their memory copy is stale def parse_groups(value, default: str = DEFAULT_GROUPS) -> list[str]: @@ -43,16 +42,12 @@ def parse_groups(value, default: str = DEFAULT_GROUPS) -> list[str]: def readings(data: dict) -> list[tuple[int, dict]]: """Ecowitt 'data' -> [(epoch, {"group.field": value})] sorted by time.""" rows: dict[int, dict] = {} - for grp, fields in data.items(): - if not isinstance(fields, dict): - continue - for field, obj in fields.items(): - if isinstance(obj, dict) and isinstance(obj.get("list"), dict): - for ts, v in obj["list"].items(): - try: - rows.setdefault(int(ts), {})[f"{grp}.{field}"] = float(v) - except (TypeError, ValueError): - pass + for group, field, obj in series_in(data): + for ts, v in obj["list"].items(): + try: + rows.setdefault(int(ts), {})[f"{group}.{field}"] = float(v) + except (TypeError, ValueError): + pass return sorted(rows.items()) @@ -89,7 +84,7 @@ def readings(data: dict) -> list[tuple[int, dict]]: REALTIME_DESCRIPTION = "Current readings from the owner's Ecowitt weather station." -LIVE_GROUPS = NOW_GROUPS.split(",") # what the once-a-minute live reading asks for: everything "weather now" and the report show +LIVE_GROUPS = list(READING_GROUPS) # what the once-a-minute live reading asks for: everything "weather now" and the report show LIVE_FRESH_SECONDS = 90 # a question is answered from that reading while it is this young @@ -100,7 +95,7 @@ def get(group: str, field: str) -> str | None: obj = (data.get(group) or {}).get(field) if not isinstance(obj, dict) or "value" not in obj: return None - unit = obj.get("unit", "").replace("℃", "°C").replace("mm/hr", "mm/h") + unit = unit_text(obj.get("unit")) or "" return f"{obj['value']}{'' if unit in ('°C', '%') else ' '}{unit}".strip() wind = get("wind", "wind_speed") parts = [f"rain {rate}" if (rate := get("rainfall", "rain_rate")) else None, @@ -110,7 +105,7 @@ def get(group: str, field: str) -> str | None: return ", ".join(p for p in parts if p) or "no readings" -class Ecowitt: +class Ecowitt(Source): name = "Ecowitt" def __init__(self, cfg: Config, transport=None): @@ -125,8 +120,8 @@ def __init__(self, cfg: Config, transport=None): self.mac = "" self.station_name = "" self.created: datetime | None = None - self.longitude = 145.0 # Melbourne; only used to time the pressure tide self.latitude: float | None = None # the station's own location (the forecast uses it unless one is configured) + self.longitude: float | None = None self.tools = [Tool("weather_now", REALTIME_DESCRIPTION, REALTIME_PARAMS, self._realtime), Tool("weather_history", HISTORY_DESCRIPTION, HISTORY_PARAMS, self._history), Tool("weather_days", DAYS_DESCRIPTION, DAYS_PARAMETERS, self._days), @@ -139,6 +134,11 @@ async def start(self): device = devices[0] self.mac = str(device["mac"]).upper() self.station_name = device.get("name") or "station" + if zone := device.get("date_zone_id"): # the station's own time zone ("Australia/Melbourne"): its days are local to it + try: + self.tz = ZoneInfo(zone) + except (ValueError, ZoneInfoNotFoundError): + log.warning("Ecowitt names a time zone Python doesn't know, %r; using %s", zone, self.tz) try: self.created = datetime.fromtimestamp(int(device["createtime"]), UTC).astimezone(self.tz) except (KeyError, TypeError, ValueError): @@ -153,7 +153,7 @@ async def start(self): pass if (problem := PublicHolidays(self.tz).problem()) and "package" in problem: log.warning("Public holiday questions won't work: pip install holidays (then restart)") - log.info("Weather station: Ecowitt '%s' (%s)%s", self.station_name, self.mac, + log.info("Weather station: Ecowitt '%s' (%s), %s%s", self.station_name, self.mac, self.tz, f", devices found: {len(devices)}" if len(devices) > 1 else "") try: size = os.path.getsize(self.cache.path) if os.path.exists(self.cache.path) else 0 @@ -169,16 +169,12 @@ def fetcher(self, groups: list[str], urgent: bool = False) -> Fetcher: """urgent: for a question (its requests go before the background ones).""" return Fetcher(self.api, self.cache, self.hot, self.mac, groups, self.tz, urgent) - def now(self) -> datetime: - return now_local(self.tz) - # ---------- tools ---------- async def _history(self, args: dict, turn: Turn | None = None) -> str: turn = turn or Turn() groups = parse_groups(args.get("groups")) groups += [g for n in stack_names(args, turn) if (g := WEATHER[n].group) not in groups] # what "plot temperature and rain" needs - named = str(turn.chart_field or args.get("chart_field") or "").strip().lower() # ... and what a chart of one reading needs - if (r := find(named)) and r.group not in groups: + if (r := find(chart_field(args, turn) or "")) and r.group not in groups: # ... and what a chart of one reading needs groups.append(r.group) return await HistoryQuery(self.fetcher(groups, urgent=not turn.ahead), args, turn).run() @@ -205,9 +201,7 @@ async def _realtime(self, args: dict, turn: Turn | None = None) -> str: newest = max(newest, int(obj.get("time") or 0)) except (TypeError, ValueError): pass - when = datetime.fromtimestamp(newest, UTC).astimezone(self.tz).strftime("%a %d %b %Y %H:%M") if newest else None - return json.dumps({"time": when, **out, **({"emoji": emoji} if emoji else {})}, - ensure_ascii=False, separators=(",", ":")) + return json_result({"time": when(newest, self.tz) if newest else None, **out, **({"emoji": emoji} if emoji else {})}) async def _live_data(self, groups: list[str]) -> dict: """The real_time data for these groups: the keep-warm's reading when it is under a minute and a half old and asked for them, @@ -251,27 +245,23 @@ async def warm(self, fresh: bool = True) -> str: "4hour": today - timedelta(days=3), "1day": today - timedelta(days=3)} fetchers = {cycle: self.fetcher(self.groups) for cycle in windows} - async def one(cycle: str): - for start, end in spans(cycle, windows[cycle], now): # each piece fits Ecowitt's per-request limit - await fetchers[cycle].get(cycle, start, end, refresh=fresh and cycle in FAST_CYCLES, load=False) - await asyncio.gather(*(one(cycle) for cycle in windows)) + await asyncio.gather(*(fetchers[cycle].get(cycle, start, now, refresh=fresh and cycle in FINE_CYCLES, load=False) + for cycle, start in windows.items())) return f"Ecowitt {sum(f.calls for f in fetchers.values())} req" def wants(self, text: str) -> bool: """Should a question start refreshing this source? Weather data is used by nearly all.""" return True - def poke(self): - self.warmer.poke() - async def readings(self, cycle: str, start: datetime, end: datetime, groups: list[str] | None = None) -> list[tuple[int, dict]]: """Readings for the alert monitors, straight from the warm data.""" return readings(await self.fetcher(groups or self.groups).get(cycle, start, end)) async def recent(self, hours: float) -> list[tuple[int, dict]]: + """The last `hours` of 5-minute readings: real hours, so the night the clocks change is no longer or shorter.""" now = self.now() - return await self.readings("5min", now - timedelta(hours=hours), now) + return await self.readings("5min", naive_local(local_epoch(now, self.tz) - hours * 3600, self.tz), now) async def close(self): await self.api.close() diff --git a/lib/ecowitt/store.py b/lib/ecowitt/store.py index 27dc46d..8832665 100644 --- a/lib/ecowitt/store.py +++ b/lib/ecowitt/store.py @@ -19,14 +19,11 @@ from bisect import bisect_left from ..timeutil import SLOT -from .api import UNIT_FIXES +from .api import CYCLES, UNIT_FIXES log = logging.getLogger(__name__) SCHEMA_VERSION = "1" -BUCKET_SECONDS = {"5min": 300, "30min": 1800, "4hour": 14400, "1day": 86400} -# How long after a bucket ends before its data is treated as final -SETTLE_SECONDS = {"5min": 3600, "30min": 7200, "4hour": 8 * 3600, "1day": 86400} # Missing readings older than this are accepted as genuinely missing (station offline, # before it existed). Newer gaps may just be Ecowitt running late, so they're re-asked. GAP_FINAL_SECONDS = 2 * 86400 @@ -40,7 +37,7 @@ def horizon(cycle: str, now: float | None = None) -> int: """Latest bucket start whose data is final.""" now = time.time() if now is None else now - return int(now - SETTLE_SECONDS[cycle] - BUCKET_SECONDS[cycle]) + return int(now - CYCLES[cycle].settle - CYCLES[cycle].seconds) def subtract(span: Interval, covered: list[Interval]) -> list[Interval]: @@ -125,24 +122,22 @@ def count(self, mac: str, cycle: str, grp: str, start: int, end: int) -> int: return self._query("SELECT COUNT(*) FROM points p JOIN series s ON s.id = p.series " "WHERE s.mac=? AND s.cycle=? AND s.grp=? AND p.ts BETWEEN ? AND ?", mac, cycle, grp, start, end)[0][0] - def newest(self, mac: str, cycle: str) -> int | None: - """The time of the newest reading held at this resolution (None if none).""" - return self._query("SELECT MAX(p.ts) FROM points p JOIN series s ON s.id = p.series WHERE s.mac=? AND s.cycle=?", mac, cycle)[0][0] - def coverage(self, mac: str, cycle: str, grp: str) -> list[Interval]: """Every time range held for this group at this resolution (merged, oldest first).""" return merge(self._query("SELECT start, end FROM coverage WHERE mac=? AND cycle=? AND grp=?", mac, cycle, grp)) - def coverage_of(self, mac: str, cycle: str, groups: list[str]) -> dict[str, list[Interval]]: - """coverage() of each of these groups, in one query.""" - held: dict[str, list[Interval]] = {g: [] for g in groups} - for grp, start, end in self._query(f"SELECT grp, start, end FROM coverage WHERE mac=? AND cycle=? AND grp IN ({','.join('?' * len(held))})", - mac, cycle, *held): - held[grp].append((start, end)) - return {g: merge(spans) for g, spans in held.items()} - - def missing(self, mac: str, cycle: str, grp: str, start: int, end: int) -> list[Interval]: - return subtract((start, end), self.coverage(mac, cycle, grp)) + def uncovered(self, mac: str, groups: list[str], windows: list[tuple[str, int, int]]) -> list[dict[str, list[Interval]]]: + """For each (cycle, start, end) window, the parts of it each group's readings are not held for: one read of the coverage, + however many windows, resolutions and groups.""" + cycles = sorted({cycle for cycle, _, _ in windows}) + held: dict[tuple[str, str], list[Interval]] = {(c, g): [] for c in cycles for g in groups} + if held: + for cycle, grp, start, end in self._query( + f"SELECT cycle, grp, start, end FROM coverage WHERE mac=? AND cycle IN ({','.join('?' * len(cycles))}) " + f"AND grp IN ({','.join('?' * len(groups))})", mac, *cycles, *groups): + held[cycle, grp].append((start, end)) + held = {key: merge(spans) for key, spans in held.items()} + return [{g: subtract((start, end), held[cycle, g]) for g in groups} for cycle, start, end in windows] def load_fields(self, mac: str, cycle: str, grp: str, fields: list[str] | None, start: int, end: int) -> dict: """A group's fields (all of them if `fields` is None) in Ecowitt's response shape: {field: {unit, list}}.""" @@ -212,8 +207,8 @@ def store(self, mac: str, cycle: str, groups: list[str], data: dict, start: int, if old_enough: covered.append((grp, limit)) elif reached is not None: - covered.append((grp, min(limit, reached + BUCKET_SECONDS[cycle] - 1))) - with self._lock: + covered.append((grp, min(limit, reached + CYCLES[cycle].seconds - 1))) + with self._lock, self.db: # one transaction: an error part-way rolls it back (no half-written response is kept) grown = set() # groups that now carry a reading the cache has never had (Ecowitt added a metric) for _, _, grp, field, _ in units: known = {r[0] for r in self.db.execute("SELECT field FROM series WHERE mac=? AND cycle=? AND grp=?", (mac, cycle, grp))} @@ -233,7 +228,6 @@ def store(self, mac: str, cycle: str, groups: list[str], data: dict, start: int, self.db.execute("DELETE FROM coverage WHERE mac=? AND cycle=? AND grp=?", (mac, cycle, grp)) self.db.executemany("INSERT INTO coverage VALUES (?,?,?,?,?)", [(mac, cycle, grp, s, e) for s, e in merge(existing + [(start, cov_end)])]) - self.db.commit() def load(self, mac: str, cycle: str, groups: list[str], start: int, end: int) -> dict: """Cached readings in Ecowitt's response shape: {grp: {field: {unit, list}}}.""" @@ -265,5 +259,10 @@ def get(self, mac: str, cycle: str, group: str, start: int, end: int) -> dict | return entry["fields"] if end <= entry["end"] or reaches_present else None def put(self, mac: str, cycle: str, group: str, start: int, end: int, fields: dict): + """Keep a response, unless a fresh one that reaches later is held: an old range (the archive's) is cached on disk, and must + not push out the copy of the last hours that questions are answered from.""" + held = self._entries.get((mac, cycle, group)) + if held and time.time() - held["fetched_at"] <= HOT_TTL_SECONDS and held["end"] > end: + return self._entries[(mac, cycle, group)] = {"start": start, "end": end, "fields": fields, "fetched_at": time.time()} diff --git a/lib/entities.py b/lib/entities.py new file mode 100644 index 0000000..99b7f7e --- /dev/null +++ b/lib/entities.py @@ -0,0 +1,21 @@ +"""A Telegram message's formatting as plain text plus entities (bold, italic, a link) rather than markup, so nothing in the text +needs escaping. Telegram places an entity in UTF-16 code units, where an emoji is two.""" + +from telegram import MessageEntity + + +def utf16_len(text: str) -> int: + """The length of `text` as Telegram counts it.""" + return len(text.encode("utf-16-le")) // 2 + + +def footer(text: str, line: str, style: str = MessageEntity.ITALIC, url: str | None = None) -> tuple[str, list[MessageEntity]]: + """`text` with `line` under it, set in `style` (italic: a small footer; bold: an error) and linked to `url` if there is one.""" + start, length = utf16_len(text) + 1, utf16_len(line) + links = [MessageEntity(MessageEntity.TEXT_LINK, start, length, url=url)] if url else [] + return f"{text}\n{line}", [MessageEntity(style, start, length), *links] + + +def with_link(text: str, link: tuple[str, str] | None) -> tuple[str, list[MessageEntity]]: + """`text`, trimmed, with the link (label, url) under it as a small italic footer, if there is one.""" + return footer(text.rstrip(), link[0], url=link[1]) if link else (text.rstrip(), []) diff --git a/lib/forecast/source.py b/lib/forecast/source.py index 9ad1787..f545fcb 100644 --- a/lib/forecast/source.py +++ b/lib/forecast/source.py @@ -1,20 +1,19 @@ """The forecast source: the daily forecast from Open-Meteo (open-meteo.com), kept warm and cached; fetched only from 6 am to 6 pm local time (lib/warm.py SYNC_HOURS), every 15 minutes, and once at the start (or the first question) when nothing is cached.""" -import json import logging import math import re import time -from datetime import date, datetime, timedelta +from datetime import date import httpx -from .. import intent from ..config import Config from ..snapshots import Snapshots -from ..timeutil import now_local, when -from ..tools import Tool, Turn +from ..sources import Source +from ..timeutil import when +from ..tools import Tool, Turn, json_result from ..warm import Warmer, in_sync_hours log = logging.getLogger(__name__) @@ -84,21 +83,22 @@ def day_label(day: date, today: date) -> str: return "Today" if day == today else f"{day:%A}" -class Forecast: +class Forecast(Source): name = "Forecast" def __init__(self, cfg: Config, location: tuple[float, float] | None = None, transport=None): """location: (lat, lon), used when the config has none (the weather station's own).""" self.tz, self.place = cfg.tz, cfg.place - lat = cfg.forecast_lat if cfg.forecast_lat is not None else (location[0] if location else None) - lon = cfg.forecast_lon if cfg.forecast_lon is not None else (location[1] if location else None) - self.location = (lat, lon) if lat is not None and lon is not None else None + configured = (cfg.forecast_lat, cfg.forecast_lon) + if configured != (None, None) and None in configured: + log.warning("FORECAST_LAT and FORECAST_LON go together: with only one set, the weather station's location is used") + self.location = configured if None not in configured else location if location and None not in location else None self.client = httpx.AsyncClient(timeout=15, headers=HEADERS, follow_redirects=True, transport=transport) self.days: list[dict] | None = None self.source = "" self.fetched_at = 0.0 self.requests = 0 - self.store = Snapshots.open(cfg.conditions_cache_path) + self.store = Snapshots(cfg.conditions_cache_path) self._load() self.warmer = Warmer(self.warm, FORECAST_REFRESH_SECONDS) self.tools = [Tool("weather_forecast", DESCRIPTION, PARAMETERS, self.handle)] @@ -119,13 +119,6 @@ def wants(self, text: str) -> bool: """Only forecast questions start a refresh; the report always uses what is cached.""" return bool(re.search(r"\bforecast\w*|\btomorrow\b|\bthis week\b", text, re.I)) - def poke(self): - self.warmer.poke() - - async def close(self): - await self.client.aclose() - self.store.close() - # ---------- kept on disk ---------- def _load(self): """Start from the last forecast fetched (before the restart), so a night-time start needs no fetch.""" @@ -139,9 +132,6 @@ def _save(self): "days": [{**d, "date": d["date"].isoformat()} for d in self.days]}, now=self.fetched_at) - def now(self) -> datetime: - return now_local(self.tz) - # ---------- fetching ---------- async def _json(self, url: str, **params) -> dict: self.requests += 1 @@ -159,7 +149,7 @@ async def _open_meteo(self) -> tuple[str, list[dict]]: "summary": WMO.get(dd["weather_code"][i], ""), "rain_chance_pct": dd["precipitation_probability_max"][i]} for i in range(len(dd["time"]))] - async def _ensemble(self, first: date, days: int) -> dict[date, int]: + async def _ensemble(self, days: int) -> dict[date, int]: """{local day: % of ensemble members with at least RAIN_MM that day}, to the nearest 5; a day with no usable members is left out. Every `hourly` series named precipitation* is one member (the response names them by member and model).""" lat, lon = self.location @@ -181,10 +171,10 @@ async def _ensemble(self, first: date, days: int) -> dict[date, int]: return {date.fromisoformat(day): 5 * round(100 * sum(t >= RAIN_MM for t in totals) / len(totals) / 5) for day, totals in sums.items()} - async def _with_chance(self, source: str, days: list[dict]) -> list[dict]: + async def _with_chance(self, days: list[dict]) -> list[dict]: """The days with the ensemble's chance of at least 1 mm added; Open-Meteo's own figure stays where the ensemble has none.""" try: - chance = await self._ensemble(days[0]["date"], len(days)) if days else {} + chance = await self._ensemble(len(days)) if days else {} except Exception as e: # the forecast still works without it log.warning("Ensemble rain chance unavailable (%s: %s); using Open-Meteo's own", type(e).__name__, e) return days @@ -197,7 +187,7 @@ async def warm(self, fresh: bool = True) -> str: return "Forecast cached" before = self.requests self.source, self.days = await self._open_meteo() # a failure raises: the warmer logs it - self.days = await self._with_chance(self.source, self.days) + self.days = await self._with_chance(self.days) self.fetched_at = time.time() self._save() return f"Forecast {self.requests - before} req" @@ -224,8 +214,8 @@ async def handle(self, args: dict, turn: Turn | None = None) -> str: count = 7 lines = self.lines(count) if not lines: - return json.dumps({"error": "No forecast is available right now."}) + return json_result({"error": "No forecast is available right now."}) if turn: turn.forecast_shown.extend(self.upcoming(count)) # remembered for the chat once the answer is sent stamped = {"time": when(int(self.fetched_at), self.tz)} if self.fetched_at else {} # when it was last fetched - return json.dumps({"lines": lines, "place": self.place, "source": self.source, **stamped}, ensure_ascii=False) + return json_result({"lines": lines, "place": self.place, "source": self.source, **stamped}) diff --git a/lib/intent.py b/lib/intent.py index 1927c90..de53ce7 100644 --- a/lib/intent.py +++ b/lib/intent.py @@ -10,17 +10,19 @@ import logging import re +from collections.abc import Collection from dataclasses import dataclass, field from datetime import date, datetime, timedelta -from .series import PRESSURE_WORDS, SPECIFIC, TEMP_WORDS, WEATHER as WEATHER_READINGS +from .daytable import GROUPING +from .series import AIR as AIR_TABLE, INDOOR_READINGS, PRESSURE_WORDS, READING_GROUPS, SPECIFIC, TEMP_WORDS, WEATHER as WEATHER_READINGS log = logging.getLogger(__name__) I = re.IGNORECASE # ---------- reasoning and fetching ---------- -EFFORT_DEFAULT, EFFORT_DESCRIBE, EFFORT_FORECAST = "none", "low", "medium" +EFFORT_DEFAULT, EFFORT_DESCRIBE, EFFORT_THINK = "none", "low", "medium" EFFORT_STEPS = ("none", "low", "medium", "high") @@ -39,16 +41,21 @@ def lower_effort(effort: str) -> str | None: # The two devices by name: "ecowitt" is the weather, "ag" / "airgradient" / "air gradient" the air quality _ECOWITT = r"ecowitt" _AG = r"air ?gradient|ag" +AIR_METRICS = [(m.words, name) for name, m in AIR_TABLE.items()] # words -> metric; checked in order, so PM10 wins over PM1 +_AIR_METRIC = "|".join(words for words, _ in AIR_METRICS) # the words that name an air reading, for every pattern below _OTHER_THAN_AIR = rf"temp\w*|rain\w*|wind\w*|humid\w*|hot|cold|warm|pressure|weather|{_ECOWITT}" # a question that is not (only) about the air +# The words of every reading but temperature and feels-like, which name their reading more broadly than a question asks for it +# ("cool, thanks", "feeling better"): those two ask with _TEMP_ASK and "feels" instead +_READING_ASK = "|".join(r.words for name, r in WEATHER_READINGS.items() if name not in ("temperature", "feels_like")) # Messages about the weather or air must fetch fresh data; anything else (thanks, chat) needn't WEATHER = re.compile( - rf"\b(weather|{_ECOWITT}|{_AG}|{_TEMP_ASK}|frost\w*|pollen|hay ?fever|asthma|" + rf"\b(weather|{_ECOWITT}|{_AG}|{_TEMP_ASK}|{_READING_ASK}|frost\w*|pollen|hay ?fever|asthma|" rf"rain\w*|showers?|drizzle|storms?|thunder\w*|hail|snow|fog\w*|cloud\w*|sun\w*|uv|solar|" rf"{_WIND}|humid\w*|dew|{_PRESSURE}|forecast\w*|umbrella|" r"highs?|lows?|max\w*|min\w*|records?|extremes?|average|chart\w*|graph\w*|plot\w*|trend\w*|" r"indoors?|outdoors?|inside|outside|conditions|feels?|today|tonight|tomorrow|yesterday|" - r"week|month|year|air|aqi|pm ?2\.?5|pm ?10|co2|voc\w*|nox|smok\w*|pollut\w*|stuffy)\b|°|" + rf"week|month|year|air|aqi|{_AIR_METRIC}|smok\w*|pollut\w*|stuffy)\b|°|" r"\b\d{1,2}(st|nd|rd|th)?\s+(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\w*|" r"\b(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\w*\s+\d{1,2}\b|\b\d{1,2}/\d{1,2}\b", I) @@ -81,7 +88,7 @@ def lower_effort(effort: str) -> str | None: # Asking the model to think: "think about it", "try to work out why", "reason it through", "predict", "estimate", "grind", "whirl" THINK = re.compile(r"\b(think\w*|try|trying|reason\w*|predict\w*|estimat\w*|grind\w*|whirl\w*)\b", I) -EFFORT_RULES = ((EFFORT_FORECAST, (LINK, THINK)), (EFFORT_DESCRIBE, (FORECAST, ANALYSIS, DESCRIBE))) +EFFORT_RULES = ((EFFORT_THINK, (LINK, THINK)), (EFFORT_DESCRIBE, (FORECAST, ANALYSIS, DESCRIBE))) def reasoning_effort(text: str) -> str: @@ -96,26 +103,29 @@ def _only(words: str, lead: str = "") -> str: return rf"^\s*(please\s+)?{lead}({words})(\s+please)?\s*[?.!]*\s*$" +_ASK = r"((give me|show me|show|get|what's|whats)\s+)?(the\s+)?" # the usual lead-in: "show me the ...", "what's the ..." + + # "status", "report": the whole current picture from both devices (a period word or a subject makes it something else) REPORT = re.compile(_only(r"status|report|sitrep|overview|dashboard|summary|everything|what you'?ve got", - r"((give me|show me|show|get|what's|whats)\s+)?(the\s+)?(a\s+)?(full\s+|current\s+|complete\s+)?"), I) + _ASK + r"(a\s+)?(full\s+|current\s+|complete\s+)?"), I) def wants_report(text: str) -> bool: return bool(REPORT.search(text)) and not TIME_WORDS.search(text) -def report_calls(ecowitt: bool, air: bool, pollen: bool = False, forecast: bool = False) -> list[tuple[str, dict]]: - """What the full report needs, fetched together before the model sees the question.""" - return ([("weather_now", {"groups": NOW_GROUPS})] if ecowitt else []) + ([("air_quality", {})] if air else []) + ( - [("pollen_asthma", {"cached": True})] if pollen else []) + ( - [("weather_forecast", {"days": 3, "cached": True})] if forecast else []) +def report_calls(sources: Collection[str]) -> list[tuple[str, dict]]: + """What the full report needs from these sources (by name), fetched together before the model sees the question.""" + calls = {"Ecowitt": ("weather_now", {"groups": NOW_GROUPS}), "AirGradient": ("air_quality", {}), + "Pollen": ("pollen_asthma", {"cached": True}), "Forecast": ("weather_forecast", {"days": 3, "cached": True})} + return [call for name, call in calls.items() if name in sources] # Pollen and thunderstorm asthma (the Pollen source) POLLEN_WORDS = re.compile(r"\b(pollen|hay ?fever|thunderstorm asthma|asthma)\b", I) POLLEN_NOW = re.compile(_only(r"pollen( count| level| forecast| today| now)?|hay ?fever|(thunderstorm )?asthma( risk)?", - r"((give me|show me|show|get|what's|whats)\s+)?(the\s+)?"), I) + _ASK), I) def mentions_pollen(text: str) -> bool: @@ -128,14 +138,18 @@ def mentions_pollen(text: str) -> bool: def pollen_ask(text: str) -> bool: """Is this only a plain question about the pollen or thunderstorm asthma ("is the pollen bad today", "how's the hay fever")?""" - return bool(mentions_pollen(text)) and all( - word in POLLEN_FILLER or word in FILLER for word in re.findall(r"[\w'’.]+", text.lower())) + return mentions_pollen(text) and _only_words(text, POLLEN_FILLER.__contains__) + + +def _only_words(text: str, fits) -> bool: + """Is every word of the text filler (FILLER: what, is, the, now ...) or one `fits` (a word -> bool) takes?""" + return all(word in FILLER or fits(word) for word in re.findall(r"[\w'’.]+", text.lower())) # "weather", "weather now", "current weather", "ecowitt": every reading the station has right now (the weather half of the report) -NOW_GROUPS = "outdoor,indoor,pressure,wind,rainfall,solar_and_uvi" +NOW_GROUPS = ",".join(READING_GROUPS) WEATHER_NOW = re.compile(_only(rf"(weather|{_ECOWITT}|conditions)(\s+(now|right now|currently|at the moment))?|(current|latest)\s+(weather|conditions)", - r"((give me|show me|show|get|what's|whats)\s+)?(the\s+)?"), I) + _ASK), I) def wants_weather_now(text: str) -> bool: @@ -220,7 +234,7 @@ def period_ranges(today: date) -> dict[str, tuple[date, date]]: def span(name: str, now: datetime) -> tuple[datetime, datetime]: """Start and end of a named period. The rolling "last 24 hours" ends now; others cover whole days.""" if name == "last 24 hours": - return now - timedelta(hours=24), now + return default_span(now)[1:] first, last = period_ranges(now.date())[name] return datetime.combine(first, datetime.min.time()), datetime.combine(last, datetime.max.time()).replace(microsecond=0) @@ -311,8 +325,8 @@ def period_hints(text: str, now: datetime) -> list[str]: r"extremes?|temperatures?|temps?)\b", I) GRAPH = re.compile(r"\b(graph\w*|chart\w*|plot\w*|trend\w*|visuali[sz]\w*)\b", I) # Anything that needs other data, a judgement, or a comparison goes the normal way -_NOT_SIMPLE = (r"rain\w*|pressure|humid\w*|uv|solar|lightning|pm ?2\.?5|pm2|pm ?10|pm ?1|" - rf"air|air quality|aqi?|co2|co₂|voc\w*|nox|smok\w*|pollut\w*|{_AG}|" +_NOT_SIMPLE = (rf"rain\w*|pressure|humid\w*|uv|solar|lightning|{_AIR_METRIC}|pm2|" + rf"air|air quality|aqi?|smok\w*|pollut\w*|{_AG}|" r"compare\w*|vs|versus|than|average|mean|median|why|how many|days (above|below|over|under)|" r"feels?|dew|forecast\w*|will|going to|tomorrow|tonight|later|now|current\w*|right now") NOT_SIMPLE = re.compile(rf"\b(wind\w*|gusts?|{_NOT_SIMPLE})\b", I) @@ -347,7 +361,8 @@ def weather_period(text: str, now: datetime) -> tuple[str, datetime, datetime] | AVERAGE = re.compile(r"\b(averages?|avg|mean)\b", I) -ALL = re.compile(r"\b(all|every\w*|each)\b", I) +# "all metrics", "everything", "aq all week" (every reading, a week), but not the period "all time" nor "every day" or "each month" +ALL = re.compile(r"\b(all(?![\s-]+time\b)|everything|(?:every|each)(?!\s+(?:days?|nights?|weeks?|months?|years?|hours?|times?)\b))\b", I) PER_DAY = re.compile(r"\b(each|every|per|by)\s+day|daily|day[- ]by[- ]day|which days|list|breakdown|one by one", I) @@ -362,7 +377,6 @@ def _named(text: str) -> dict[str, int]: FEELS_LIKE_IN_ALL = False # CHART_ALL_FEELS_LIKE=on puts the feels-like panel in "weather all week" (set from the config at startup) VPD_IN_ALL = False # CHART_ALL_VPD=on puts the vapour pressure deficit panel in "weather all week" (it is temperature and humidity combined) ALL_ORDER = ("temperature", "humidity", "solar", "pressure", "rain", "dew_point", "feels_like", "vpd", "uv", "wind") # the panels of "weather all week", top to bottom -INDOOR_READINGS = ("temperature", "humidity", "dew_point", "feels_like") # what the indoor sensor measures LEFT_OUT_OF_ALL = ("uv",) # "weather all week" leaves these out: the UV index has the same shape as solar radiation @@ -383,7 +397,7 @@ def wants_chart_all(text: str) -> bool: if named < 2 and ALL.search(text) and WEATHER_WORD.search(text): return True asked = GRAPH.search(text) or WEATHER_WORD.search(text) and _has_days(text, 2) # "weather 30d" is a chart too - return named == 0 and bool(asked) and not mentions_air(text) and not any(p.search(text) for p in JUDGEMENT) + return named == 0 and bool(asked) and not mentions_air(text) and not _judgement(text) def chart_field(text: str) -> str | None: @@ -405,11 +419,24 @@ def wants_rain_caption(text: str) -> bool: JUDGEMENT = (FORECAST, LINK, THINK, ANALYSIS, DESCRIBE) # questions that want the model's thinking, not just a chart +def _judgement(text: str) -> bool: + return any(p.search(text) for p in JUDGEMENT) + + def _sides(text: str) -> list[str]: indoor, outdoor = bool(INDOOR.search(text)), bool(OUTDOOR.search(text)) return ["indoor"] if indoor and not outdoor else ["outdoor"] if outdoor and not indoor else ["outdoor", "indoor"] +def _groups(text: str, names: list[str]) -> list[str]: + """The Ecowitt groups for these readings. Temperature and humidity come from both sensors unless one side is named; the + rest of what the indoor sensor measures (dew point, feels like) from the indoor one only when it is named.""" + groups = list(dict.fromkeys(WEATHER_READINGS[n].group for n in names)) + if "outdoor" in groups and ({"temperature", "humidity"} & set(names) or INDOOR.search(text) and set(INDOOR_READINGS) & set(names)): + groups = [g for g in groups if g != "outdoor"] + _sides(text) + return list(dict.fromkeys(groups)) + + def bare_chart_ask(text: str, now: datetime) -> bool: """Nothing but readings (or "weather") and a period: "humidity 7d", "weather 30d". Anything with a time range is a chart.""" if not _has_days(text, 2, now) or not (_named(text) or WEATHER_WORD.search(text)): @@ -418,24 +445,21 @@ def bare_chart_ask(text: str, now: datetime) -> bool: for said, _ in _periods(text, now).values(): rest = re.sub(rf"\b{re.escape(said)}\b", " ", rest, flags=I) patterns = [re.compile(rf"({r.words}|weather|{_ECOWITT}|conditions|indoors?|outdoors?)", I) for r in WEATHER_READINGS.values()] - return all(word in FILLER or any(p.fullmatch(word) for p in patterns) for word in re.findall(r"[\w'’.]+", rest.lower())) + return _only_words(rest, lambda word: any(p.fullmatch(word) for p in patterns)) def weather_chart(text: str, now: datetime) -> tuple[str, datetime, datetime, list[str]] | None: """(period name, start, end, Ecowitt groups) for a plain chart of named readings ("rain chart 7d", "plot temperature and humidity"), else None. A comparison, a forecast or a question that wants thinking is the model's.""" if (not (GRAPH.search(text) or bare_chart_ask(text, now)) or mentions_air(text) or _NOT_CHART.search(text) - or any(p.search(text) for p in JUDGEMENT) or not (fields := chart_fields(text) or list(_named(text)))): + or _judgement(text) or not (fields := chart_fields(text) or list(_named(text)))): return None spans = spans_in(text, now) if not spans and not TIME_WORDS.search(text): spans = [default_span(now)] # a chart with no period is the last day if len(spans) != 1: return None - groups = list(dict.fromkeys(WEATHER_READINGS[f].group for f in fields)) - if "outdoor" in groups and {"temperature", "humidity"} & set(fields): # these two have an indoor sensor as well - groups = [g for g in groups if g != "outdoor"] + _sides(text) - return (*spans[0], list(dict.fromkeys(groups))) + return (*spans[0], _groups(text, fields)) AVERAGE_READINGS = ("temperature", "humidity", "pressure", "wind", "dew_point", "feels_like", "vpd", "solar", "uv") # a rain total has no average @@ -447,16 +471,13 @@ def weather_chart(text: str, now: datetime) -> tuple[str, datetime, datetime, li def weather_average(text: str, now: datetime) -> tuple[str, datetime, datetime, list[str]] | None: """(period name, start, end, Ecowitt groups) for a plain "average temp last week" or "highest humidity yesterday" (the chart-less cousin of weather_chart).""" - if (not (AVERAGE.search(text) or HILO.search(text)) or mentions_air(text) or _NOT_CHART.search(text) or any(p.search(text) for p in JUDGEMENT) + if (not (AVERAGE.search(text) or HILO.search(text)) or mentions_air(text) or _NOT_CHART.search(text) or _judgement(text) or not (names := list(_named(text))) or not set(names) <= set(AVERAGE_READINGS)): return None spans = spans_in(text, now) if len(spans) != 1: return None - groups = list(dict.fromkeys(WEATHER_READINGS[n].group for n in names)) - if "outdoor" in groups and {"temperature", "humidity"} & set(names): - groups = [g for g in groups if g != "outdoor"] + _sides(text) - return (*spans[0], list(dict.fromkeys(groups))) + return (*spans[0], _groups(text, names)) def weather_groups(text: str) -> str: @@ -470,7 +491,7 @@ def weather_groups(text: str) -> str: # ---------- air-quality fast path ---------- # "how's the air?", "what's the AQI", "report my airgradient aq". History, comparisons and mixed # weather questions go the normal way. -AIR = re.compile(rf"\b(air|aqi|aq|pm ?2\.?5|pm ?10|pm ?1|co2|voc\w*|nox|smok\w*|pollut\w*|particulates?|particles?|{_AG})\b", I) +AIR = re.compile(rf"\b(air|aqi|aq|{_AIR_METRIC}|smok\w*|pollut\w*|particulates?|particles?|{_AG})\b", I) AIR_QUALITY = re.compile(r"\bair quality\b", I) AIR_NOT_NOW = re.compile( r"\b(yesterday|overnight|last|past|week|month|year|since|earlier|this morning|was|were|been|trend\w*|" @@ -478,10 +499,6 @@ def weather_groups(text: str) -> str: r"average|highest|lowest|peak|max\w*|min\w*|record\w*|" rf"{_OTHER_THAN_AIR})\b", I) AIR_CHART_NOT = re.compile(rf"\b(compare\w*|vs|versus|than|why|forecast\w*|tomorrow|will|{_OTHER_THAN_AIR})\b", I) -AIR_METRICS = [ # word -> metric; checked in order, so PM10 wins over PM1 - (r"pm ?2\.?5|pm25", "pm2_5"), (r"pm ?10", "pm10"), (r"pm ?1(?![0-9])", "pm1"), (r"co2|co₂", "co2"), - (r"voc\w*", "voc_index"), (r"nox", "nox_index"), -] def mentions_air(text: str) -> bool: @@ -496,7 +513,7 @@ def air_period(text: str, now: datetime) -> tuple[str, datetime, datetime] | Non return None spans = spans_in(text, now) if not spans and GRAPH.search(text) and not TIME_WORDS.search(text): - spans = [("last 24 hours", *numbered_span("24", "h", now)[1:])] + spans = [default_span(now)] return spans[0] if len(spans) == 1 else None @@ -529,19 +546,16 @@ def air_metrics(text: str) -> list[str]: "now currently current right at moment outside outdoors outdoor indoors indoor inside out there any we point".split()) FORECAST_PLAIN = re.compile(_only(r"(weather\s+)?forecast(\s+(for\s+)?(the\s+)?(this\s+|next\s+)?(week|7 days|seven days|coming days))?|" r"(the\s+)?(7|seven)[- ]day (weather )?forecast|(the\s+)?week ahead", - r"((give me|show me|show|get|what's|whats)\s+)?(the\s+)?"), I) + _ASK), I) def reading_now(text: str) -> list[str] | None: """The readings named when the text is only a plain question about them right now ("how hot is it", "is it raining", "uv"), else None.""" if (GRAPH.search(text) or TIME_WORDS.search(text) or SPECIFIC_MOMENT.search(text) or mentions_air(text) - or any(p.search(text) for p in JUDGEMENT) or not (names := list(_named(text))) or len(names) > 2): + or _judgement(text) or not (names := list(_named(text))) or len(names) > 2): return None patterns = [re.compile(rf"({WEATHER_READINGS[n].words}|indoors?|outdoors?)", I) for n in names] - for word in re.findall(r"[\w'’.]+", text.lower()): - if word not in FILLER and not any(p.fullmatch(word) for p in patterns): - return None - return names + return names if _only_words(text, lambda word: any(p.fullmatch(word) for p in patterns)) else None # Counting, ranking or totalling over periods ("count days over PM2.5 of 90 per year", "how many times", "rain per month"): the model @@ -552,18 +566,52 @@ def reading_now(text: str) -> list[str] | None: # "days over UVI 10 by year", "count days over PM2.5 of 90 per year", "how many days was UV 9 or more": one reading against one limit, # counted in code with weather_days or air_days (the model is not asked, so the answer is always the same bar chart or count) -GROUPING = re.compile(r"\b(?:per|by|each|every)\s+(year|month)\b|\b(yearly|annual\w*)\b|\b(monthly)\b", I) OR_MORE = re.compile(r"\bor (?:more|higher|above|greater|over)\b", I) OR_LESS = re.compile(r"\bor (?:less|lower|below|under)\b", I) AT_LEAST = re.compile(r"\b(?:at least|reach\w*|hit|hits)\b", I) ABOVE = re.compile(r"\b(?:over|above|more than|greater than|higher than|exceed\w*)\b", I) BELOW = re.compile(r"\b(?:under|below|less than|lower than|fewer than|at most)\b", I) -DAY_COUNT = {"temperature": "temp", "humidity": "humidity", "pressure": "pressure", "dew_point": "dew_point", "feels_like": "feels_like", - "vpd": "vpd", "solar": "solar", "uv": "uv"} # reading -> the stem of its daily field (temp_max, uv_max ...) +def _figures(name: str, stem: str) -> tuple[dict[str, str], str]: + """A reading's daily fields by stat (temp_max, temp_min, temp_avg); none for its lowest when it has no low worth telling.""" + return {"max": f"{stem}_max", **({"min": f"{stem}_min"} if WEATHER_READINGS[name].low else {}), "avg": f"{stem}_avg"}, "avg" + + +# A reading's weather_days field for each stat a question asks of it ("max" its highest, "min" its lowest, "avg" its average, +# "sum" its total), and the stat when it names none. Without a "min" it has no lowest to ask for. +DAILY = { + **{name: _figures(name, stem) for name, stem in (("temperature", "temp"), ("humidity", "humidity"), ("pressure", "pressure"), + ("dew_point", "dew_point"), ("feels_like", "feels_like"), ("vpd", "vpd"), + ("solar", "solar"))}, + "uv": ({"max": "uv_max", "min": "uv_max", "avg": "uv_max"}, "max"), # a UV index is its day's high, whatever is asked of it + "wind": ({"max": "wind_gust", "avg": "wind_speed_avg"}, "avg"), # its highest is a gust + "rain": ({"max": "rain", "min": "rain", "sum": "rain"}, "sum"), # a day's rain is its total +} HOLIDAYISH = re.compile(r"\b(holidays?|weekends?|saturdays?|sundays?)\b", I) -def count_days_call(text: str, now: datetime, ecowitt: bool, air: bool) -> tuple[str, dict, str] | None: +def _limits(rest: str) -> list[str]: + """The numbers in the text that aren't a year or part of a period: "UV over 9 for 3 months in 2024" -> ["9"].""" + return re.findall(r"(? tuple[str, datetime, datetime] | None: + """(name, start, end) of what a count or a figure per period covers: the period named, a calendar year ("in 2024": up to now, + if it is this one), else everything on record; None when the text names more than one.""" + spans, years = spans_in(rest, now), set(re.findall(r"\b((?:19|20)\d\d)\b", rest)) + if len(spans) + len(years) > 1: + return None + if years: + year = int(years.pop()) + return str(year), datetime(year, 1, 1), min(datetime(year, 12, 31, 23, 59, 59), now) + return spans[0] if spans else ("on record", *span("on record", now)) + + +def _by(group: re.Match) -> str: + """"month" or "year", as GROUPING found it.""" + return "month" if (group.group(1) or "").lower() == "month" or group.group(3) else "year" + + +def count_days_call(text: str, now: datetime, sources: Collection[str]) -> tuple[str, dict, str] | None: """(tool, arguments, label) when the text asks how many days one reading was over or under one limit (optionally per year or month), else None and the model decides.""" if (not re.search(r"\bdays?\b|\btimes\b", text, I) or any(p.search(text) for p in (FORECAST, LINK, THINK, DESCRIBE, HOLIDAYISH)) @@ -575,45 +623,32 @@ def count_days_call(text: str, now: datetime, ecowitt: bool, air: bool) -> tuple op = ">=" if OR_MORE.search(text) or AT_LEAST.search(text) else "<=" if OR_LESS.search(text) else "<" if below else ">" group = GROUPING.search(text) rest = GROUPING.sub(" ", text) - plain = NUMBERED_PERIOD.sub(" ", re.sub(r"\b(?:19|20)\d\d\b", " ", rest)) - numbers = re.findall(r"(?": # a UV index is quoted in whole numbers: "over 10" means the days that read 10 op = ">=" - spans = spans_in(rest, now) - years = set(re.findall(r"\b((?:19|20)\d\d)\b", rest)) - if len(spans) + len(years) > 1: + if not (period := _whole_period(rest, now)): return None - if years: # "in 2024": that calendar year (so far, if it is this one) - year = int(years.pop()) - spans = [(str(year), datetime(year, 1, 1), min(datetime(year, 12, 31, 23, 59, 59), now))] - period = spans[0] if spans else ("on record", *span("on record", now)) - _, start, end = period + name_, start, end = period args = {"where": [{"field": field, "op": op, "value": float(numbers[0])}], "count_only": True, "start_date": start.strftime("%Y-%m-%d"), "end_date": end.strftime("%Y-%m-%d")} if group: - args["group_by"] = "month" if (group.group(1) or "").lower() == "month" or group.group(3) else "year" - return tool, args, f"days count, {period[0]}" + args["group_by"] = _by(group) + return tool, args, f"days count, {name_}" # "rain by month for 2 years", "average temperature per year", "highest PM2.5 by month": one reading, a figure for each month or year, worked @@ -622,62 +657,48 @@ def count_days_call(text: str, now: datetime, ecowitt: bool, air: bool) -> tuple LOWER = re.compile(r"\b(coldest|lowest|min\w*|driest|coolest|calmest|smallest)\b", I) -def period_figures_call(text: str, now: datetime, ecowitt: bool, air: bool) -> tuple[str, dict, str] | None: +def period_figures_call(text: str, now: datetime, sources: Collection[str]) -> tuple[str, dict, str] | None: """(tool, arguments, label) when the text asks for one reading's figure by or per month or year, else None (the model decides).""" group = GROUPING.search(text) if (not group or any(p.search(text) for p in (FORECAST, LINK, THINK, DESCRIBE, HOLIDAYISH, ABOVE, BELOW, OR_MORE, OR_LESS, AT_LEAST)) or re.search(r"\b(and|also|both|but|days|times)\b", text, I)): return None rest = GROUPING.sub(" ", text) - if re.findall(r"(? 1: + fields, plain = DAILY[name] + stat = "max" if high else "min" if low else plain + if stat not in fields: # a lowest of a reading that has none (wind, solar) + return None + field = fields[stat] + if not (period := _whole_period(rest, now)): return None - if years: - year = int(years.pop()) - spans = [(str(year), datetime(year, 1, 1), min(datetime(year, 12, 31, 23, 59, 59), now))] - name_, start, end = spans[0] if spans else ("on record", *span("on record", now)) - by = "month" if (group.group(1) or "").lower() == "month" or group.group(3) else "year" + name_, start, end = period + by = _by(group) return tool, {"stat": stat, "of": field, "count_only": True, "group_by": by, "start_date": start.strftime("%Y-%m-%d"), "end_date": end.strftime("%Y-%m-%d")}, f"figures by {by}, {name_}" -def fast_call(text: str, now: datetime, ecowitt: bool, air: bool, pollen: bool = False, - forecast: bool = False) -> tuple[str, dict, str] | None: - """(tool name, arguments, what it is) for a question the bot can fetch for without the model.""" - if (counted := count_days_call(text, now, ecowitt, air)) or (counted := period_figures_call(text, now, ecowitt, air)): +def fast_call(text: str, now: datetime, sources: Collection[str]) -> tuple[str, dict, str] | None: + """(tool name, arguments, what it is) for a question the bot can fetch for without the model, from these sources (by name).""" + if (counted := count_days_call(text, now, sources)) or (counted := period_figures_call(text, now, sources)): return counted + ecowitt, air, pollen, forecast = ("Ecowitt" in sources, "AirGradient" in sources, "Pollen" in sources, "Forecast" in sources) if SPECIFIC_MOMENT.search(text) or COUNTING.search(text): # "high on 5 Jan this year", "at 3pm today": a whole period would be the wrong data return None if air and (period := air_period(text, now)): @@ -731,11 +752,8 @@ def topics(text: str, before: list[str] = ()) -> set[str]: ("air", mentions_air(said)), ("days", bool(DAYS_WORDS.search(said) or ANALYSIS.search(said) or SPECIFIC_MOMENT.search(said))), ("link", bool(LINK.search(said)) or bool(COMPARE.search(said)) and bool(OUTLOOK_WORDS.search(said))), ("outlook", bool(FORECAST.search(said)) or bool(OUTLOOK_WORDS.search(said))), - ("forecast", bool(FORECAST.search(said)) or bool(OUTLOOK_WORDS.search(said))), ("wind", bool(WIND_WORDS.search(said))), ("describe", bool(DESCRIBE.search(said))), - ("pollen", mentions_pollen(said)), ("bot", bool(BOT_WORDS.search(said)) or about_the_bot(said) or bool(COMMAND.search(said))), - ("compose", mentions_air(said) and bool(COMPARE.search(said) or GRAPH.search(said) or LINK.search(said) or ANALYSIS.search(said))), ) if hit} return found @@ -746,16 +764,17 @@ def topics(text: str, before: list[str] = ()) -> set[str]: WEEK_AHEAD = re.compile(r"\b(week|days|weekend|next \w+day)\b", I) -def model_prefetch(text: str, now: datetime, ecowitt: bool, forecast: bool) -> list[tuple[str, dict]]: +def model_prefetch(text: str, now: datetime, sources: Collection[str]) -> list[tuple[str, dict]]: """What the model would fetch first for these questions, fetched here instead (so it answers in one call, not two): for rain ahead, the current reading, the last 3 hours and the forecast (cached); for a described today or yesterday, that day's readings.""" calls: list[tuple[str, dict]] = [] + ecowitt = "Ecowitt" in sources if RAIN_AHEAD.search(text): if ecowitt: groups = "outdoor,pressure,rainfall,rainfall_piezo,wind" calls += [("weather_now", {"groups": groups}), ("weather_history", {"groups": groups, "start_date": (now - timedelta(hours=3)).strftime(FMT), "end_date": now.strftime(FMT)})] - if forecast: + if "Forecast" in sources: calls.append(("weather_forecast", {"days": 7 if WEEK_AHEAD.search(text) else 3, "cached": True})) elif ecowitt and DESCRIBE.search(text) and not SPECIFIC_MOMENT.search(text): spans = spans_in(text, now) @@ -767,7 +786,7 @@ def model_prefetch(text: str, now: datetime, ecowitt: bool, forecast: bool) -> l @dataclass -class Reading: +class Intent: """Everything decided about one message before the model sees it.""" effort: str = EFFORT_DEFAULT needs_data: bool = False # the model must call a tool first @@ -783,14 +802,19 @@ class Reading: weather_now: bool = False # "weather now": every reading the station has, in the report's layout rain_caption: bool = False # "rain chart 7d": the caption is just the least and most rain chart_in_code: bool = False # a chart asked for plainly: fetched and captioned in code, no model - chart_all: bool = False # "weather all week": every reading charted, sent with no caption readings: list[str] = field(default_factory=list) # the readings the words name per_day: bool = False # figures day by day were asked for extra: list[tuple[str, dict]] = field(default_factory=list) # fetched with the model's first step (the forecast, for "will it rain?") - lookup: str = "" # "reading", "air", "pollen", "forecast" or "about": answered in code, no model + lookup: str = "" # "reading", "air", "days", "extremes", "pollen", "forecast" or "about": answered in code, no model lookup_arg: list[str] = field(default_factory=list) # the readings named ("reading", "air"; none: all) sides: list[str] = field(default_factory=list) # "indoor", "outdoor" or both, for a reading + @property + def tool_choice(self) -> str: + """What the model's first step may do with the tools (see Agent.run): none about the bot itself, a fetch it must make + when the answer needs data, else its own choice.""" + return "none" if self.about_the_bot else "required" if self.needs_data else "auto" + def plain_lookup(text: str, fast: tuple | None) -> tuple[str, list[str]]: """(kind, readings named) for a question answered in code from one tool result, else ("", []).""" @@ -805,10 +829,10 @@ def plain_lookup(text: str, fast: tuple | None) -> tuple[str, list[str]]: return "days", [] if tool == "weather_forecast": return "forecast", [] - if any(p.search(text) for p in JUDGEMENT): + if _judgement(text): return "", [] - if tool == "weather_history" and not args.get("chart") and not any(p.search(text) for p in JUDGEMENT): - return "extremes", [n for n in _named(text)] + if tool == "weather_history" and not args.get("chart"): + return "extremes", list(_named(text)) if tool == "weather_now" and (names := reading_now(text)): return "reading", names if tool == "air_quality" and not args and len(text.split()) <= 6: @@ -816,21 +840,28 @@ def plain_lookup(text: str, fast: tuple | None) -> tuple[str, list[str]]: return "", [] -def read(text: str, now: datetime, ecowitt: bool = True, air: bool = True, pollen: bool = False, - forecast: bool = False) -> Reading: - """The decisions made in code for this message. A shortcut that fails is dropped: the model handles the question.""" +APOSTROPHES = str.maketrans("’‘", "''") # a phone's typographic apostrophes ("what’s"), read as the plain one every pattern uses + + +def read(text: str, now: datetime, sources: Collection[str] = ("Ecowitt", "AirGradient")) -> Intent: + """The decisions made in code for this message, given the sources the bot has (by name: "Ecowitt", "AirGradient", "Pollen", + "Forecast"). A shortcut that fails is dropped: the model handles the question.""" + text = text.translate(APOSTROPHES) + ecowitt = "Ecowitt" in sources report = wants_report(text) - calls = report_calls(ecowitt, air, pollen, forecast) if report else [] + calls = report_calls(sources) if report else [] try: - fast = (*calls[0], "report") if calls else fast_call(text, now, ecowitt, air, pollen, forecast) + fast = (*calls[0], "report") if calls else fast_call(text, now, sources) except Exception: log.exception("Fast path failed; using the normal path") fast = None in_code = bool(fast and fast[0] in ("weather_history", "air_quality") and fast[1].get("chart") - and not any(p.search(text) for p in JUDGEMENT)) + and not _judgement(text)) rain_caption = ecowitt and wants_rain_caption(text) lookup, named = plain_lookup(text, fast) if not (report or in_code) else ("", []) - return Reading(reasoning_effort(text), needs_data(text), about_the_bot(text), report, period_hints(text, now), - fast, bool(GRAPH.search(text)), chart_field(text), chart_fields(text), bool(AVERAGE.search(text)), - more=calls[1:], weather_now=ecowitt and wants_weather_now(text), rain_caption=rain_caption, chart_in_code=in_code, chart_all=in_code and wants_chart_all(text), - lookup=lookup, lookup_arg=named, sides=_sides(text), extra=model_prefetch(text, now, ecowitt, forecast) if not fast else [], readings=list(_named(text)), per_day=bool(PER_DAY.search(text))) + return Intent(effort=reasoning_effort(text), needs_data=needs_data(text), about_the_bot=about_the_bot(text), report=report, + hints=period_hints(text, now), fast=fast, chart_asked=bool(GRAPH.search(text)), chart_field=chart_field(text), + chart_fields=chart_fields(text), average_asked=bool(AVERAGE.search(text)), more=calls[1:], + weather_now=ecowitt and wants_weather_now(text), rain_caption=rain_caption, chart_in_code=in_code, + readings=list(_named(text)), per_day=bool(PER_DAY.search(text)), + extra=model_prefetch(text, now, sources) if not fast else [], lookup=lookup, lookup_arg=named, sides=_sides(text)) diff --git a/lib/irrigation/menu.py b/lib/irrigation/menu.py index ede55c4..91fbc6b 100644 --- a/lib/irrigation/menu.py +++ b/lib/irrigation/menu.py @@ -3,7 +3,7 @@ Collapsed: [💧 Water] [⏸ Pause] / [🔔 Alerts] [🔄 Refresh]. While the valve is on, a [Water off ❌] row follows. While the timer is paused, Pause is [▶️ Unpause] (one press ends the pause) and Water is hidden. A press on Water, Pause or Alerts opens its options under the rows (▸ after the text marks a menu, ▾ the open one; pressing it again closes them). Water lists 5, 10, 20 and 30 minutes (a run is always timed). Alerts has -one toggle per alert type, green while this chat gets it, struck through and uncoloured while it does not; pressing it flips it. +one toggle per alert type, blue while this chat gets it, struck through and uncoloured while it does not; pressing it flips it. Callback data (short, self-contained, so an old message's buttons still work after a restart): ir:open: | ir:close | ir:refresh (read the state again, change nothing) @@ -17,11 +17,13 @@ from telegram import InlineKeyboardMarkup, MessageEntity from ..buttons import button, opener, toggle_button +from ..entities import footer +from .tuya import DELAYS, is_paused TITLE = "🌱 Irrigation" WATER_MINUTES = (5, 10, 20, 30) -DELAYS = ("24h", "48h", "72h", "cancel") SECTIONS = ("water", "delay", "alerts") +ALERTS = {"batt": ("irrigation_battery", "battery"), "pause": ("irrigation_pause", "pause")} # an Alerts toggle's word -> (its alert type, its name) LOW_BATTERY = 10 # below this the battery shows as 🪫 PAUSED_ERROR = "error: can not turn on water while paused!" KEYWORDS = re.compile(r"\b(irrigation|irrigate|sprinklers?|taps?|water(?:ing)?)\b", re.IGNORECASE) @@ -33,17 +35,11 @@ def asked(text: str) -> bool: return 0 < len(text.split()) <= MAX_WORDS and bool(KEYWORDS.search(text)) -def is_paused(status: dict) -> bool: - """Is the timer paused (the controller has a weather delay other than "cancel")?""" - return status.get("weather_delay") not in (None, "cancel") - - def paused_in(text: str | None) -> bool: - """Does a status message (as status_text wrote it) say the timer is paused? For redrawing only the buttons, without asking the controller. - Messages sent before the line was reworded ("pause time: inactive") still read right.""" - line = next((l for l in (text or "").splitlines() if l.startswith(("pause:", "pause time:"))), "") - value = line.partition(":")[2].strip() - return bool(value) and not value.startswith(("not paused", "inactive")) + """Does a status message (as status_text wrote it) say the timer is paused? For redrawing only the buttons, without asking the controller.""" + line = next((l for l in (text or "").splitlines() if l.startswith("pause:")), "") + value = line.removeprefix("pause:").strip() + return bool(value) and not value.startswith("not paused") def running_in(text: str | None) -> bool: @@ -52,11 +48,8 @@ def running_in(text: str | None) -> bool: def with_error(text: str, error: str = PAUSED_ERROR) -> tuple[str, list[MessageEntity]]: - """The status with an error under it as a bold footer. Plain text plus entities (no markup to escape); Telegram counts - positions in UTF-16 code units, where an emoji is two.""" - units = lambda s: len(s.encode("utf-16-le")) // 2 - head = f"{text}\n\n" - return head + error, [MessageEntity(MessageEntity.BOLD, units(head), units(error))] + """The status with an error under it, after a blank line, in bold.""" + return footer(f"{text}\n", error, MessageEntity.BOLD) def status_text(status: dict, alerts: bool | None = None, pause_alerts: bool | None = None) -> str: @@ -79,25 +72,27 @@ def status_text(status: dict, alerts: bool | None = None, pause_alerts: bool | N return "\n".join([TITLE, f"mode: {status.get('work_state') or 'unknown'}", f"state: {'on ✅' if on else 'off ❌'}{left}", - f"pause: {'not paused' if delay in (None, 'cancel') else delay}{bracket(pause_alerts)}", + f"pause: {delay if is_paused(status) else 'not paused'}{bracket(pause_alerts)}", f"battery: {level}{bracket(alerts)}"]) def keyboard(battery: bool, pause: bool, open: str | None = None, paused: bool = False, running: bool = False) -> InlineKeyboardMarkup: """The buttons under the status. `battery` and `pause`: this chat gets the irrigation battery / pause alerts; each Alerts button - is a toggle (green when on, struck through and uncoloured when off). `paused`: the timer is paused, so its - button unpauses and Water is hidden. `running`: the valve is on, so Water off is a top-level row.""" + is a toggle (blue when on, struck through and uncoloured when off). `paused`: the timer is paused, so its + button unpauses and Water is hidden. `running`: the valve is on, so Water off is always shown. While a menu is open only its own button (the heading) and its list are shown.""" head = lambda name, text: opener(text, open == name, f"ir:open:{name}", "ir:close") delay = button("▶️ Unpause", "ir:delay:cancel") if paused else head("delay", "⏸ Pause") - rows = [[delay] if paused else [head("water", "💧 Water"), delay], - [head("alerts", "🔔 Alerts"), button("🔄 Refresh", "ir:refresh")]] - if running: # the valve is open: off is one press away, whatever else is open - rows.append([button("Water off ❌", "ir:sw:off")]) if open == "water" and not paused: - rows.append([button(f"{m} min", f"ir:water:{m}") for m in WATER_MINUTES]) + rows = [[head("water", "💧 Water")], [button(f"{m} min", f"ir:water:{m}") for m in WATER_MINUTES]] elif open == "delay" and not paused: - rows.append([button(d, f"ir:delay:{d}") for d in DELAYS if d != "cancel"]) + rows = [[delay], [button(d, f"ir:delay:{d}") for d in DELAYS if d != "cancel"]] elif open == "alerts": - for on, kind, name in ((battery, "batt", "battery"), (pause, "pause", "pause")): - rows.append([toggle_button(f"{name} alerts", on, lambda turn_on, kind=kind: f"ir:{kind}:{'on' if turn_on else 'off'}")]) + rows = [[head("alerts", "🔔 Alerts")]] + for on, (word, (_, name)) in zip((battery, pause), ALERTS.items()): + rows.append([toggle_button(f"{name} alerts", on, lambda turn_on, word=word: f"ir:{word}:{'on' if turn_on else 'off'}")]) + else: + rows = [[delay] if paused else [head("water", "💧 Water"), delay], + [head("alerts", "🔔 Alerts"), button("🔄 Refresh", "ir:refresh")]] + if running: # the valve is open: off is one press away, whatever else is open + rows.insert(1 if rows[0][0].text.endswith(("▾",)) else len(rows), [button("Water off ❌", "ir:sw:off")]) return InlineKeyboardMarkup(rows) diff --git a/lib/irrigation/tuya.py b/lib/irrigation/tuya.py index 37f9fc9..92aae2c 100644 --- a/lib/irrigation/tuya.py +++ b/lib/irrigation/tuya.py @@ -25,6 +25,11 @@ MAX_MINUTES = 60 # the longest run asked for from a button +def is_paused(status: dict) -> bool: + """Is the timer paused: a weather delay other than "cancel" (which is none)?""" + return status.get("weather_delay") not in (None, "cancel") + + class IrrigationError(Exception): """The controller could not be read or told something. The message is safe to log (no secrets, no raw response).""" @@ -46,7 +51,7 @@ def __init__(self, cfg: Config, transport=None): async def close(self): await self.http.aclose() - async def _send(self, method: str, path: str, payload: dict | None, token: str): + async def _send(self, method: str, path: str, payload: dict | None, token: str) -> dict: body = json.dumps(payload, separators=(",", ":"), ensure_ascii=False).encode() if payload is not None else b"" timestamp, nonce = str(int(time.time() * 1000)), uuid.uuid4().hex headers = {"client_id": self.client_id, "sign_method": "HMAC-SHA256", "t": timestamp, "nonce": nonce, @@ -58,17 +63,23 @@ async def _send(self, method: str, path: str, payload: dict | None, token: str): try: response = await self.http.request(method, path, content=body if payload is not None else None, headers=headers) response.raise_for_status() - return response.json() + data = response.json() except (httpx.HTTPError, ValueError) as e: # the URL and body stay out of the message raise IrrigationError(f"Tuya request failed ({type(e).__name__})") from e + if not isinstance(data, dict): + raise IrrigationError(f"Tuya sent a {type(data).__name__}, not a reply") + return data async def _authenticate(self): data = await self._send("GET", TOKEN_PATH, None, "") if not data.get("success"): raise IrrigationError(f"Tuya refused the credentials (code {data.get('code')}: {data.get('msg')})") - result = data["result"] + result = data.get("result") + if not isinstance(result, dict) or not isinstance(result.get("access_token"), str): + raise IrrigationError("Tuya's token reply has no access token") + lasts = result.get("expire_time") self.token = result["access_token"] - self.token_expires = time.time() + max(60, int(result.get("expire_time", 7200)) - 60) + self.token_expires = time.time() + max(60, (lasts if isinstance(lasts, int) else 7200) - 60) async def request(self, method: str, path: str, payload: dict | None = None, retry: bool = True): """The `result` of a signed request. An expired token is replaced and the request sent once more.""" @@ -85,7 +96,9 @@ async def request(self, method: str, path: str, payload: dict | None = None, ret async def status(self) -> dict: """{code: value} of the device's status points (switch, battery_percentage, weather_delay, countdown, work_state, ...).""" result = await self.request("GET", f"/v1.0/devices/{self.device}/status") - return {item["code"]: item["value"] for item in result or [] if "code" in item} + if not isinstance(result, list | None): + raise IrrigationError(f"Tuya's status is a {type(result).__name__}, not a list") + return {item["code"]: item.get("value") for item in result or [] if isinstance(item, dict) and "code" in item} async def command(self, *commands: tuple[str, object]): """Send one or more (code, value) commands together.""" diff --git a/lib/lines.py b/lib/lines.py index 76875cf..6ce02d2 100644 --- a/lib/lines.py +++ b/lib/lines.py @@ -1,24 +1,43 @@ """One rule for turning readings into the line a chart draws, whichever device they came from. -A reading is (epoch, value, low, high, seconds): the value, the lowest and highest reading it stands for (None when the -source gives no range) and how many seconds of time it covers (300 for a 5-minute reading, 1800 for a 30-minute one, 86400 -for one of Ecowitt's daily records). `build_line` picks how to draw a whole period of them: +A reading is a Sample (epoch, value, low, high, seconds): the value, the lowest and highest reading it stands for (None when +the source gives no range) and how many seconds of time it covers (300 for a 5-minute reading, 1800 for a 30-minute one, +86400 for one of Ecowitt's daily records). `build_line` picks how to draw a whole period of them: - up to about 500 points, the readings themselves (a slow 5-minute reading is lightly smoothed); - more than that, buckets of 30 minutes, an hour, 2 hours, 4 hours, and past that one point a day (the bucket's mean). Every line keeps its range as a band at every width: the lowest and highest reading in each bucket, or a reading's own range when the source gives one (Ecowitt's 30-minute lows and highs; wind: the average speed, shaded up to the gusts). The readings themselves (up to about 500 points) have a band only where the source gave them a range.""" -from dataclasses import dataclass -from datetime import date, datetime, time +import math +from bisect import bisect_left, bisect_right +from dataclasses import dataclass, replace +from datetime import date, tzinfo +from itertools import accumulate +from statistics import fmean +from typing import NamedTuple from .specs import Line -from .timeutil import SLOT, WIDTH_NAMES, bucket_width, bucketed, daily_summary, local_date, local_epoch, rolling_mean +from .timeutil import SLOT, at_hour, local_date DAY = 86400 SMOOTH_POINTS = 3 # a smoothed 5-minute line: each point is the mean of this many readings (15 minutes) -type Reading = tuple[int, float, float | None, float | None, int] SHORT_DAYS = 31 # up to this long a weather line is drawn with no shaded range (see Plotted.unbanded) +POINT_BUDGET = 500 # about as many points as a chart can show legibly +WIDTHS = (300, SLOT, 3600, 7200, 14400, DAY) # bucket sizes a chart line can be drawn at: 5 min ... a day +WIDTH_NAMES = {300: "5-minute", 1800: "30-minute", 3600: "hourly", 7200: "2-hour", 14400: "4-hour", 86400: "daily"} + + +class Sample(NamedTuple): + t: int # epoch seconds + value: float + low: float | None # the lowest and highest reading it stands for, None when the source gives no range + high: float | None + seconds: int # the time it covers + + def bounds(self) -> tuple[float, float]: + """(low, high), the value itself for a side the source gave no range.""" + return (self.value if self.low is None else self.low), (self.value if self.high is None else self.high) @dataclass @@ -33,10 +52,8 @@ class Plotted: def unbanded(self, days: float, keep: bool = False) -> "Plotted": """The line without its shaded range when the period is SHORT_DAYS or less (a short period reads best as a clean line); `keep` - leaves it (wind: the mean shaded up to the gusts).""" - if not keep and days <= SHORT_DAYS: - self.low = self.high = None - return self + leaves it (wind: the mean shaded up to the gusts). A new line: this one is left as it is.""" + return self if keep or days > SHORT_DAYS else replace(self, low=None, high=None) @property def name(self) -> str: @@ -50,35 +67,89 @@ def spec(self, label: str, records: dict | None = None, reading: str = "", indoo return Line(label, self.x, self.y, self.low, self.high, self.smoothed, records or {}, reading, indoor) -def slot_readings(values: dict[int, float], lows: dict[int, float] | None = None, highs: dict[int, float] | None = None, - seconds: int = SLOT) -> list[Reading]: - """Readings from {epoch: value} maps, all covering `seconds`; the range is what the maps hold, else None.""" - lows, highs = lows or {}, highs or {} - return [(t, v, lows.get(t), None if t not in highs else max(highs[t], v), seconds) for t, v in values.items()] +def bucket_width(span_seconds: float, source_seconds: float, budget: int = POINT_BUDGET) -> int: + """The finest bucket (never finer than the readings themselves) that keeps a line of this length within the + point budget; a day is the widest.""" + return next((w for w in WIDTHS if w >= source_seconds and span_seconds / w <= budget), WIDTHS[-1]) + + +def _summarise(points, key) -> dict: + """{bucket: (mean, lowest low, highest high)} from (ts, value, low, high, fine) readings. The mean takes one value + per 30-minute slot, so a stretch held as 5-minute readings (fine) counts no more than the same stretch held as + 30-minute ones: a slot's fine readings win where it has them.""" + slots: dict = {} + extremes: dict = {} + for ts, value, low, high, fine in points: + k = key(ts) + slots.setdefault((k, ts // SLOT), {}).setdefault(fine, []).append(value) + lo, hi = extremes.get(k, (low, high)) + extremes[k] = (min(lo, low), max(hi, high)) + means: dict = {} + for (k, _), got in slots.items(): + values = got.get(True) or got[False] + means.setdefault(k, []).append(fmean(values)) + return {k: (fmean(v), *extremes[k]) for k, v in means.items()} + + +def daily_summary(points, tz: tzinfo) -> dict[date, tuple[float, float, float]]: + """{local day: (mean, lowest low, highest high)}; see _summarise.""" + return _summarise(points, lambda ts: local_date(ts, tz)) -def _own_range(r: Reading) -> tuple[float, float]: - return (r[1] if r[2] is None else r[2]), (r[1] if r[3] is None else r[3]) +def bucket_summary(points, width: int) -> dict[int, tuple[float, float, float]]: + """{bucket start epoch: (mean, lowest low, highest high)} for buckets of `width` seconds; see _summarise.""" + return _summarise(points, lambda ts: ts // width * width) + + +def bucketed(points, width: int) -> tuple[list[int], list[float], list[float], list[float]]: + """(x, mean, low, high) lists for a chart line in epoch-aligned buckets of this width (under a day), each drawn at its + middle. A line of days is _daily's.""" + buckets = bucket_summary(points, width) + keys = sorted(buckets) + rows = [buckets[k] for k in keys] + return [k + width // 2 for k in keys], [r[0] for r in rows], [r[1] for r in rows], [r[2] for r in rows] + + +def rolling_mean(values: dict[int, float], half_window: int) -> dict[int, float]: + """Each reading replaced by the mean of the readings within half_window seconds either side of it (itself included): + a light smoothing of the drawn line. Timestamps are unchanged, nothing is interpolated, and a gap only means fewer + readings in the window. A reading that is not a finite number is left out.""" + ts = sorted(t for t, v in values.items() if math.isfinite(v)) + prefix = [0.0, *accumulate(values[t] for t in ts)] + out: dict[int, float] = {} + for t in ts: + lo, hi = bisect_left(ts, t - half_window), bisect_right(ts, t + half_window) + out[t] = (prefix[hi] - prefix[lo]) / (hi - lo) + return out + + +def slot_readings(values: dict[int, float], lows: dict[int, float] | None = None, highs: dict[int, float] | None = None, + seconds: int = SLOT) -> list[Sample]: + """Readings from {epoch: value} maps, all covering `seconds`; the range is what the maps hold (taken out to the value where it + falls short of it), else None.""" + lows, highs = lows or {}, highs or {} + return [Sample(t, v, None if t not in lows else min(lows[t], v), None if t not in highs else max(highs[t], v), seconds) + for t, v in values.items()] -def build_line(readings: list[Reading], tz, span_seconds: float, *, smooth: bool = False, +def build_line(readings: list[Sample], tz, span_seconds: float, *, smooth: bool = False, force_daily: bool = False, until: date | None = None) -> Plotted | None: - """The line for these readings over a period of span_seconds; None when there is too little to draw. - force_daily: one point a day whatever the length (an average was asked for). until: with daily records in the mix, - days from this one on are not averaged from sub-daily readings (the day is not over).""" - rs = sorted(readings) - fine = [r for r in rs if r[4] < DAY] - records = [r for r in rs if r[4] >= DAY] + """The line for these readings (Samples, or tuples in their order) over a period of span_seconds; None when there is too + little to draw. force_daily: one point a day whatever the length (an average was asked for). until: with daily records in + the mix, days from this one on are not averaged from sub-daily readings (the day is not over).""" + rs = sorted(map(Sample._make, readings)) + fine = [r for r in rs if r.seconds < DAY] + records = [r for r in rs if r.seconds >= DAY] if not fine: # only daily records: as they are, each with its own range return _as_they_are(records) if len(records) >= 2 else None - source = min(r[4] for r in fine) + source = min(r.seconds for r in fine) width = DAY if records or force_daily else bucket_width(span_seconds, source) if width >= DAY: return _daily(fine, records, tz, until) - if width > source or len({r[4] for r in fine}) > 1: # too many for the chart, or two resolutions side by side - xs, mean, low, high = bucketed([(r[0], r[1], *_own_range(r), r[4] < SLOT) for r in fine], tz, max(width, SLOT)) + if width > source or len({r.seconds for r in fine}) > 1: # too many for the chart, or two resolutions side by side + xs, mean, low, high = bucketed([(r.t, r.value, *r.bounds(), r.seconds < SLOT) for r in fine], max(width, SLOT)) return Plotted(xs, mean, max(width, SLOT), False, low, high) if len(xs) >= 2 else None - line = _as_they_are([r for r in fine if r[4] == source]) + line = _as_they_are([r for r in fine if r.seconds == source]) if line and smooth and source == 300: # slow readings come in 0.1-degree steps: a light average of the real readings smoothed = rolling_mean(dict(zip(line.x, line.y)), SMOOTH_POINTS * source // 2) if len(smoothed) == len(line.x): # every reading was a finite number @@ -87,25 +158,25 @@ def build_line(readings: list[Reading], tz, span_seconds: float, *, smooth: bool return line -def _as_they_are(rs: list[Reading]) -> Plotted | None: +def _as_they_are(rs: list[Sample]) -> Plotted | None: if len(rs) < 2: return None - ranges = [_own_range(r) for r in rs] - ranged = sum(1 for r in rs if r[2] is not None or r[3] is not None) >= len(rs) // 2 - return Plotted([r[0] for r in rs], [r[1] for r in rs], rs[0][4], True, + ranges = [r.bounds() for r in rs] + ranged = sum(1 for r in rs if r.low is not None or r.high is not None) >= len(rs) // 2 + return Plotted([r.t for r in rs], [r.value for r in rs], rs[0].seconds, True, [lo for lo, _ in ranges] if ranged else None, [hi for _, hi in ranges] if ranged else None) -def _daily(fine: list[Reading], records: list[Reading], tz, until: date | None) -> Plotted | None: +def _daily(fine: list[Sample], records: list[Sample], tz, until: date | None) -> Plotted | None: """One point a day. Days held as sub-daily readings get their own mean, lowest and highest over the local day; the older days come from the daily records, each with its own range.""" - keep = [r for r in fine if until is None or local_date(r[0], tz) < until] - days = daily_summary(((r[0], r[1], *_own_range(r), r[4] < SLOT) for r in keep), tz) + keep = [r for r in fine if until is None or local_date(r.t, tz) < until] + days = daily_summary(((r.t, r.value, *r.bounds(), r.seconds < SLOT) for r in keep), tz) hour = 10 if records else 12 # Ecowitt's daily records are 10am-10am windows: draw the exact days where they sit - points = {local_epoch(datetime.combine(d, time(hour)), tz): v for d, v in days.items()} + points = {at_hour(d, hour, tz): v for d, v in days.items()} for r in records: - if local_date(r[0], tz) not in days: - points[r[0]] = (r[1], *_own_range(r)) if r[2] is not None and r[3] is not None else (r[1], None, None) + if local_date(r.t, tz) not in days: + points[r.t] = (r.value, *r.bounds()) if r.low is not None and r.high is not None else (r.value, None, None) xs = sorted(points) if len(xs) < 2: return None diff --git a/lib/llm.py b/lib/llm.py index 754560b..c3aa573 100644 --- a/lib/llm.py +++ b/lib/llm.py @@ -7,13 +7,18 @@ from openai import AsyncOpenAI -from .timeutil import CHARS_PER_TOKEN, WARN_TOKENS +from .lines import POINT_BUDGET from .tools import Tools, Turn log = logging.getLogger(__name__) MAX_STEPS = 10 MAX_HISTORY = 16 # messages kept per chat (a follow-up needs the last few, every call resends them) +TOKENS_PER_POINT = 10 # a dated low/high row is about 10 tokens as the model reads it +CHARS_PER_TOKEN = 4 +# Tool results past twice a full query's data (10,000 tokens) get the chat a heads-up: about $0.0125 of input at Grok 4.3's +# $1.25 per million, and it rides on every later step +WARN_TOKENS = 2 * POINT_BUDGET * TOKENS_PER_POINT LIMIT_NOTICE = ("(System: tool limit reached. Answer now using only the data already fetched, " "and briefly say if anything is missing.)") @@ -67,29 +72,31 @@ async def _stream(self, kwargs: dict, on_text) -> tuple[str, list[dict], object] on_text(content) return content, [calls[i] for i in sorted(calls)], usage - async def run(self, messages: list[dict], system_prompt: str, effort: str, - first_call: tuple[str, dict] | list[tuple[str, dict]] | None = None, require_tool: bool = True, no_tools: bool = False, - turn: Turn | None = None, on_text=None, tool_names: list[str] | None = None, on_heavy=None, conv_id: str | None = None) -> str: + async def run(self, messages: list[dict], system_prompt: str, effort: str, *, + first_call: tuple[str, dict] | list[tuple[str, dict]] | None = None, tool_choice: str = "required", + turn: Turn | None = None, on_text=None, on_heavy=None, conv_id: str | None = None) -> str: """Runs the tool loop, appending assistant/tool turns to `messages` in place. The prompt is passed per question (not stored) so concurrent chats can't clash. first_call (tool name, args), or a list of them, is what the bot already worked out (the fast path): the calls run - straight away, together, and the model is only invoked once the data is in. require_tool forces a - fresh fetch on the first model call (weather questions); off for chat, so it can just reply. - on_text(text so far) is called as each answer streams in (private chats show it as a draft). conv_id names the conversation for the provider's prompt cache (see above); on_heavy(tokens) is called once, before the model is - sent a lot of data: when the tool results of a question pass WARN_TOKENS (twice a 500-point query), so the chat can be told + straight away, together, and the model is only invoked once the data is in. tool_choice is what the model's first + step may do with the tools: "required" makes it fetch fresh data first (weather questions), "auto" lets it just + reply (chat), "none" gives it no tools at all (a question about the bot itself). Every later step is "auto". + on_text(text so far) is called as each answer streams in (private chats show it as a draft). conv_id names the + conversation for the provider's prompt cache; on_heavy(tokens) is called once, before the model is sent a lot of + data: when the tool results of a question pass WARN_TOKENS (twice a 500-point query), so the chat can be told first. The work carries on; nothing waits for a reply.""" cache: dict = {} # identical tool calls within one question are only made once async def call(name: str, args: str) -> str: - if (name, args) not in cache: - cache[(name, args)] = asyncio.ensure_future(self.tools.call(name, args, turn)) - return await cache[(name, args)] + key = (name, _same(args)) + if key not in cache: + cache[key] = asyncio.ensure_future(self.tools.call(name, args, turn)) + return await cache[key] log.info("Tool call %s served from cache (repeat)", name) return ("[You already made this exact call - same result as before. Don't repeat it; " - "change the request or answer now.]\n" + await cache[(name, args)]) + "change the request or answer now.]\n" + await cache[key]) - schemas = self.tools.schemas_for(tool_names) # only the tools the question can use: each definition is sent on every call warned = False async def heavy_check(): @@ -102,18 +109,23 @@ async def heavy_check(): await on_heavy(tokens) llm_time = tool_time = 0.0 + + async def run_calls(calls: list[dict]): + """The calls ({"id", "name", "arguments"}), made together; their results join the messages.""" + nonlocal tool_time + t0 = time.monotonic() + results = await asyncio.gather(*(call(c["name"], c["arguments"]) for c in calls)) + tool_time += time.monotonic() - t0 + messages.extend({"role": "tool", "tool_call_id": c["id"], "content": r} for c, r in zip(calls, results)) + await heavy_check() + first_step = 0 try: if first_call: - fast = [(n, json.dumps(a)) for n, a in (first_call if isinstance(first_call, list) else [first_call])] - messages.append({"role": "assistant", "content": "", "tool_calls": [ - {"id": f"fast_{i}", "type": "function", "function": {"name": n, "arguments": a}} - for i, (n, a) in enumerate(fast, 1)]}) - t0 = time.monotonic() - results = await asyncio.gather(*(call(n, a) for n, a in fast)) - tool_time += time.monotonic() - t0 - messages.extend({"role": "tool", "tool_call_id": f"fast_{i}", "content": r} for i, r in enumerate(results, 1)) - await heavy_check() + fast = [{"id": f"fast_{i}", "name": n, "arguments": json.dumps(a)} + for i, (n, a) in enumerate(first_call if isinstance(first_call, list) else [first_call], 1)] + messages.append({"role": "assistant", "content": "", "tool_calls": [_tool_call(c) for c in fast]}) + await run_calls(fast) first_step = 1 # data is in; the model just answers (and may still call tools) for step in range(first_step, MAX_STEPS + 1): final = step == MAX_STEPS # out of steps: force an answer from what we have @@ -122,10 +134,9 @@ async def heavy_check(): "extra_body": {"reasoning_effort": effort}} # Grok 4.3: none / low / medium / high if conv_id: # the same id for every call of a chat sends them to the server holding its cached prompt kwargs["extra_headers"] = {"x-grok-conv-id": conv_id} - if schemas: - kwargs["tools"] = schemas - kwargs["tool_choice"] = ("none" if final or no_tools else - "required" if step == 0 and require_tool else "auto") + if self.tools.schemas: # every tool, every time: the same list keeps the provider's cached prompt valid + kwargs["tools"] = self.tools.schemas + kwargs["tool_choice"] = "none" if final or tool_choice == "none" else tool_choice if step == 0 else "auto" if final: log.warning("Tool step limit (%d) reached - asking for a final answer", MAX_STEPS) kwargs["messages"].append({"role": "user", "content": LIMIT_NOTICE}) @@ -134,23 +145,28 @@ async def heavy_check(): llm_time += secs entry = {"role": "assistant", "content": content} if tool_calls and not final: - entry["tool_calls"] = [{"id": tc["id"], "type": "function", - "function": {"name": tc["name"], "arguments": tc["arguments"]}} - for tc in tool_calls] + entry["tool_calls"] = [_tool_call(tc) for tc in tool_calls] messages.append(entry) if not tool_calls or final: return content.strip() or "Sorry, I couldn't get an answer together - try a narrower question." - - t0 = time.monotonic() - results = await asyncio.gather(*(call(tc["name"], tc["arguments"]) for tc in tool_calls)) - tool_time += time.monotonic() - t0 - for tc, result in zip(tool_calls, results): - messages.append({"role": "tool", "tool_call_id": tc["id"], "content": result}) - await heavy_check() + await run_calls(tool_calls) finally: log.info("Timing: LLM %.1fs, tools %.1fs", llm_time, tool_time) +def _tool_call(c: dict) -> dict: + """A call ({"id", "name", "arguments"}) as an assistant message carries it.""" + return {"id": c["id"], "type": "function", "function": {"name": c["name"], "arguments": c["arguments"]}} + + +def _same(args: str) -> str: + """A call's arguments in one form whatever their order and spacing, so a repeat is known; text that isn't JSON as it is.""" + try: + return json.dumps(json.loads(args), sort_keys=True) + except ValueError: + return args + + def strip_tool_turns(messages: list[dict]) -> list[dict]: """Keep only the questions and final answers, dropping tool calls and raw tool results, so later questions can't reuse old data (and requests stay small).""" diff --git a/lib/logs.py b/lib/logs.py index 3144521..30f2d99 100644 --- a/lib/logs.py +++ b/lib/logs.py @@ -1,6 +1,6 @@ -"""How the bot's log lines look. Every line of an INFO or DEBUG entry is cut to the log width (LOG_WIDTH), an ellipsis (…) marking the -cut, whatever its level. A line that quotes a request or a chat message (the quote last on the line) also has the quote -put on one line first.""" +"""How the bot's log lines look. Every line of a message is cut to the log width (LOG_WIDTH), an ellipsis (…) marking the cut, +whatever its level; a traceback after it is logged whole. A line that quotes a request or a chat message (the quote last on the +line) also has the quote put on one line first.""" import logging @@ -13,7 +13,8 @@ def fit(text: str, room: int) -> str: class LogFormat(logging.Formatter): - """"INFO message": the level (WARNING is shown as WARN) and the message, with no module name. Cuts each line to the width.""" + """"INFO message": the level (WARNING is shown as WARN) and the message, with no module name. Cuts each line of the message to + the width; a traceback (or stack) follows whole.""" def __init__(self, fmt: str, width: int = 0): """width: the most characters of a line of the log; 0 cuts none.""" @@ -26,12 +27,17 @@ def format(self, record): if isinstance(text, str): # the HTTP client's own lines: "POST https://... "HTTP/1.1 200 OK"", not "HTTP Request: POST ..." record.msg = text.removeprefix("HTTP Request: ") try: - line = super().format(record) + record.message = record.getMessage() + if self.usesTime(): + record.asctime = self.formatTime(record, self.datefmt) + line = self.formatMessage(record) finally: record.levelname, record.msg = shown, text quote = str(record.args[-1]) if getattr(record, "quotes", False) and isinstance(record.args, tuple) and record.args else None - if quote and line.endswith(quote): # (not when a traceback follows it) + if quote and line.endswith(quote): line = line[:len(line) - len(quote)] + " ".join(quote.split()) # the quote on one line - if not self.width: - return line - return "\n".join(fit(part, self.width) for part in line.split("\n")) + if self.width: + line = "\n".join(fit(part, self.width) for part in line.split("\n")) + if record.exc_info and not record.exc_text: + record.exc_text = self.formatException(record.exc_info) + return "\n".join(filter(None, [line, record.exc_text, record.stack_info and self.formatStack(record.stack_info)])) diff --git a/lib/panels.py b/lib/panels.py index 3ab5566..2db5bd4 100644 --- a/lib/panels.py +++ b/lib/panels.py @@ -2,8 +2,7 @@ air-quality reading on its own, its good / poor / very poor zones. Every chart producer (a history answer, plot_chart, the weather links, the air-quality charts) builds its panels here, so a reading looks the same whichever path drew it.""" -from .airgradient.metrics import CHART_UNITS, LABELS, RATINGS -from .series import WEATHER +from .series import AIR, WEATHER from .specs import Bars, Line, Panel @@ -16,5 +15,5 @@ def panel_for(name: str | list[str], lines: list[Line] | None = None, bars: Bars return Panel(reading.label, reading.unit, lines, bars=bars, reading=name) names = [name] if isinstance(name, str) else name one = len(names) == 1 - return Panel(", ".join(LABELS[n] for n in names), CHART_UNITS[names[0]], lines, bars=bars, reading=names[0] if one else "", - zones=tuple(RATINGS[names[0]]) if one and len(lines) == 1 else None, peaks="pills" if one else "aside") + return Panel(", ".join(AIR[n].label for n in names), AIR[names[0]].unit, lines, bars=bars, reading=names[0] if one else "", + zones=AIR[names[0]].limits if one and len(lines) == 1 else None, peaks="pills" if one else "aside") diff --git a/lib/periods.py b/lib/periods.py index 1677aac..d5643f4 100644 --- a/lib/periods.py +++ b/lib/periods.py @@ -1,15 +1,17 @@ """The period buttons under a chart: 1D, 7D, 1M, 3M and 1Y (1, 7, 30, 90 and 365 days), always all five, the chart's own marked with ●. Pressing one redraws the chart for that period (Bot.on_period_button); the question behind each chart is remembered for that.""" +import time from collections import OrderedDict from dataclasses import dataclass from telegram import InlineKeyboardMarkup +from . import intent from .buttons import button -PERIODS = (1, 7, 30, 90, 365) -NAMES = {1: "1D", 7: "7D", 30: "1M", 90: "3M", 365: "1Y"} # what the buttons say +NAMES = {1: "1D", 7: "7D", 30: "1M", 90: "3M", 365: "1Y"} # each period, in days, and what its button says +PERIODS = tuple(NAMES) PREFIX = "pd:" REMEMBERED = 300 # charts whose question is kept, newest first to stay @@ -42,15 +44,6 @@ def _key(chat_id: int, message_id: int) -> str: def get(self, chat_id: int, message_id: int) -> str | None: return self.store.get(self._key(chat_id, message_id)) - def __getitem__(self, ids: tuple[int, int]) -> str: - return self.store[self._key(*ids)] - - def __contains__(self, ids: tuple[int, int]) -> bool: - return self._key(*ids) in self.store - - def __len__(self) -> int: - return len(self.store) - def remember(self, chat_id: int, message_id: int, question: str): key = self._key(chat_id, message_id) self.store.pop(key, None) # re-added at the end: the newest @@ -88,28 +81,29 @@ class ImageCache: FRESH = 330 # a little over the time between two rounds of drawing ahead, so a press always finds one def __init__(self, clock=None): - import time self.clock = clock or time.monotonic self.items: OrderedDict = OrderedDict() @staticmethod def key(question: str, days: int, now) -> str: """The question with its period made explicit, so "Temperature chart 30d" and a button's "…30d" are the one chart.""" - from . import intent return intent.with_period(question, days, now).lower() @classmethod def canonical(cls, question: str, now) -> str | None: """The key of a question that is its own key, word for word ("Temperature chart 7d", a button's): its chart is the same whoever asks it. None for any other ("temperature chart today" is kept under the 1d key, but it is another chart).""" - from . import intent key = cls.key(question, intent.period_days(question, now), now) return key if key == " ".join(question.split()).lower() else None + def _kept(self, entry: Drawn | None) -> bool: + """Is there a chart, and is it still within its time to live?""" + return entry is not None and self.age(entry) <= entry.ttl + def get(self, key: str) -> tuple[bytes | str, str] | None: """(the picture: Telegram's file id once it has been sent, else the PNG; its title), or None.""" entry = self.items.get(key) - if entry is None or self.clock() - entry.at > entry.ttl: + if not self._kept(entry): self.items.pop(key, None) return None self.items.move_to_end(key) @@ -118,14 +112,14 @@ def get(self, key: str) -> tuple[bytes | str, str] | None: def has_copy(self, key: str) -> bool: """Does Telegram hold this chart (drawn lately and already sent, so it has a file id)?""" entry = self.items.get(key) - return entry is not None and entry.file_id is not None and self.clock() - entry.at <= entry.ttl + return self._kept(entry) and entry.file_id is not None def ready(self, question: str, now) -> Drawn | None: """The chart made in code for exactly these words in the last FRESH seconds (drawn ahead, or for someone's question), with the text that went with it; None if there is none.""" key = self.canonical(question, now) entry = self.items.get(key) if key else None - if entry is None or entry.caption is None or self.clock() - entry.at > self.FRESH: + if entry is None or entry.caption is None or self.age(entry) > self.FRESH: return None self.items.move_to_end(key) return entry diff --git a/lib/pollen/parse.py b/lib/pollen/parse.py index dd28e76..d1bd82b 100644 --- a/lib/pollen/parse.py +++ b/lib/pollen/parse.py @@ -11,6 +11,7 @@ LEVEL_EMOJI = {"Low": "🟢", "Moderate": "🟠", "High": "🔴"} DISTRICTS = {"Central", "East Gippsland", "Mallee", "North Central", "North East", "Northern Country", "South West", "West and South Gippsland", "Wimmera"} +GRASS_HEADING, ASTHMA_HEADING = "Victorian District Grass Pollen Forecast", "Thunderstorm Asthma Forecast" class _TextLines(HTMLParser): @@ -34,15 +35,19 @@ def handle_data(self, data): self.lines.append(text) +def _find(lines: list[str], heading: str) -> int | None: + """The first line that starts with `heading`, whatever its case (the site adds words after a heading at times).""" + return next((i for i, line in enumerate(lines) if line.lower().startswith(heading.lower())), None) + + def _section(lines: list[str], heading: str, stops: list[str]) -> list[str]: - """Lines after the first line equal to `heading`, up to a line starting with a stop.""" - try: - i = lines.index(heading) - except ValueError: + """Lines after the heading's, up to a line starting with a stop.""" + i = _find(lines, heading) + if i is None: return [] out = [] for line in lines[i + 1:]: - if any(line.startswith(s) for s in stops): + if any(line.lower().startswith(s.lower()) for s in stops): break out.append(line) return out @@ -79,9 +84,9 @@ def parse_melbourne_pollen(html: str) -> dict: except ValueError: pass - grass = _section(lines, "Victorian District Grass Pollen Forecast", ["Thunderstorm Asthma Forecast"]) - asthma = _section(lines, "Thunderstorm Asthma Forecast", ["Note:", "Recent News"]) - after = lines[lines.index("Thunderstorm Asthma Forecast"):] if "Thunderstorm Asthma Forecast" in lines else [] + grass = _section(lines, GRASS_HEADING, [ASTHMA_HEADING]) + asthma = _section(lines, ASTHMA_HEADING, ["Note:", "Recent News"]) + after = lines[i:] if (i := _find(lines, ASTHMA_HEADING)) is not None else [] return { "melbourne_grass": level, "melbourne_date": date, diff --git a/lib/pollen/source.py b/lib/pollen/source.py index 6337c0a..721028e 100644 --- a/lib/pollen/source.py +++ b/lib/pollen/source.py @@ -7,17 +7,18 @@ import json import logging import time -from datetime import date, datetime +from datetime import date import httpx from .. import intent from ..config import Config from ..snapshots import Snapshots -from ..timeutil import now_local, when -from ..tools import Tool, Turn +from ..sources import Source +from ..timeutil import day_text, local_date, when +from ..tools import Tool, Turn, json_result from ..warm import Warmer, in_sync_hours -from .parse import LEVEL_EMOJI, parse_melbourne_pollen +from .parse import DISTRICTS, LEVEL_EMOJI, parse_melbourne_pollen log = logging.getLogger(__name__) @@ -33,17 +34,26 @@ PARAMETERS = {"type": "object", "properties": {}} -class Pollen: +def _district(name: str) -> str: + """The site's name for this district, whatever its case. One the site doesn't have is warned about: no level can be found for it.""" + known = {d.lower(): d for d in DISTRICTS} + if name.lower() not in known: + log.warning("POLLEN_DISTRICT %r is not one of the site's districts (%s): no district level will be found", name, + ", ".join(sorted(DISTRICTS))) + return known.get(name.lower(), name) + + +class Pollen(Source): name = "Pollen" def __init__(self, cfg: Config, transport=None): - self.tz, self.district, self.place = cfg.tz, cfg.pollen_district, cfg.place + self.tz, self.district, self.place = cfg.tz, _district(cfg.pollen_district), cfg.place self.client = httpx.AsyncClient(timeout=15, headers=HEADERS, follow_redirects=True, transport=transport) self.data: dict | None = None # the parsed page self.fetched_at = 0.0 # when (epoch) self._validators: dict = {} # ETag / Last-Modified for the conditional request self.requests = 0 - self.store = Snapshots.open(cfg.conditions_cache_path) + self.store = Snapshots(cfg.conditions_cache_path) self._load() self.warmer = Warmer(self.warm, POLLEN_REFRESH_SECONDS) self.tools = [Tool("pollen_asthma", DESCRIPTION, PARAMETERS, self.handle)] @@ -54,7 +64,7 @@ async def start(self): return try: await self.warm(True) - log.info("Pollen: %s", self.summary()) + log.info("Pollen: %s", "; ".join(self.lines()) or "no levels on the page") except Exception as e: # the site may just be briefly unreachable log.warning("Pollen not readable yet: %s", e) @@ -65,13 +75,6 @@ def wants(self, text: str) -> bool: """Only pollen questions start a refresh; the report always uses what is cached.""" return intent.mentions_pollen(text) - def poke(self): - self.warmer.poke() - - async def close(self): - await self.client.aclose() - self.store.close() - # ---------- kept on disk ---------- def _load(self): """Start from the last page seen (before the restart), so a night-time start needs no fetch.""" @@ -87,19 +90,25 @@ def _save(self): self.store.put("pollen_validators", json.dumps(self._validators)) # ---------- fetching ---------- - def now(self) -> datetime: - return now_local(self.tz) - def in_season(self) -> bool: return self.now().month in SEASON_MONTHS + @property + def page(self) -> dict | None: + """The parsed page, if it is this season's: its grass forecast's date says (else when it was fetched). Last season's page, + kept over the summer or still on the site on 1 October, is never shown.""" + if self.data is None or not self.in_season(): + return None + day = self.data.get("melbourne_date") or local_date(int(self.fetched_at), self.tz) + return self.data if day >= date(self.now().year, SEASON_MONTHS[0], 1) else None + async def warm(self, fresh: bool = True) -> str: - """Fetch the page when it is due: never out of season, never outside the sync hours (unless nothing is cached), and not - more often than the poll interval.""" + """Fetch the page when it is due: never out of season, never outside the sync hours (unless this season's page isn't + cached), and not more often than the poll interval.""" if not self.in_season(): return "Pollen out of season" - age = time.time() - self.fetched_at - if self.data is not None and (not in_sync_hours(self.now()) or age < POLLEN_REFRESH_SECONDS - 5 and not fresh): + young = self.data is not None and time.time() - self.fetched_at < POLLEN_REFRESH_SECONDS - 5 and not fresh + if young or self.page is not None and not in_sync_hours(self.now()): return "Pollen cached" self.requests += 1 res = await self.client.get(URL, headers=self._validators) @@ -114,34 +123,25 @@ async def warm(self, fresh: bool = True) -> str: self._save() if not self.data["melbourne_grass"] and not self.data["district_grass"]: log.warning("Pollen: no grass pollen level found on the page (has its layout changed?)") + if self.district not in self.data["thunderstorm_asthma"]: + log.warning("Pollen: no thunderstorm asthma level for %s on the page (has its layout changed?)", self.district) return "Pollen 1 req" # ---------- reading ---------- def current(self) -> dict: """{"grass": {level, emoji, date} | None, "asthma": {level, emoji, updated} | None, "fetched_at": epoch}""" - d = self.data if self.in_season() else {} # last season's page is never shown - d = d or {} + d = self.page or {} grass = d.get("melbourne_grass") or (d.get("district_grass") or {}).get(self.district) asthma = (d.get("thunderstorm_asthma") or {}).get(self.district) return {"grass": {"level": grass, "emoji": LEVEL_EMOJI[grass], "date": d.get("melbourne_date")} if grass else None, "asthma": {"level": asthma, "emoji": LEVEL_EMOJI[asthma], "updated": d.get("asthma_updated")} if asthma else None, "fetched_at": self.fetched_at} - def summary(self) -> str: - """The levels in a few words for the log: "grass 🔴 High (Sat 17 Oct), asthma 🟢 Low".""" - cur, parts = self.current(), [] - if g := cur["grass"]: - when = f" ({g['date']:%a %d %b})" if g["date"] and g["date"] != self.now().date() else "" - parts.append(f"grass {g['emoji']} {g['level']}{when}") - if a := cur["asthma"]: - parts.append(f"asthma {a['emoji']} {a['level']}") - return ", ".join(parts) or "no levels on the page" - def lines(self) -> list[str]: """The ready-made report lines; the asthma line is left out when there is no risk forecast (outside the season).""" cur, out = self.current(), [] if g := cur["grass"]: - when = f" (forecast for {g['date']:%a %d %b})" if g["date"] and g["date"] != self.now().date() else "" + when = f" (forecast for {day_text(g['date'], year=False)})" if g["date"] and g["date"] != self.now().date() else "" out.append(f"Grass pollen: {g['emoji']} {g['level']}{when}") if a := cur["asthma"]: out.append(f"Thunderstorm asthma risk: {a['emoji']} {a['level']}") @@ -149,15 +149,15 @@ def lines(self) -> list[str]: async def handle(self, args: dict, turn: Turn | None = None) -> str: if not self.in_season(): - return json.dumps({"error": OFF_SEASON}) - if self.data is None and not args.get("cached"): # the report ("cached") never fetches + return json_result({"error": OFF_SEASON}) + if self.page is None and not args.get("cached"): # the report ("cached") never fetches try: await self.warm(True) except Exception as e: log.warning("Pollen not readable: %s", e) lines = self.lines() if not lines: - return json.dumps({"error": "No pollen levels are available right now."}) + return json_result({"error": "No pollen levels are available right now."}) cur = self.current() out = {"lines": lines, "place": self.place, "source": "melbournepollen.com.au (grass pollen from pollen-trap data; asthma risk from the " "Victorian Department of Health and BOM, 1 Oct - 31 Dec)"} @@ -165,6 +165,6 @@ async def handle(self, args: dict, turn: Turn | None = None) -> str: out["time"] = when(int(cur["fetched_at"]), self.tz) # when the page was last fetched if cur["asthma"] and cur["asthma"]["updated"]: out["asthma_updated"] = cur["asthma"]["updated"] - if not cur["asthma"]: - out["asthma_note"] = "No thunderstorm asthma forecast now (it runs 1 Oct - 31 Dec)." - return json.dumps(out, ensure_ascii=False) + if not cur["asthma"]: # in season the page always has one: it may have changed + out["asthma_note"] = "The page had no thunderstorm asthma level for this district when it was last read." + return json_result(out) diff --git a/lib/prompt.py b/lib/prompt.py index 7d1cb2c..902c115 100644 --- a/lib/prompt.py +++ b/lib/prompt.py @@ -1,13 +1,26 @@ """The system prompt. Everything the model would otherwise have to work out (dates, devices, units, wording of times) is decided in code and handed over ready-made.""" +from collections.abc import Collection from datetime import datetime from .airgradient.metrics import ALL_METRICS, CHART_UNITS, LABELS +from .alerts.forecast import TEMP_CHANGE_C +from .alerts.menu import TYPES +from .alerts.weather import GUST_ALERT_KMH +from .ecowitt.glance import UVI_ALERT from .intent import period_ranges from .series import WEATHER +from .timeutil import day_text WEATHER_NAMES = list(WEATHER) +NEVER_AVAILABLE = ["lightning", "soil or extra sensor channels", "indoor air quality", "other stations or places"] +WITHOUT = {"Pollen": "pollen and thunderstorm asthma", "Forecast": "forecasts (only a short read of the pressure trend)"} # what a source that is off leaves out +# Each alert type as the prompt describes it, with its own threshold; told only when the source that sends it is on (alerts.menu.TYPES) +ALERT_WORDS = {"rain": "rain starting or stopping", "gusts": f"wind gusts over {GUST_ALERT_KMH} km/h", + "uv": f"UV index of {UVI_ALERT} or more", "temps": "indoor/outdoor temperature crossing", + "particulates": "air-quality mask alerts", "pollen_and_asthma": "pollen or thunderstorm asthma risk reaching High", + "forecast": f"a forecast it sent being revised (rain, or max temperature over {TEMP_CHANGE_C:g} degrees)"} PROMPT = """\ You are a friendly weather bot on Telegram. @@ -145,7 +158,7 @@ def date_ranges(now: datetime) -> str: """Pre-computed date ranges so the model never does calendar maths (same definitions as the fast path).""" - fmt = lambda d: d.strftime("%a %d %b %Y") + fmt = day_text periods = period_ranges(now.date()) lines = [f" Today: {fmt(now)} (00:00 to now)"] for name, label in DATE_LABELS: @@ -154,15 +167,12 @@ def date_ranges(now: datetime) -> str: return "\n".join(lines) -def capabilities(sources: list[str], brief: bool = False) -> str: - """What the bot can and can't do, for questions about the bot itself ("what metrics do you have?") and so the model - knows what is possible before it says something can't be done. Built from the code's own lists.""" - have = " ".join(sources) +def capabilities(have: Collection[str], brief: bool = False) -> str: + """What the bot can and can't do with the sources it has (by name), for questions about the bot itself ("what metrics do you + have?") and so the model knows what is possible before it says something can't be done. Built from the code's own lists.""" + missing = ", ".join([*NEVER_AVAILABLE, *(what for name, what in WITHOUT.items() if name not in have)]) if brief: # most questions: only the limits, so it never promises what it can't do - missing = ["lightning", "soil or extra sensor channels", "indoor air quality", "other stations or places"] - missing += (["pollen and thunderstorm asthma"] if "melbournepollen" not in have else []) + ( - ["forecasts (only a short read of the pressure trend)"] if "Open-Meteo" not in have else []) - return ("\nLIMITS: not available: " + ", ".join(missing) + ". Custom alerts (\"tell me when winds reach 100\") can't be added: say so. " + return (f"\nLIMITS: not available: {missing}. Custom alerts (\"tell me when winds reach 100\") can't be added: say so. " "A chart series or style the tools don't offer can't be drawn: say so.\n") lines = ["\nWHAT THIS BOT CAN AND CAN'T DO (answer questions about the bot from this, with no tool call)"] if "Ecowitt" in have: @@ -182,24 +192,18 @@ def capabilities(sources: list[str], brief: bool = False) -> str: "cleaner air, and air_scan scans every air metric against every weather reading for what goes with what. " "Use them for \"plot X against Y\", \"does rain affect air quality\" and \"is there a correlation between air " "quality and other metrics\"; a series or style not listed can't be plotted: say so.") - if "melbournepollen" in have: + if "Pollen" in have: lines.append("- Pollen: Melbourne's grass pollen level and the thunderstorm asthma risk (Low, Moderate, High), tool pollen_asthma. " "Both forecasts only run from October to December: outside those months the tool says so.") - if "Open-Meteo" in have: + if "Forecast" in have: lines.append("- Forecast: today and the days ahead from Open-Meteo (summary, temperatures, chance of at least 1 mm of rain), tool weather_forecast.") lines.append("- Charts: any one reading, or several readings together on one time axis, one panel each (" + ", ".join(WEATHER_NAMES) + "; \"weather all week\" draws every reading); wind as average speed with gusts beside a compass rose of directions; " "air quality with ratings.") - lines.append("- Alerts, sent to chats automatically: rain starting or stopping, wind gusts over 40 km/h, UV index of 10 or more, " - "indoor/outdoor temperature crossing, air-quality mask alerts" + (", pollen or thunderstorm asthma risk reaching High" if "melbournepollen" in have else "") + ( - ", a forecast it sent being revised (rain, or max temperature over 2 degrees)" if "Open-Meteo" in have else "") + ". Each alert has buttons to subscribe or unsubscribe by type, and /alerts opens the same settings. Custom alerts " - "(\"tell me when winds reach 100\", another limit) can't be added: say so.") - missing = ["lightning", "soil or extra sensor channels", "indoor air quality", "other stations or places"] - if "melbournepollen" not in have: - missing.append("pollen and thunderstorm asthma") - if "Open-Meteo" not in have: - missing.append("forecasts (only a short read of the pressure trend)") - lines.append("- Not available: " + ", ".join(missing) + ".") + if alerts := ", ".join(words for kind, words in ALERT_WORDS.items() if TYPES[kind][1] in have): + lines.append(f"- Alerts, sent to chats automatically: {alerts}. Each alert has buttons to subscribe or unsubscribe by type, and " + "/alerts opens the same settings. Custom alerts (\"tell me when winds reach 100\", another limit) can't be added: say so.") + lines.append(f"- Not available: {missing}.") return "\n".join(lines) + "\n" @@ -209,12 +213,13 @@ def capabilities(sources: list[str], brief: bool = False) -> str: """ -def build(now: datetime, sources: list[str], hints: list[str] = (), about_bot: bool = False, rain_caption: bool = False, +def build(now: datetime, sources: dict[str, str], hints: list[str] = (), about_bot: bool = False, rain_caption: bool = False, topics: set[str] | None = None) -> str: - """The system prompt. `topics` (see intent.topics) picks the guidance the question needs; None puts all of it in.""" + """The system prompt for the bot's sources ({name: its line for the prompt, Source.describe()}). `topics` (see intent.topics) + picks the guidance the question needs; None puts all of it in.""" # Most stable first: the model provider caches a prompt's start, so what changes (the topics, the date, then the time) goes last everything = topics is None - text = PROMPT.format(sources="\n".join(f"- {s}" for s in sources) or "(none)", + text = PROMPT.format(sources="\n".join(f"- {s}" for s in sources.values()) or "(none)", capabilities=capabilities(sources, brief=not (everything or about_bot or "bot" in topics))) text += "".join(f"\n{block}" for name, block in BLOCKS.items() if everything or name in topics) text += f"\nRIGHT NOW (exact ranges, then the time)\n{date_ranges(now)}\nIt is now {now.strftime('%A %d %B %Y, %H:%M %Z')}.\n" diff --git a/lib/rain.py b/lib/rain.py index b4c28cf..bb9cc63 100644 --- a/lib/rain.py +++ b/lib/rain.py @@ -1,10 +1,11 @@ """Rain from the station's running daily total, at 30-minute slots: how much fell in each, which slots make up a spell, and the bars a chart draws. Shared by the analyses and the charts.""" +from bisect import bisect_right from datetime import date, timedelta, tzinfo -from .specs import Bars -from .timeutil import SLOT, day_bounds, to_local +from .specs import Bars, calendar_bars +from .timeutil import SLOT, at_hour, day_bounds, to_local SPELL_GAP = 6 # slots: 3 dry hours end a rain spell SPELL_MM = 1.0 # a spell needs this much rain to count as an event @@ -38,12 +39,19 @@ def rain_spells(rain: dict[int, float], wet: list[int]) -> list[list[int]]: return [s for s in spells if sum(rain[t] for t in s) >= SPELL_MM] -def bar_layout(tz: tzinfo, first: date, last: date) -> tuple[int, int, str]: - """(origin epoch, seconds per bar, what a bar covers) for a bar chart of this period: hourly up to 4 days, 6-hourly - up to a month, daily beyond. Bars start at local midnight of the first day.""" +def bar_layout(tz: tzinfo, first: date, last: date) -> tuple[list[int], int, str]: + """(each bar's start, seconds per bar, what a bar covers) for a bar chart of this period: hourly up to 4 days, 6-hourly up + to a month, daily beyond. The bars start at the same local clock times every day (midnight, 6am ...) from midnight of the + first day, so a daylight-saving change doesn't move them (a day of 23 or 25 hours has one bar the shorter or longer).""" width = next(w for limit, w in CHART_BARS if (last - first).days + 1 <= limit) - return (day_bounds(first, tz)[0], width, - {3600: "hour", 6 * 3600: "6 hours", 86400: "day"}[width]) + starts = {at_hour(first + timedelta(days=n), hour, tz) for n in range((last - first).days + 1) for hour in range(0, 24, width // 3600)} + return sorted(starts), width, {3600: "hour", 6 * 3600: "6 hours", 86400: "day"}[width] + + +def bar_of(starts: list[int], t: int) -> int | None: + """The start of the bar that holds epoch `t` (None before the first).""" + i = bisect_right(starts, t) - 1 + return starts[i] if i >= 0 else None def rain_bars(rain: dict[int, float], tz: tzinfo, first: date, last: date, until: int | None = None, by: str | None = None, @@ -53,30 +61,22 @@ def rain_bars(rain: dict[int, float], tz: tzinfo, first: date, last: date, until own start) at the bar that holds it: a rolling last 24 hours has no bars left of its data.""" if by in ("month", "year"): # a figure per month or year was asked for: one bar for each, whatever the length return _calendar_bars(rain, tz, first, last, by) - origin, width, per = bar_layout(tz, first, last) + starts, width, per = bar_layout(tz, first, last) end = min(day_bounds(last, tz)[1], until) if until else day_bounds(last, tz)[1] - begin = origin + (since - origin) // width * width if since is not None and since > origin else origin - bars: dict[int, float] = dict.fromkeys(range(begin, end, width), 0.0) + begin = bar_of(starts, since) if since is not None and since > starts[0] else starts[0] + bars: dict[int, float] = dict.fromkeys((s for s in starts if begin <= s < end), 0.0) for t, mm in rain.items(): - if mm > 0: - k = origin + (t - origin) // width * width - if k >= begin: # rain before the period's start is not part of it - bars[k] = bars.get(k, 0.0) + mm - xs = sorted(bars) - return Bars("Rain", "mm", xs, [round(bars[k], 2) for k in xs], width, per) + if mm > 0 and begin <= t < end: # rain before the period's start is not part of it + bars[bar_of(starts, t)] += mm + return Bars("Rain", "mm", list(bars), [round(v, 2) for v in bars.values()], width, per) def _calendar_bars(rain: dict[int, float], tz: tzinfo, first: date, last: date, by: str) -> Bars: - """Rain summed per calendar month or year over the period (each with its figure above the bar).""" - key = (lambda d: (d.year, d.month)) if by == "month" else (lambda d: (d.year,)) - totals: dict[tuple, float] = {} - day = first - while day <= last: # every month or year of the period gets a bar, a dry one as 0 - totals.setdefault(key(day), 0.0) - day += timedelta(days=1) + """Rain summed per calendar month or year over the period, each with its total above its bar (every month or year of the + period has one, a dry one 0).""" + start = (lambda d: d.replace(day=1)) if by == "month" else (lambda d: d.replace(month=1, day=1)) + totals = dict.fromkeys((start(first + timedelta(days=n)) for n in range((last - first).days + 1)), 0.0) for t, mm in rain.items(): if mm > 0 and first <= (d := to_local(t, tz).date()) <= last: - totals[key(d)] += mm - starts = [date(k[0], k[1] if by == "month" else 1, 1) for k in totals] - return Bars("Rain", "mm", [day_bounds(d, tz)[0] for d in starts], [round(v, 1) for v in totals.values()], - 28 * 86400 if by == "month" else 365 * 86400, by, values=True) + totals[start(d)] += mm + return calendar_bars("Rain", "mm", {d: round(v, 1) for d, v in totals.items()}, by, tz) diff --git a/lib/report.py b/lib/report.py index 9d9b45d..9d0ad64 100644 --- a/lib/report.py +++ b/lib/report.py @@ -3,14 +3,15 @@ import json import re -from datetime import datetime +from datetime import date, datetime from telegram import MessageEntity +from .ecowitt.api import unit_text from .ecowitt.glance import solar_band -from .series import WEATHER, find - -AIR_ROWS = (("pm2_5", "PM2.5"), ("pm10", "PM10"), ("pm1", "PM1"), ("co2", "CO₂"), ("voc_index", "VOC index"), ("nox_index", "NOx index")) +from .entities import utf16_len +from .series import AIR, INDOOR_READINGS, WEATHER, find +from .timeutil import clock_text, day_text, to_local def _load(result: str | None) -> dict | None: @@ -22,11 +23,6 @@ def _load(result: str | None) -> dict | None: return data if isinstance(data, dict) and "error" not in data else None -def _units(text: str | None) -> str | None: - """"12.3 ℃" -> "12.3 °C", "1.8 mm/hr" -> "1.8 mm/h".""" - return text.replace("℃", "°C").replace(" º", "°").replace("º", "°").replace("mm/hr", "mm/h").strip() if text else None - - def _zero(text: str | None) -> bool: """Is this reading missing or zero ("0.0 W/m²", "0")?""" try: @@ -47,32 +43,19 @@ def _join(parts: list[str | None], sep: str = ", ") -> str: return sep.join(p for p in parts if p) -HEADINGS = ("Weather station", "Air quality", "Pollen & asthma", "Forecast") # the names the report's sections (and the pollen and forecast lookups) start with -HEADING_LINE = re.compile(rf"^({'|'.join(map(re.escape, HEADINGS))})(?= \(|$)", re.M) - - -def heading_entities(text: str) -> list[MessageEntity]: - """Bold entities over the name of each heading line in `text` (not over its brackets). Telegram counts positions in UTF-16 code - units, where an emoji is two.""" - units = lambda s: len(s.encode("utf-16-le")) // 2 - return [MessageEntity(MessageEntity.BOLD, units(text[:m.start()]), units(m.group(1))) for m in HEADING_LINE.finditer(text)] - - -def _stamp(result: dict) -> str | None: - """The time of day of the data, "11:45pm", from the result's "time" ("Sun 11 Oct 2026 23:50" from the station, "Sun 11 Oct 2026 11:45pm" - from the rest); None when it is missing or unreadable.""" - for form in ("%a %d %b %Y %I:%M%p", "%a %d %b %Y %H:%M"): - try: - at = datetime.strptime(result.get("time") or "", form) - except ValueError: - continue - return f"{at.hour % 12 or 12}:{at:%M}{'am' if at.hour < 12 else 'pm'}" - return None +def _stamp(result: dict, today: date) -> str | None: + """The time of the data, "11:45pm", with its day when that isn't today ("Sat 10 Oct 11:45pm"), from the result's "time" (as + timeutil.when writes it, "Sun 11 Oct 2026 11:45pm"); None when it is missing or unreadable.""" + try: + at = datetime.strptime(result.get("time") or "", "%a %d %b %Y %I:%M%p") + except ValueError: + return None + return clock_text(at) if at.date() == today else f"{day_text(at, year=False)} {clock_text(at)}" -def _heading(name: str, result: dict) -> str: +def _heading(name: str, result: dict, today: date) -> str: """"Forecast (Melbourne, 6:00pm)": the place the result names and the time of its data, whichever it has.""" - note = ", ".join(x for x in (result.get("place"), _stamp(result)) if x) + note = ", ".join(x for x in (result.get("place"), _stamp(result, today)) if x) return f"{name} ({note})" if note else name @@ -81,7 +64,7 @@ def _getters(now: dict): emoji = now.get("emoji") or {} def get(group: str, field: str) -> str | None: - return _units((now.get(group) or {}).get(field)) + return unit_text((now.get(group) or {}).get(field)) def tag(group: str, field: str, text: str | None = None) -> str | None: text = text or get(group, field) @@ -89,18 +72,26 @@ def tag(group: str, field: str, text: str | None = None) -> str | None: return get, tag +def _rain(now: dict) -> str: + """Today's rain, the month's and, while it rains, the rate: "☔ 6.3 mm (month total 49.8 mm); 🌧️ raining (1.8 mm/h)".""" + get, tag = _getters(now) + rate, month = get("rainfall", "rain_rate"), get("rainfall", "monthly") + return _join([_join([tag("rainfall", "daily"), f"(month total {month})" if month else None], " "), + None if _zero(rate) else tag("rainfall", "rain_rate", f"raining ({rate})")], "; ") + + +def _wind(now: dict) -> str | None: + """The wind, where it comes from and its gusts: "14.0 km/h from 131°, gust 31.0 km/h"; None when it is calm.""" + get, tag = _getters(now) + if _zero(get("wind", "wind_speed")) and _zero(get("wind", "wind_gust")): + return None + direction, gust = get("wind", "wind_direction"), tag("wind", "wind_gust") + return _join([_join([tag("wind", "wind_speed"), f"from {direction}" if direction else None], " "), f"gust {gust}" if gust else None]) + + def weather_lines(now: dict) -> list[str]: """The weather station's bullets from a weather_now result: an emoji sits right before the reading it is keyed to.""" get, tag = _getters(now) - rain_rate = (now.get("rainfall") or {}).get("rain_rate", "") - raining = tag("rainfall", "rain_rate", f"raining ({get('rainfall', 'rain_rate')})") if rain_rate.split()[:1] not in ([], ["0"], ["0.0"]) else None - month = get("rainfall", "monthly") - rain = _join([_join([tag("rainfall", "daily"), f"(month total {month})" if month else None], " "), raining], "; ") - - degrees = (now.get("wind") or {}).get("wind_direction", "") - calm = _zero(get("wind", "wind_speed")) and _zero(get("wind", "wind_gust")) # no wind: no line - wind = None if calm else _join([_join([tag("wind", "wind_speed"), f"from {_units(degrees)}" if degrees else None], " "), - f"gust {tag('wind', 'wind_gust')}" if get("wind", "wind_gust") else None]) solar, uvi = get("solar_and_uvi", "solar"), get("solar_and_uvi", "uvi") sun = _join([None if _zero(solar) else tag("solar_and_uvi", "solar", f"solar radiation {solar}{_band(solar)}"), # night: no sun line None if _zero(uvi) else tag("solar_and_uvi", "uvi", f"UV index {uvi}")]) @@ -109,15 +100,15 @@ def weather_lines(now: dict) -> list[str]: tag("outdoor", "vpd", f"VPD {get('outdoor', 'vpd')}" if get("outdoor", "vpd") else None)])), ("Indoor", _join([tag("indoor", "temperature"), tag("indoor", "humidity")])), ("Pressure", tag("pressure", "relative")), - ("Rain today", rain), + ("Rain today", _rain(now)), ("Sun", sun), - ("Wind", wind)] + ("Wind", _wind(now))] # calm: no line return [f"• {name}: {text}" for name, text in bullets if text] def air_lines(air: dict, only: list[str] | None = None) -> list[str]: out = [] - for key, label in AIR_ROWS: + for key, label in ((name, m.label) for name, m in AIR.items()): entry = air.get(key) if only is None or key in only else None if not isinstance(entry, dict) or entry.get("value") is None: continue @@ -135,64 +126,50 @@ def reading_lines(now: dict, names: list[str], sides: list[str]) -> list[str]: lines = [] for name in names: reading = WEATHER[name] - if reading.group in ("outdoor", "indoor") and name in ("temperature", "humidity"): + if name in ("temperature", "humidity"): # a line for each side asked about; the temperature with what it feels like for side in sides: - text = _join([tag(side, name), f"feels like {get(side, 'feels_like')}" if name == "temperature" and get(side, "feels_like") else None], ", ") - lines.append(f"• {side.capitalize()}: {text}" if text else None) + feels = get(side, "feels_like") if name == "temperature" else None + if text := _join([tag(side, reading.field), f"feels like {feels}" if feels else None]): + lines.append(f"• {side.capitalize()}: {text}") elif name == "wind": - wind = (now.get("wind") or {}) - if _zero(get("wind", "wind_speed")) and _zero(get("wind", "wind_gust")): - lines.append("• Wind: calm") - else: - degrees = wind.get("wind_direction") - lines.append("• Wind: " + _join([_join([tag("wind", "wind_speed"), f"from {_units(degrees)}" if degrees else None], " "), - f"gust {tag('wind', 'wind_gust')}" if get("wind", "wind_gust") else None])) + lines.append(f"• Wind: {_wind(now) or 'calm'}") elif name == "rain": - lines += [line for line in weather_lines(now) if line.startswith("• Rain today")] or [ - f"• Rain today: {get('rainfall', 'daily') or 'none'}"] + lines.append(f"• Rain today: {_rain(now) or 'none'}") else: - text = tag(reading.group, reading.field) - if text: - lines.append(f"• {reading.label}: {text}") - return [line for line in lines if line] + group = sides[0] if name in INDOOR_READINGS and len(sides) == 1 else reading.group # "indoor dew point": that sensor's + if text := tag(group, reading.field): + lines.append(f"• {reading.label}{' (indoor)' if group == 'indoor' else ''}: {text}") + return lines -def lookup(kind: str, result: str | None, names: list[str], sides: list[str]) -> str | None: - """A plain lookup written from its one tool result: "reading", "air", "pollen" or "forecast". None when the result can't be - used (the model then takes the question).""" - data = _load(result) - if not data: - return None - if kind == "reading": - lines = reading_lines(data, names, sides) - elif kind == "air": - lines = air_lines(data, [k for k in names] or None) - elif kind == "days": # "137 of 1,454 days with UV index \u2265 10 (period)": the chart (if any) carries the breakdown - if "matching_days" not in data: - return None - if data.get("stat") and data.get("value") is not None: # a total, average, highest or lowest over the period - unit = _units(data["stat"].get("unit")) or "" - return f"{data['stat']['what']}: {data['value']:g} {unit}".rstrip() + f" over {data['days_checked']:,} days ({_period(data)})." - what = f" with {data['condition']}" if data.get("condition") else "" - note = " Older days use averages, so a count near the limit may fall slightly short." if data.get("note_averaged") else "" - return f"{data['matching_days']:,} of {data['days_checked']:,} days{what} ({_period(data)}).{note}" - elif kind == "extremes": # the highs and lows (or averages) of a short period, each with its day and time - series = data.get("series") or {} - wanted = [f for n in names if find(n) for f in (find(n).field, find(n).band_field) if f] or ["temperature"] # wind: speed and gust - keys = [k for k in series if k.split(".", 1)[1] in wanted and "low" in series[k]] - lines = [] - for k in keys: - group, field = k.split(".", 1) - label = ("Wind gust" if field == "wind_gust" else find(field).label if find(field) else field.replace("_", " ").capitalize()) + ( - f" ({group})" if sum(x.split(".", 1)[1] == field for x in keys) > 1 else "") - lines.append(f"• {_range(label, series[k], 'average' in series[k])}") - return "\n".join([_period(data), *lines]) if lines else None - elif kind in ("pollen", "forecast"): - lines = [f"• {line}" for line in data.get("lines", [])] - return "\n".join([_heading("Pollen & asthma" if kind == "pollen" else "Forecast", data), *lines]) if lines else None - else: - return None - return "\n".join(lines) or None +def _own_lines(result: dict) -> list[str]: + """The bullets of a result that brings its own lines (pollen and asthma, the forecast).""" + return [f"• {line}" for line in result.get("lines", [])] + + +# The report's sections in order: the tool, the section's heading, its bullets from the tool's result, and whether it is still +# shown when that result failed ("• not available right now") rather than left out +SECTIONS = (("weather_now", "Weather station", weather_lines, True), + ("air_quality", "Air quality", air_lines, True), + ("pollen_asthma", "Pollen & asthma", _own_lines, False), + ("weather_forecast", "Forecast", _own_lines, False)) +HEADINGS = {tool: heading for tool, heading, _, _ in SECTIONS} # the pollen and forecast lookups start with theirs too +HEADING_LINE = re.compile(rf"^({'|'.join(map(re.escape, HEADINGS.values()))})(?= \(|$)", re.M) + + +def heading_entities(text: str) -> list[MessageEntity]: + """Bold entities over the name of each heading line in `text` (not over its brackets).""" + return [MessageEntity(MessageEntity.BOLD, utf16_len(text[:m.start()]), utf16_len(m.group(1))) for m in HEADING_LINE.finditer(text)] + + +def report(results: dict[str, str], today: date) -> str: + """The full report from the tool results by tool name (weather_now, air_quality, pollen_asthma, weather_forecast), on `today` (local).""" + sections = [] + for tool, heading, bullets, always in SECTIONS: + data = _load(results.get(tool)) + if data or (always and tool in results): + sections.append("\n".join([_heading(heading, data or {}, today), *(bullets(data) if data else ["• not available right now"])])) + return "\n\n".join(sections) or "No readings are available right now." def weather_now(result: str | None) -> str | None: @@ -201,20 +178,60 @@ def weather_now(result: str | None) -> str | None: return "\n".join(weather_lines(now)) if now else None -def report(results: dict[str, str]) -> str: - """The full report from the tool results by tool name (weather_now, air_quality, pollen_asthma, weather_forecast).""" - sections = [] - if "weather_now" in results: - now = _load(results["weather_now"]) - sections.append([_heading("Weather station", now or {}), *(weather_lines(now) if now else ["• not available right now"])]) - if "air_quality" in results: - air = _load(results["air_quality"]) - sections.append([_heading("Air quality", air or {}), *(air_lines(air) if air else ["• not available right now"])]) - if pollen := _load(results.get("pollen_asthma")): - sections.append([_heading("Pollen & asthma", pollen), *(f"• {line}" for line in pollen.get("lines", []))]) - if forecast := _load(results.get("weather_forecast")): - sections.append([_heading("Forecast", forecast), *(f"• {line}" for line in forecast.get("lines", []))]) - return "\n\n".join("\n".join(section) for section in sections) or "No readings are available right now." +# ---------- plain lookups ---------- +def _alone(tool: str, result: dict, today: date) -> str | None: + """A pollen or forecast result on its own: its section of the report, or None when it has no lines.""" + lines = _own_lines(result) + return "\n".join([_heading(HEADINGS[tool], result, today), *lines]) if lines else None + + +def _days(data: dict) -> str | None: + """"137 of 1,454 days with UV index ≥ 10 (period)", or a total, average, highest or lowest over the period: the chart (if any) + carries the breakdown.""" + if "matching_days" not in data: + return None + if data.get("stat") and data.get("value") is not None: + unit = unit_text(data["stat"].get("unit")) or "" + return f"{data['stat']['what']}: {data['value']:g} {unit}".rstrip() + f" over {data['days_checked']:,} days ({_period(data)})." + what = f" with {data['condition']}" if data.get("condition") else "" + note = " Older days use averages, so a count near the limit may fall slightly short." if data.get("note_averaged") else "" + return f"{data['matching_days']:,} of {data['days_checked']:,} days{what} ({_period(data)}).{note}" + + +def _series_label(field: str) -> str: + """A series' name: its reading's label ("Temperature"), the gusts their own ("Wind gust").""" + if field == "wind_gust": + return "Wind gust" + return reading.label if (reading := find(field)) else field.replace("_", " ").capitalize() + + +def _extremes(data: dict, names: list[str]) -> str | None: + """The period, then the low and high (or the average) of each reading named, each with its day and time: wind, its speed and + its gusts; nothing named, the temperature. A reading on both sides (indoor, outdoor) has a line for each.""" + series = data.get("series") or {} + wanted = [f for r in filter(None, map(find, names)) for f in (r.field, r.band_field) if f] or ["temperature"] + keys = [k for k in series if k.split(".", 1)[1] in wanted and "low" in series[k]] + fields = [k.split(".", 1)[1] for k in keys] + lines = [] + for key in keys: + group, field = key.split(".", 1) + label = _series_label(field) + (f" ({group})" if fields.count(field) > 1 else "") + lines.append(f"• {_range(label, series[key], 'average' in series[key])}") + return "\n".join([_period(data), *lines]) if lines else None + + +def lookup(kind: str, result: str | None, names: list[str], sides: list[str], today: date) -> str | None: + """A plain lookup written from its one tool result: `kind` is "reading", "air", "days", "extremes", "pollen" or "forecast", + `names` the readings asked about and `sides` where (indoor, outdoor), on `today` (local). None when the result can't be used + (the model then takes the question).""" + data = _load(result) + answers = {"reading": lambda: "\n".join(reading_lines(data, names, sides)), + "air": lambda: "\n".join(air_lines(data, names or None)), + "days": lambda: _days(data), + "extremes": lambda: _extremes(data, names), + "pollen": lambda: _alone("pollen_asthma", data, today), + "forecast": lambda: _alone("weather_forecast", data, today)} + return (answers[kind]() or None) if data and kind in answers else None # ---------- chart captions ---------- @@ -235,11 +252,11 @@ def _period(data: dict) -> str: def _at(entry: dict, which: str) -> str: """"high 24.3 °C, Tue 29 Sep around 3pm" for one record of a series.""" when = _year(f"{entry.get(f'{which}_date', '')} {entry.get(f'{which}_when', '')}".strip()) - return f"{entry[which]} {_units(entry.get('unit')) or ''}".strip() + (f", {when}" if when else "") + return f"{entry[which]} {unit_text(entry.get('unit')) or ''}".strip() + (f", {when}" if when else "") def _range(label: str, entry: dict, average: bool) -> str: - unit = _units(entry.get("unit")) or "" + unit = unit_text(entry.get("unit")) or "" if average and entry.get("average"): return f"{label}: average {entry['average']} {unit} (low {entry['low']}, high {entry['high']})".replace(" )", ")") return f"{label}: low {_at(entry, 'low')} · high {_at(entry, 'high')}" @@ -251,13 +268,14 @@ def _rain_line(turn, tz) -> list[str]: if not bars: return [] def day(i: int) -> str: - return datetime.fromtimestamp(bars.x[i], tz).strftime("%a %-d %b" + (" %-I%p" if bars.width < 86400 else "")).replace("AM", "am").replace("PM", "pm") + start = to_local(bars.x[i], tz) + return day_text(start, year=False) + (f" {clock_text(start, whole_hours=True)}" if bars.width < 86400 else "") lo, hi = min(range(len(bars.y)), key=bars.y.__getitem__), max(range(len(bars.y)), key=bars.y.__getitem__) per = f" per {bars.per}" if bars.per else "" return [f"☔ Rain{per}: least {bars.y[lo]:g} mm ({day(lo)}) · most {bars.y[hi]:g} mm ({day(hi)}) · {sum(bars.y):.1f} mm in all"] -def _weather_caption(result: dict, args: dict, turn, tz) -> list[str]: +def _weather_caption(result: dict, turn, tz) -> list[str]: series = result.get("series") or {} names = [n for n in dict.fromkeys(turn.chart_fields or [turn.chart_field or "temperature"]) if find(n)] lines: list[str] = [] @@ -280,13 +298,13 @@ def _weather_caption(result: dict, args: dict, turn, tz) -> list[str]: return lines -def _air_caption(result: dict, args: dict, turn) -> list[str]: +def _air_caption(result: dict, args: dict) -> list[str]: lines = [] - for key, label in AIR_ROWS: + for key, label in ((name, m.label) for name, m in AIR.items()): entry = result.get(key) if key not in (args.get("metrics") or ["pm2_5"]) or not isinstance(entry, dict): continue - unit = _units(entry.get("unit")) or "" + unit = unit_text(entry.get("unit")) or "" rating = entry.get("high_rating") lines.append(f"• {label}: peak {entry['high']} {unit} ({_year(entry.get('high_time'))})".replace(" )", ")") + (f" · {rating}" if rating else "") + f" · average {entry['average']}") @@ -299,8 +317,7 @@ def chart_caption(tool: str, args: dict, result: str | None, turn, tz) -> str | data = _load(result) if not data or not turn.charts: return None - lines = (_air_caption(data, args, turn) if tool == "air_quality" else - _weather_caption(data, args, turn, tz)) + lines = _air_caption(data, args) if tool == "air_quality" else _weather_caption(data, turn, tz) if not lines: return None notes = (["(Older days are still downloading, so they are left out.)"] if data.get("note_missing") else diff --git a/lib/series.py b/lib/series.py index cf2b90a..5f24274 100644 --- a/lib/series.py +++ b/lib/series.py @@ -1,6 +1,6 @@ -"""The weather readings a chart can plot, in one place: what each is called, where it lives in Ecowitt's data and the -words that name it. Adding a reading here makes it plottable, chartable side by side with others, and recognised in -a question.""" +"""The weather readings a chart can plot and the air-quality metrics, in one place: what each is called, where a reading lives +in Ecowitt's data, the words that name it, and whether its lowest is worth telling. Adding a reading here makes it plottable, +chartable side by side with others, and recognised in a question.""" from typing import NamedTuple @@ -15,6 +15,7 @@ class Reading(NamedTuple): unit: str words: str # a pattern for the words that name it in a question band_field: str | None = None # a field its line is shaded up to (wind: the gusts); it keeps that band at every width + low: bool = True # its lowest is worth telling: not when it is nothing every night, calm or dry spell # A rain chart puts the rain behind the first of these readings that is on it (rain tracks humidity and pressure more than @@ -25,15 +26,40 @@ class Reading(NamedTuple): "temperature": Reading("outdoor", "temperature", "Temperature", "°C", TEMP_WORDS), "humidity": Reading("outdoor", "humidity", "Humidity", "%", r"humid\w*"), "pressure": Reading("pressure", "relative", "Pressure", "hPa", PRESSURE_WORDS), - "wind": Reading("wind", "wind_speed", "Wind", "km/h", r"gusts?|wind\w*", "wind_gust"), - "rain": Reading("rainfall", "daily", "Rain", "mm", r"rain\w*|precip\w*"), + "wind": Reading("wind", "wind_speed", "Wind", "km/h", r"gusts?|wind\w*", "wind_gust", low=False), + "rain": Reading("rainfall", "daily", "Rain", "mm", r"rain\w*|precip\w*", low=False), "dew_point": Reading("outdoor", "dew_point", "Dew point", "°C", r"dew\w*"), "feels_like": Reading("outdoor", "feels_like", "Feels like", "°C", r"feel\w*|apparent"), "vpd": Reading("outdoor", "vpd", "Vapour pressure deficit", "kPa", r"vpd|vapou?r pressure deficit"), - "solar": Reading("solar_and_uvi", "solar", "Solar radiation", "W/m²", r"solar\w*|radiation|sunshine|sun(?!\s+\d)"), - "uv": Reading("solar_and_uvi", "uvi", "UV index", "", r"uvi?|ultraviolet"), + "solar": Reading("solar_and_uvi", "solar", "Solar radiation", "W/m²", r"solar\w*|radiation|sunshine|sun(?!\s+\d)", low=False), + "uv": Reading("solar_and_uvi", "uvi", "UV index", "", r"uvi?|ultraviolet", low=False), } SPECIFIC = ("dew_point", "feels_like", "vpd") # kinds of temperature and humidity, named on their own: they outrank "temperature" in a question +INDOOR_READINGS = ("temperature", "humidity", "dew_point", "feels_like") # what the indoor sensor measures as well +READING_GROUPS = tuple(dict.fromkeys(["outdoor", "indoor", *(r.group for r in WEATHER.values())])) # the groups they are in, the indoor sensor's too + + + + +class Metric(NamedTuple): + label: str + unit: str # on a chart's axis ("" for an index) + words: str # a pattern for the words that name it in a question + limits: tuple[float, float] # good up to the first, poor up to the second, very poor above + low: bool = False # its lowest is worth telling: the others are often 0 or at their baseline + + +# The air sensor's metrics, in the order a question's words are tried (PM10 before PM1). The ratings: the particles follow the +# US AQI (very poor = AQI 151+, the level of the mask alerts), PM1 has no standard so it uses PM2.5's, and CO2, VOC and NOx +# follow AirGradient's colour scales. +AIR = { + "pm2_5": Metric("PM2.5", "µg/m³", r"pm ?2\.?5|pm25", (9.0, 55.4)), + "pm10": Metric("PM10", "µg/m³", r"pm ?10", (54.0, 254.0)), + "pm1": Metric("PM1", "µg/m³", r"pm ?1(?![0-9])", (9.0, 55.4)), + "co2": Metric("CO₂", "ppm", r"co2|co₂", (799.0, 1499.0), low=True), # an outdoor floor near 400 ppm: its lowest says something + "voc_index": Metric("VOC index", "", r"voc\w*", (150.0, 250.0)), + "nox_index": Metric("NOx index", "", r"nox", (20.0, 150.0)), +} def find_name(name: str) -> str | None: diff --git a/lib/snapshots.py b/lib/snapshots.py index 8b9219d..a4d2ca1 100644 --- a/lib/snapshots.py +++ b/lib/snapshots.py @@ -13,23 +13,10 @@ class Snapshots: - """Use Snapshots.open(path): the pollen and forecast sources share one connection to the file (closed by the last user).""" - - _shared: dict = {} - _guard = threading.Lock() - - @classmethod - def open(cls, path) -> "Snapshots": - with cls._guard: - store = cls._shared.get(str(path)) - if store is None: - store = cls._shared[str(path)] = cls(path) - else: - store._users += 1 - return store + """One source's connection to the file: the pollen and forecast sources each open their own (the file is in WAL mode, + so two connections take turns to write) and close it when they stop.""" def __init__(self, path): - self.path, self._users = str(path), 1 self._lock = threading.Lock() self.db = sqlite3.connect(path, check_same_thread=False) self.db.execute("PRAGMA journal_mode=WAL") @@ -42,11 +29,6 @@ def __init__(self, path): self.db.commit() def close(self): - with self._guard: - self._users -= 1 - if self._users > 0: - return - self._shared.pop(self.path, None) with self._lock: self.db.close() @@ -72,10 +54,6 @@ def latest(self, kind: str) -> tuple[float, dict] | None: row = self.db.execute("SELECT last_seen, payload FROM snapshots WHERE kind=? ORDER BY id DESC LIMIT 1", (kind,)).fetchone() return (row[0], json.loads(row[1])) if row else None - def count(self, kind: str) -> int: - with self._lock: - return self.db.execute("SELECT COUNT(*) FROM snapshots WHERE kind=?", (kind,)).fetchone()[0] - def get(self, key: str) -> str | None: with self._lock: row = self.db.execute("SELECT value FROM meta WHERE key=?", (key,)).fetchone() diff --git a/lib/sources.py b/lib/sources.py new file mode 100644 index 0000000..3c37226 --- /dev/null +++ b/lib/sources.py @@ -0,0 +1,47 @@ +"""What a data source is to the bot: a name, its tools, a local clock and a Warmer that keeps it fresh; started once, poked +when a question wants it, closed at the end. The bot (bot.py) and the startup (ecobot.py) use nothing else of the four sources: +the weather station (lib/ecowitt), the air sensor (lib/airgradient), the pollen page (lib/pollen) and the forecast (lib/forecast).""" + +from abc import ABC, abstractmethod +from datetime import datetime, tzinfo + +from .timeutil import now_local +from .tools import Tool +from .warm import Warmer + + +class Source(ABC): + """Each source sets `name`, `tz`, `tools` and `warmer` (a Warmer of its own `warm`). close() closes its `client` and its + `store` on disk; the weather station, which has an API client and a history cache instead, closes its own.""" + name: str + tz: tzinfo + tools: list[Tool] + warmer: Warmer + + @abstractmethod + async def start(self): + """Get ready: find the device, read the first page. A failure that leaves the source of no use raises (the bot then runs + without it); one that only means "not yet" is logged, and the next refresh tries again.""" + + @abstractmethod + def describe(self) -> str: + """The source in a line for the prompt, with the words people use for it.""" + + @abstractmethod + def wants(self, text: str) -> bool: + """Should a question start refreshing this source?""" + + @abstractmethod + async def warm(self, fresh: bool = True) -> str: + """Fetch what questions usually need, if it is due (`fresh`: a timed refresh, not a question's); says what it did.""" + + def now(self) -> datetime: + return now_local(self.tz) + + def poke(self): + """A question that wants this source arrived: refresh it in the background (Warmer.poke).""" + self.warmer.poke() + + async def close(self): + await self.client.aclose() + self.store.close() diff --git a/lib/specs.py b/lib/specs.py index 5a729ba..13a9d43 100644 --- a/lib/specs.py +++ b/lib/specs.py @@ -9,12 +9,13 @@ built, not when it is drawn.""" from dataclasses import dataclass, field -from datetime import date +from datetime import date, tzinfo -from .series import RAIN_WITH +from .series import RAIN_WITH, WEATHER +from .timeutil import day_bounds, day_text -NO_LOW = ("solar", "uv", "wind", "rain") # readings whose low is always nothing (night, calm, dry): only the high is tagged +NO_LOW = tuple(name for name, r in WEATHER.items() if not r.low) # readings whose low is nothing every night or calm hour: only the high is tagged @dataclass @@ -59,6 +60,13 @@ def __post_init__(self): raise ValueError(f"bars {self.label!r}: x and y differ in length") +def calendar_bars(label: str, unit: str, figures: dict[date, float], by: str, tz: tzinfo) -> Bars: + """A bar per calendar month or year (`by`) from {its first day: figure}, the figure written above it. A bar is as wide as + the shortest month or year.""" + return Bars(label, unit, [day_bounds(d, tz)[0] for d in figures], [float(v) for v in figures.values()], + 28 * 86400 if by == "month" else 365 * 86400, by, values=True) + + @dataclass class Shares: label: str @@ -86,13 +94,18 @@ class Panel: peaks: str = "pills" # how peaks are labelled: "pills" on the lines, "aside" in empty space, "none" def __post_init__(self): - if not (self.lines or self.bars or self.shares): + if not (self.lines or self.bars and self.bars.x or self.shares and self.shares.x): raise ValueError(f"panel {self.label!r}: nothing to draw") if self.shares and (self.lines or self.bars): raise ValueError(f"panel {self.label!r}: shares draw alone") if self.peaks not in ("pills", "aside", "none"): raise ValueError(f"panel {self.label!r}: peaks are 'pills', 'aside' or 'none'") + @property + def bars_only(self) -> bool: + """Bars and no lines: rain on its own, or a count or figure per month or year.""" + return bool(self.bars) and not self.lines + @property def xs(self) -> list[int]: """Every x the panel draws, bars to the end of the last one.""" @@ -117,11 +130,17 @@ class Chart: subtitle: str panels: list[Panel] compass: Compass | None = None + period: tuple[date, date] | None = None # the first and last day it covers, for its caption + + @property + def single(self) -> bool: + """One panel of lines: drawn big, with the room for a compass beside it.""" + return len(self.panels) == 1 and bool(self.panels[0].lines) def __post_init__(self): if not self.panels: raise ValueError("a chart needs a panel") - if self.compass and not (len(self.panels) == 1 and self.panels[0].lines): + if self.compass and not self.single: raise ValueError("a compass goes beside a single line panel") for panel in self.panels: # every line is labelled with its high and low, so each carries its records for line in panel.lines: @@ -129,37 +148,26 @@ def __post_init__(self): if panel.reading in NO_LOW: line.records.pop("low", None) - -def _caption(self, today: date) -> str: - """The chart's caption. One panel: one line, "Temperature, since Tue 7 Jul 2026" (a period that runs up to today), "Temperature, - Wed 24 – Tue 30 Sep 2026" otherwise. Several panels: the title, then the time range on a line of its own.""" - period = self.subtitle.split(" · ")[0].strip() # always starts with the time range - if not period: - return self.title - if len(self.panels) != 1: - return f"{self.title}\n{period}" - day = lambda d: f"{d:%a} {d.day} {d:%b %Y}" - left, _, right = period.partition(" – ") - if not right: # one day - return f"{self.title}, {'today' if period == day(today) else period}" - if right != day(today): - return f"{self.title}, {period}" - first, last = left.split(), right.split() # "Wed 24" / "Thu 27 Aug" take their month and year from the end of the range - return f"{self.title}, since {' '.join([*first, *last[len(first):]])}" - - -Chart.caption = _caption + def caption(self, today: date) -> str: + """The chart's caption. One panel: one line, "Temperature, since Tue 7 Jul 2026" (a period that runs up to today), + "Temperature, Wed 24 – Tue 30 Sep 2026" otherwise. Several panels: the title, then the period on a line of its own.""" + if not self.period: + return self.title + first, last = self.period + if len(self.panels) != 1: + return f"{self.title}\n{period_text(first, last)}" + if first == last: + return f"{self.title}, {'today' if first == today else day_text(first)}" + return f"{self.title}, since {day_text(first)}" if last == today else f"{self.title}, {period_text(first, last)}" def period_text(a: date, b: date) -> str: """'Tue 30 Sep 2026', 'Wed 24 – Tue 30 Sep 2026', 'Thu 27 Aug – Tue 30 Sep 2026'; the year goes on both dates when it differs.""" if a == b: - return f"{a:%a} {a.day} {a:%b %Y}" + return day_text(a) if (a.year, a.month) == (b.year, b.month): - return f"{a:%a} {a.day} – {b:%a} {b.day} {b:%b %Y}" - if a.year == b.year: - return f"{a:%a} {a.day} {a:%b} – {b:%a} {b.day} {b:%b %Y}" - return f"{a:%a} {a.day} {a:%b %Y} – {b:%a} {b.day} {b:%b %Y}" + return f"{a:%a} {a.day} – {day_text(b)}" + return f"{day_text(a, year=a.year != b.year)} – {day_text(b)}" def rain_behind(panels: list[Panel]) -> list[Panel]: @@ -168,7 +176,7 @@ def rain_behind(panels: list[Panel]) -> list[Panel]: stays a panel of its own.""" lines = [p for p in panels if p.lines and not p.bars] line_panel = next((p for name in RAIN_WITH for p in lines if p.reading == name), lines[0] if lines else None) - bars_panel = next((p for p in panels if p.bars and not p.lines), None) + bars_panel = next((p for p in panels if p.bars_only), None) if not (line_panel and bars_panel): return panels line_panel.bars = bars_panel.bars @@ -181,4 +189,5 @@ def stack(panels: list[Panel], first: date, last: date) -> Chart: per = next((f"{'rain' if b is p.bars else 'rating'} per {b.per}" for p in panels for b in (p.bars, p.shares) if b and b.per), "") shaded = any(s.low is not None for p in panels for s in p.lines) return Chart(", ".join(p.label for p in panels), - " · ".join(filter(None, [period_text(first, last), per, "range shaded" if shaded else ""])), rain_behind(panels)) + " · ".join(filter(None, [period_text(first, last), per, "range shaded" if shaded else ""])), rain_behind(panels), + period=(first, last)) diff --git a/lib/templates.py b/lib/templates.py index 784f7a7..51cbee1 100644 --- a/lib/templates.py +++ b/lib/templates.py @@ -2,6 +2,7 @@ message, so the answer comes through the usual path; the label stands for a plain sentence (its words without the emoji).""" import hashlib +from collections.abc import Collection from telegram import ReplyKeyboardMarkup @@ -15,7 +16,6 @@ [("\U0001f3ed Air Quality", "Air quality all metrics 1d"), ("\U0001f52c Particulates", "Particulates chart 1d"), ("\U0001f4a8 Wind", "Wind chart 1d")]] SENTENCES = dict(button for row in ROWS for button in row) -LABELS = set(SENTENCES) # Telegram keeps the keyboard in each app until a message brings a new one; this changes whenever the buttons do, so a chat # whose stored value differs gets the current keyboard with its next reply. HIDDEN is what a chat that hid the buttons stores. HIDDEN = "hidden" @@ -30,9 +30,6 @@ def version(irrigation: bool = False) -> str: return hashlib.md5("|".join(label for row in rows(irrigation) for label, _ in row).encode()).hexdigest()[:8] -VERSION = version() - - def keyboard(irrigation: bool = False) -> ReplyKeyboardMarkup: return ReplyKeyboardMarkup([[label for label, _ in row] for row in rows(irrigation)], resize_keyboard=True, is_persistent=True, input_field_placeholder="Ask about the weather or air…") @@ -43,16 +40,16 @@ def sentence(text: str) -> str | None: return SENTENCES.get(text.strip()) -WEATHER_METRICS = ["Temperature, humidity (indoor, outdoor)", "Dew point, vapour pressure deficit, pressure, wind speed", - "Rain, solar radiation, UV index"] -AIR_METRICS = ["PM1, PM2.5, PM10, CO₂, VOC, NOx"] -POLLEN = ["Pollen and thunderstorm asthma risk (October to December)"] -FORECAST = ["Forecast"] -HISTORY = ["History charts"] +MEASURES = { # a source (by name) -> what it measures, as the Capabilities button lists it + "Ecowitt": ["Temperature, humidity (indoor, outdoor)", "Dew point, vapour pressure deficit, pressure, wind speed", + "Rain, solar radiation, UV index"], + "AirGradient": ["PM1, PM2.5, PM10, CO₂, VOC, NOx"], + "Pollen": ["Pollen and thunderstorm asthma risk (October to December)"], + "Forecast": ["Forecast"], +} -def capabilities_text(weather: bool = True, air: bool = True, pollen: bool = False, forecast: bool = False) -> str: - """What the bot measures, as a short bulleted list (no dates, no detail): the answer to the Capabilities button.""" - lines = (WEATHER_METRICS if weather else []) + (AIR_METRICS if air else []) + (POLLEN if pollen else []) + ( - FORECAST if forecast else []) + HISTORY +def capabilities_text(sources: Collection[str] = ("Ecowitt", "AirGradient")) -> str: + """What the bot measures with these sources, as a short bulleted list (no dates, no detail): the answer to the Capabilities button.""" + lines = [line for name, measured in MEASURES.items() if name in sources for line in measured] + ["History charts"] return "\n".join(f"\u2022 {line}" for line in lines) diff --git a/lib/timeutil.py b/lib/timeutil.py index 671a33f..cb138da 100644 --- a/lib/timeutil.py +++ b/lib/timeutil.py @@ -1,10 +1,6 @@ """Local-time helpers shared by the sources: naive local datetimes, epochs and day boundaries.""" -import math -from bisect import bisect_left, bisect_right from datetime import date, datetime, time, timedelta, tzinfo, UTC -from itertools import accumulate -from statistics import fmean def now_local(tz: tzinfo) -> datetime: @@ -17,40 +13,66 @@ def to_local(ts: int, tz: tzinfo) -> datetime: return datetime.fromtimestamp(ts, UTC).astimezone(tz) +def day_text(day: date | datetime, year: bool = True) -> str: + """"Sat 3 Oct 2026", or "Sat 3 Oct" without the year: how the bot writes a day, everywhere.""" + return f"{day:%a} {day.day} {day:%b}" + (f" {day:%Y}" if year else "") + + +def clock_text(t: datetime, whole_hours: bool = False) -> str: + """"3:45pm", "12:05am": how the bot writes a time of day; whole_hours writes one on the hour as "10am".""" + hour, half = t.hour % 12 or 12, "am" if t.hour < 12 else "pm" + return f"{hour}{half}" if whole_hours and not t.minute else f"{hour}:{t:%M}{half}" + + def when(ts: int, tz: tzinfo) -> str: """An epoch as "Sun 11 Oct 2026 11:45pm" in tz (the form the tools put in their "time" field).""" dt = to_local(ts, tz) - return f"{dt:%a} {dt.day} {dt:%b %Y} {dt:%I:%M%p}".replace(" 0", " ").replace("AM", "am").replace("PM", "pm") + return f"{day_text(dt)} {clock_text(dt)}" def local_date(ts: int, tz: tzinfo) -> date: return to_local(ts, tz).date() +def naive_local(ts: float, tz: tzinfo) -> datetime: + """An epoch as a naive local datetime. It keeps `fold`, so local_epoch gives the epoch back exactly, even in the hour that + happens twice when the clocks go back.""" + return to_local(ts, tz).replace(tzinfo=None) + + +def parse_local(text, tz: tzinfo) -> datetime: + """A date or time as sent ("2026-10-01", "2026-10-01 14:00", "2026-10-01T03:00:00Z") as a naive local time: one with an offset + is moved into tz first. ValueError for anything else.""" + dt = datetime.fromisoformat(str(text)) + return dt.astimezone(tz).replace(tzinfo=None) if dt.tzinfo else dt + + +def seconds_until(hour: int, minute: int, tz: tzinfo, now: datetime | None = None) -> float: + """Real seconds from `now` (naive local; default the present) to the next `hour`:`minute` local time. Two clock times of one zone + are an hour more or less apart than they look across a daylight-saving change; this counts the real seconds.""" + now = now or now_local(tz) + target = now.replace(hour=hour, minute=minute, second=0, microsecond=0) + if target <= now: + target += timedelta(days=1) + return target.replace(tzinfo=tz).timestamp() - now.replace(tzinfo=tz).timestamp() + + def local_epoch(local: datetime, tz: tzinfo) -> int: """A naive local datetime as an epoch.""" return int(local.replace(tzinfo=tz).timestamp()) +def at_hour(day: date, hour: int, tz: tzinfo) -> int: + """The epoch of `hour` o'clock local time on `day`.""" + return local_epoch(datetime.combine(day, time(hour)), tz) + + def day_bounds(day: date, tz: tzinfo, last_second: bool = False) -> tuple[int, int]: """Epoch range of one local day: [start, next midnight), or up to its last second if last_second.""" start = datetime.combine(day, time()).replace(tzinfo=tz) return int(start.timestamp()), int((start + timedelta(days=1)).timestamp()) - last_second -def rolling_mean(values: dict[int, float], half_window: int) -> dict[int, float]: - """Each reading replaced by the mean of the readings within half_window seconds either side of it (itself included): - a light smoothing of the drawn line. Timestamps are unchanged, nothing is interpolated, and a gap only means fewer - readings in the window. A reading that is not a finite number is left out.""" - ts = sorted(t for t, v in values.items() if math.isfinite(v)) - prefix = [0.0, *accumulate(values[t] for t in ts)] - out: dict[int, float] = {} - for t in ts: - lo, hi = bisect_left(ts, t - half_window), bisect_right(ts, t + half_window) - out[t] = (prefix[hi] - prefix[lo]) / (hi - lo) - return out - - def parse_period(args: dict, today: date, default_days: int) -> tuple[date, date] | str: """(first, last) day from start_date / end_date, or an error text. The last day is at most yesterday: today is still settling. A missing start is default_days before the end.""" @@ -62,62 +84,11 @@ def parse_period(args: dict, today: date, default_days: int) -> tuple[date, date return (first, last) if first <= last else "start_date must be before end_date (today isn't final, so the latest day is yesterday)" -POINT_BUDGET = 500 # about as many points as a chart can show legibly -WARN_POINTS = 2 * POINT_BUDGET # a question that sends the model more than twice a 500-point query's data gets a heads-up -TOKENS_PER_POINT = 10 # a dated low/high row is about 10 tokens as the model reads it -CHARS_PER_TOKEN = 4 -WARN_TOKENS = WARN_POINTS * TOKENS_PER_POINT # 10,000 tokens: about $0.0125 of input at Grok 4.3's $1.25 per million, and it rides on every later step +def period_parameters(default_days: int) -> dict: + """The start_date and end_date in a tool's parameters, for a period parse_period reads.""" + return {"start_date": {"type": "string", "description": f"First day, 'YYYY-MM-DD'. Default: {default_days} days before the end."}, + "end_date": {"type": "string", "description": "Last day, 'YYYY-MM-DD' (up to yesterday). Default: yesterday."}} + + SLOT = 1800 # the 30-minute slots both devices are lined up on MIN_DAY_SLOTS = 24 # a day counts for a daily comparison with at least 12 hours of readings -WIDTHS = (300, SLOT, 3600, 7200, 14400, 86400) # bucket sizes a chart line can be drawn at: 5 min ... a day -WIDTH_NAMES = {300: "5-minute", 1800: "30-minute", 3600: "hourly", 7200: "2-hour", 14400: "4-hour", 86400: "daily"} - - -def bucket_width(span_seconds: float, source_seconds: float, budget: int = POINT_BUDGET) -> int: - """The finest bucket (never finer than the readings themselves) that keeps a line of this length within the - point budget; a day is the widest.""" - return next((w for w in WIDTHS if w >= source_seconds and span_seconds / w <= budget), WIDTHS[-1]) - - -def _summarise(points, key) -> dict: - """{bucket: (mean, lowest low, highest high)} from (ts, value, low, high, fine) readings. The mean takes one value - per 30-minute slot, so a stretch held as 5-minute readings (fine) counts no more than the same stretch held as - 30-minute ones: a slot's fine readings win where it has them.""" - slots: dict = {} - extremes: dict = {} - for ts, value, low, high, fine in points: - k = key(ts) - slots.setdefault((k, ts // SLOT), {}).setdefault(fine, []).append(value) - lo, hi = extremes.get(k, (low, high)) - extremes[k] = (min(lo, low), max(hi, high)) - means: dict = {} - for (k, _), got in slots.items(): - values = got.get(True) or got[False] - means.setdefault(k, []).append(fmean(values)) - return {k: (fmean(v), *extremes[k]) for k, v in means.items()} - - -def daily_summary(points, tz: tzinfo) -> dict[date, tuple[float, float, float]]: - """{local day: (mean, lowest low, highest high)}; see _summarise.""" - return _summarise(points, lambda ts: local_date(ts, tz)) - - -def bucket_summary(points, width: int) -> dict[int, tuple[float, float, float]]: - """{bucket start epoch: (mean, lowest low, highest high)} for buckets of `width` seconds; see _summarise.""" - return _summarise(points, lambda ts: ts // width * width) - - -def bucketed(points, tz: tzinfo, width: int) -> tuple[list[int], list[float], list[float], list[float]]: - """(x, mean, low, high) lists for a chart line in buckets of this width: local days (drawn at noon) for a day, - otherwise epoch-aligned buckets (drawn at their middle).""" - if width >= 86400: - days = daily_summary(points, tz) - keys = sorted(days) - xs = [local_epoch(datetime.combine(d, time(12)), tz) for d in keys] - rows = [days[d] for d in keys] - else: - buckets = bucket_summary(points, width) - keys = sorted(buckets) - xs = [k + width // 2 for k in keys] - rows = [buckets[k] for k in keys] - return xs, [r[0] for r in rows], [r[1] for r in rows], [r[2] for r in rows] diff --git a/lib/tools.py b/lib/tools.py index b69f719..0b0dcee 100644 --- a/lib/tools.py +++ b/lib/tools.py @@ -11,7 +11,11 @@ log = logging.getLogger(__name__) MAX_OUTPUT_CHARS = 60000 -TOOL_TIMEOUT = 90 + + +def json_result(out) -> str: + """A tool's result as the model is sent it: compact JSON, with °, ² and emoji written as they are, not escaped.""" + return json.dumps(out, ensure_ascii=False, separators=(",", ":")) @dataclass @@ -30,6 +34,12 @@ class Turn: times: dict = field(default_factory=dict) # seconds spent by step ("data", "draw", "png"), for the reply's timing line in the log ahead: bool = False # a chart drawn ahead, nobody waiting: its requests wait their turn and its routine log lines are quiet + @classmethod + def of(cls, read, text: str, **more) -> "Turn": + """The Turn for a question as intent.read read it (`read`): what its words ask of a chart and of the figures.""" + return cls(chart_asked=read.chart_asked, chart_field=read.chart_field, chart_fields=read.chart_fields, + average_asked=read.average_asked, readings=read.readings, per_day=read.per_day, text=text, **more) + @dataclass class Tool: @@ -48,10 +58,6 @@ def __init__(self, tools: list[Tool]): self.by_name = {t.name: t for t in tools} self.schemas = [t.schema() for t in tools] - def schemas_for(self, names=None) -> list[dict]: - """The tool definitions to send: those named (that exist), or all when no names are given.""" - return self.schemas if names is None else [self.by_name[n].schema() for n in names if n in self.by_name] - async def call(self, name: str, raw_args: str, turn: Turn | None = None) -> str: """Run a tool for the model; failures come back as text the model can explain.""" tool = self.by_name.get(name) @@ -63,10 +69,8 @@ async def call(self, name: str, raw_args: str, turn: Turn | None = None) -> str: return f"Error: tool arguments were not valid JSON ({e})" level = logging.DEBUG if turn and turn.ahead else logging.INFO log.log(level, "Tool call %s %s", name, raw_args, extra=QUOTES) - try: - out = await asyncio.wait_for(tool.handler(args, turn or Turn()), TOOL_TIMEOUT) - except TimeoutError: - return f"Error: tool timed out after {TOOL_TIMEOUT}s" + try: # no time limit of its own: the question's (bot.TURN_SECONDS) runs from before the call + out = await tool.handler(args, turn or Turn()) except Exception as e: log.exception("Tool %s failed", name) return f"Error calling tool: {e}" @@ -78,3 +82,7 @@ async def call(self, name: str, raw_args: str, turn: Turn | None = None) -> str: if turn: turn.result_chars += len(out) return out + + async def call_all(self, calls: list[tuple[str, dict]], turn: Turn | None = None) -> list[str]: + """The bot's own calls (name, arguments), made together; their results in the same order.""" + return await asyncio.gather(*(self.call(name, json.dumps(args), turn) for name, args in calls)) diff --git a/lib/warm.py b/lib/warm.py index 808831d..48014c0 100644 --- a/lib/warm.py +++ b/lib/warm.py @@ -23,21 +23,23 @@ def in_sync_hours(now: datetime) -> bool: return SYNC_HOURS[0] <= now.hour < SYNC_HOURS[1] +# Errors that point at the code, logged with their traceback; any other failure (an outage, a refusal) is one line +BUGS = (AssertionError, AttributeError, IndexError, KeyError, NameError, TypeError, ValueError, ZeroDivisionError) + + async def safely(fn: Callable[..., Awaitable], *args): """Await fn(*args); log a failure instead of raising (background work must not die).""" try: return await fn(*args) - except asyncio.CancelledError: - raise except Exception as e: - log.warning("%s failed: %s: %s", getattr(fn, "__qualname__", fn), type(e).__name__, e) + log.warning("%s failed: %s: %s", getattr(fn, "__qualname__", fn), type(e).__name__, e, exc_info=isinstance(e, BUGS)) -async def every(seconds: float, fn: Callable[[], Awaitable]): - """Call fn now, then every `seconds`, until cancelled.""" +async def every(delay: float | Callable[[], float], fn: Callable[[], Awaitable]): + """Call fn now, then again after `delay` seconds (or as many as delay() says each time), until cancelled.""" while True: await safely(fn) - await asyncio.sleep(seconds) + await asyncio.sleep(delay() if callable(delay) else delay) class Warmer: @@ -45,16 +47,27 @@ def __init__(self, refresh: Callable[[bool], Awaitable], interval: float = REFRE self.refresh, self.interval = refresh, interval self.after: list[Callable[[], Awaitable]] = [] # run after each timed refresh (alert checks) self._task: asyncio.Task | None = None + self._lock = asyncio.Lock() # one refresh at a time: a timed one waits for a question's + + async def _refresh(self, fresh: bool): + async with self._lock: + await safely(self.refresh, fresh) + + async def stop(self): + """Cancel a refresh a question started, if one is still running (the timed ones end with their task).""" + if self._task is not None: + self._task.cancel() + await asyncio.gather(self._task, return_exceptions=True) def poke(self): - """A question arrived: refresh in the background unless one is already running.""" - if self._task is None or self._task.done(): - self._task = asyncio.create_task(safely(self.refresh, False)) + """A question arrived: refresh in the background, unless a refresh (a question's or a timed one) is running.""" + if not self._lock.locked() and (self._task is None or self._task.done()): + self._task = asyncio.create_task(self._refresh(False)) async def run(self): """Refresh every `interval` seconds until cancelled (startup already fetched once).""" while True: await asyncio.sleep(self.interval) - await safely(self.refresh, True) + await self._refresh(True) for check in self.after: await safely(check) diff --git a/scripts/cache_status.py b/scripts/cache_status.py index 453fd36..9f51d30 100644 --- a/scripts/cache_status.py +++ b/scripts/cache_status.py @@ -15,7 +15,7 @@ from _common import parser # noqa: E402 (also puts the repo on sys.path) from lib.config import data_dir # noqa: E402 -from lib.ecowitt.api import GROUPS, RETENTION # noqa: E402 +from lib.ecowitt.api import CYCLES, GROUPS, kept_from # noqa: E402 from lib.ecowitt.store import horizon, subtract # noqa: E402 @@ -30,8 +30,8 @@ def ecowitt(path: Path): rows = db.execute("SELECT COUNT(*) FROM points p JOIN series s ON s.id = p.series WHERE s.mac=?", (mac,)).fetchone()[0] print(f"Ecowitt {mac}: {rows:,} readings, station data from {day(first) if first else 'n/a'}") now = time.time() - for cycle in ("5min", "30min", "4hour", "1day"): - start = max(now - (RETENTION[cycle] - 2) * 86400, (first or now) - 86400) + for cycle in CYCLES: + start = max(datetime.combine(kept_from(cycle, date.today()), datetime.min.time()).timestamp(), (first or now) - 86400) end = horizon(cycle) for group in GROUPS: cov = db.execute("SELECT start, end FROM coverage WHERE mac=? AND cycle=? AND grp=?", diff --git a/scripts/check_rain.py b/scripts/check_rain.py index c4e5e97..3975674 100644 --- a/scripts/check_rain.py +++ b/scripts/check_rain.py @@ -13,15 +13,13 @@ so days just over the 1 mm line can be missed. """ -import os import sqlite3 import sys from datetime import datetime, timedelta from pathlib import Path -from zoneinfo import ZoneInfo from _common import parser # noqa: E402 (also puts the repo on sys.path) -from lib.config import data_dir # noqa: E402 +from lib.config import data_dir, local_tz # noqa: E402 from lib.ecowitt import days as day_tool # noqa: E402 from lib.ecowitt.api import UNITS # noqa: E402 @@ -49,8 +47,7 @@ def totals(cache: HistoryCache, mac: str, tz, cycle: str, days: list) -> dict: lo, hi = day_bounds(min(days), tz, True)[0], day_bounds(max(days), tz, True)[1] if cycle == "1day": lo, hi = lo - 86400, hi + 86400 - store = day_tool.load_cached(cache, mac, cycle, ["rain"], lo, hi) - return day_tool.per_day(store, "rain", set(days), tz) + return day_tool.day_figures(cache, mac, cycle, ["rain"], set(days), tz, lo, hi)["rain"] def compare_rain(cache: HistoryCache, mac: str, tz, today, days_back: int = 88) -> dict: @@ -110,7 +107,7 @@ def main(): ap.add_argument("--days", type=int, default=88) ap.add_argument("--cache", type=Path, default=data_dir() / "ecowitt_cache.sqlite") args = ap.parse_args() - tz = ZoneInfo(os.getenv("TZ") or "Australia/Melbourne") + tz = local_tz() macs = [m for (m,) in sqlite3.connect(f"file:{args.cache}?mode=ro", uri=True).execute("SELECT DISTINCT mac FROM coverage")] if not macs: sys.exit(f"{args.cache} has no cached history") diff --git a/scripts/conditions.py b/scripts/conditions.py index 3b42df9..731e289 100644 --- a/scripts/conditions.py +++ b/scripts/conditions.py @@ -16,7 +16,7 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) -from lib.config import _tz # noqa: E402 +from lib.config import local_tz # noqa: E402 from lib.forecast import Forecast # noqa: E402 from lib.pollen import Pollen # noqa: E402 @@ -28,7 +28,7 @@ async def main(): ap.add_argument("--district", default="Central") ap.add_argument("--raw", action="store_true") args = ap.parse_args() - cfg = SimpleNamespace(tz=_tz(), pollen_district=args.district, forecast_lat=args.lat, forecast_lon=args.lon, + cfg = SimpleNamespace(tz=local_tz(), pollen_district=args.district, forecast_lat=args.lat, forecast_lon=args.lon, place=os.getenv("PLACE", "Melbourne"), conditions_cache_path=os.path.join(tempfile.mkdtemp(), "conditions.sqlite")) # a scratch cache: the bot's is left alone pollen = Pollen(cfg) diff --git a/scripts/dump_specs.py b/scripts/dump_specs.py index 4c10f32..76e9ab9 100644 --- a/scripts/dump_specs.py +++ b/scripts/dump_specs.py @@ -125,7 +125,8 @@ async def main(): db.close() cfg = Config(telegram_token="", xai_api_key="", xai_base_url="", xai_model="", tz=tz, ecowitt_api_key="offline", ecowitt_app_key="offline", airgradient_token="offline", airgradient_location=str(loc or "offline"), - airgradient_dashboard="", state_path=Path(tmp) / "state.json", cache_path=eco_path, air_cache_path=air_path) + airgradient_dashboard="", state_path=Path(tmp) / "state.json", cache_path=eco_path, air_cache_path=air_path, + conditions_cache_path=Path(tmp) / "conditions.sqlite") eco = air = None if have_eco: eco = Ecowitt(cfg, transport=Offline()) diff --git a/scripts/ecowitt_metrics.py b/scripts/ecowitt_metrics.py index a26c5fd..6926102 100644 --- a/scripts/ecowitt_metrics.py +++ b/scripts/ecowitt_metrics.py @@ -19,7 +19,7 @@ from _common import parser # noqa: E402 (also puts the repo on sys.path) -from lib.config import _tz # noqa: E402 +from lib.config import local_tz # noqa: E402 from lib.ecowitt.api import GROUPS, EcowittAPI, EcowittError # noqa: E402 # Data groups the API's documentation (real_time and history "call_back") lists. Channels are numbered. @@ -50,7 +50,7 @@ async def probe(api: EcowittAPI, mac: str, group: str, history: bool) -> tuple[s return "no data", [], [] fields: list[str] = [] if history: - now = datetime.now(_tz()) + now = datetime.now(local_tz()) try: hist = await api.history(mac, "30min", (now - timedelta(hours=6)).replace(tzinfo=None), now.replace(tzinfo=None), group) fields = sorted(name for name, _, _ in leaves(hist.get(group) or {})) diff --git a/scripts/report_inputs.py b/scripts/report_inputs.py index 1dcd6fb..d0a03a4 100644 --- a/scripts/report_inputs.py +++ b/scripts/report_inputs.py @@ -23,11 +23,11 @@ async def main(): parser(__doc__).parse_args() cfg = Config.from_env() - sources = await start_sources(cfg) + sources, cfg = await start_sources(cfg) try: tools = Tools([t for s in sources for t in s.tools]) names = {s.name for s in sources} - for name, args in intent.report_calls("Ecowitt" in names, "AirGradient" in names, "Pollen" in names, "Forecast" in names): + for name, args in intent.report_calls(names): print(f"=== {name} {json.dumps(args)}") print(await tools.call(name, json.dumps(args))) print() diff --git a/scripts/show_request.py b/scripts/show_request.py index 8ca76d7..155a542 100644 --- a/scripts/show_request.py +++ b/scripts/show_request.py @@ -12,16 +12,14 @@ import asyncio import json -import os import sqlite3 import sys from datetime import datetime, timedelta from pathlib import Path from types import SimpleNamespace as NS -from zoneinfo import ZoneInfo from _common import parser # noqa: E402 (also puts the repo on sys.path) -from lib.config import data_dir # noqa: E402 +from lib.config import data_dir, local_tz # noqa: E402 from lib import intent, prompt # noqa: E402 from lib.airgradient import source as air # noqa: E402 @@ -66,7 +64,7 @@ async def main(): ap.add_argument("--call", help="JSON tool call the stand-in model makes first") args = ap.parse_args() - tz = ZoneInfo(os.getenv("TZ") or "Australia/Melbourne") + tz = local_tz() now = datetime.now(tz) macs = [m for (m,) in sqlite3.connect(f"file:{args.cache}?mode=ro", uri=True).execute("SELECT DISTINCT mac FROM coverage")] if not macs: @@ -85,14 +83,14 @@ async def unavailable(a): Tool("air_quality", air.DESCRIPTION, air.PARAMETERS, unavailable)]) text = args.question r = intent.read(text, now.replace(tzinfo=None)) - system = prompt.build(now, ["Ecowitt weather station", "AirGradient outdoor air-quality sensor"], r.hints) + system = prompt.build(now, {"Ecowitt": "Ecowitt weather station", "AirGradient": "AirGradient outdoor air-quality sensor"}, r.hints) effort, fast = r.effort, r.fast call = json.loads(args.call) if args.call else default_call(now.date()) client = Recorder([call, "(the model's answer goes here)"]) messages = [{"role": "user", "content": text}] await Agent(client, "grok-4.3", tools).run(messages, system, effort, first_call=([fast[:2], *r.more] if r.more else fast[:2]) if fast else None, - require_tool=r.needs_data) + tool_choice=r.tool_choice) banner("DECISIONS MADE IN CODE") print(f"reasoning effort: {effort} fast path: {fast[2] if fast else 'no (the model chooses the tool)'} " diff --git a/tests/charts/air.png b/tests/charts/air.png index a164156..3c1b544 100644 Binary files a/tests/charts/air.png and b/tests/charts/air.png differ diff --git a/tests/charts/all_week.png b/tests/charts/all_week.png index 396a970..57712f9 100644 Binary files a/tests/charts/all_week.png and b/tests/charts/all_week.png differ diff --git a/tests/charts/humidity_rain.png b/tests/charts/humidity_rain.png index 3795a04..2b444f2 100644 Binary files a/tests/charts/humidity_rain.png and b/tests/charts/humidity_rain.png differ diff --git a/tests/charts/quarter.png b/tests/charts/quarter.png index 4b76313..721d9e8 100644 Binary files a/tests/charts/quarter.png and b/tests/charts/quarter.png differ diff --git a/tests/charts/temperature.png b/tests/charts/temperature.png index 09b6189..5cb8ff2 100644 Binary files a/tests/charts/temperature.png and b/tests/charts/temperature.png differ diff --git a/tests/charts/wind.png b/tests/charts/wind.png index f4c1e64..2d0fe81 100644 Binary files a/tests/charts/wind.png and b/tests/charts/wind.png differ diff --git a/tests/conftest.py b/tests/conftest.py index 2a8b1b1..75791df 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -8,6 +8,13 @@ import pytest +@pytest.fixture +def alert_state(tmp_path): + """The bot's chats and settings (an AlertState) in a file of the test's own: what a Bot is given, as in the running bot.""" + from lib.alerts import AlertState + return AlertState(tmp_path / "state.json") + + @pytest.fixture(scope="session") def archived_cache(tmp_path_factory): """A cache file holding the fake station's whole archive, built once. Tests copy it (archived_station).""" @@ -40,6 +47,7 @@ async def _inline(func, /, *args, **kwargs): def no_request_spacing(monkeypatch): """Also closes every SQLite connection a test opened, so none is left for the garbage collector (a ResourceWarning).""" import asyncio + import contextlib import sqlite3 monkeypatch.setattr(asyncio, "to_thread", _inline) real_connect = sqlite3.connect @@ -48,7 +56,8 @@ def no_request_spacing(monkeypatch): def fast_connect(*args, **kwargs): # tests don't need the caches to survive a power cut db = real_connect(*args, **kwargs) db.execute("PRAGMA synchronous=OFF") - db.execute("PRAGMA journal_mode=MEMORY") + with contextlib.suppress(sqlite3.OperationalError): # a file another connection holds in WAL mode stays in it + db.execute("PRAGMA journal_mode=MEMORY") opened.append(db) return db monkeypatch.setattr(sqlite3, "connect", fast_connect) diff --git a/tests/evalkit.py b/tests/evalkit.py index 5fdb060..405a74c 100644 --- a/tests/evalkit.py +++ b/tests/evalkit.py @@ -17,6 +17,7 @@ from lib.ecowitt import days, link, source as station from lib.forecast import source as forecast from lib.pollen import source as pollen +from tests.fakes import EVERY_SOURCE from lib.llm import Agent from lib.tools import Tool, Tools @@ -135,7 +136,7 @@ def deterministic(case: Case) -> list[str]: """Failures of the decisions made in code (no model): reasoning effort, fast path, hints, chart choices.""" e, fails = case.expect, [] text = case.ask - r = intent.read(text, NOW, pollen=True, forecast=True) + r = intent.read(text, NOW, EVERY_SOURCE) if "effort" in e and r.effort != e["effort"]: fails.append(f"effort {r.effort}, expected {e['effort']}") if "fast" in e: @@ -167,16 +168,16 @@ async def run_live(case: Case, client, model: str, effort: str | None = None) -> """Ask the real model; returns the tool calls it made and its final reply.""" calls: list[tuple[str, dict]] = [] text = case.ask - r = intent.read(text, NOW, pollen=True, forecast=True) - system = prompt.build(NOW, ["Ecowitt weather station", "AirGradient outdoor air-quality sensor", - "Melbourne pollen forecast and thunderstorm asthma risk (melbournepollen.com.au)", - "Weather forecast for the owner's location (Open-Meteo)"], - r.hints, r.about_the_bot, r.rain_caption, found := intent.topics(content(case), [m["content"] for m in case.history if m["role"] == "user"][-2:])) + r = intent.read(text, NOW, EVERY_SOURCE) + system = prompt.build(NOW, {"Ecowitt": "Ecowitt weather station", "AirGradient": "AirGradient outdoor air-quality sensor", + "Pollen": "Melbourne pollen forecast and thunderstorm asthma risk (melbournepollen.com.au)", + "Forecast": "Weather forecast for the owner's location (Open-Meteo)"}, + r.hints, r.about_the_bot, r.rain_caption, intent.topics(content(case), [m["content"] for m in case.history if m["role"] == "user"][-2:])) fast = r.fast messages = [*case.history, {"role": "user", "content": content(case)}] reply = await Agent(client, model, make_tools(calls)).run( - messages, system, effort or r.effort, first_call=[*([fast[:2], *r.more] if fast else []), *r.extra] or None, require_tool=r.needs_data, - no_tools=r.about_the_bot) + messages, system, effort or r.effort, first_call=[*([fast[:2], *r.more] if fast else []), *r.extra] or None, + tool_choice=r.tool_choice) if fast or r.extra: calls[0:0] = [*([(fast[0], fast[1]), *r.more] if fast else []), *r.extra] # the bot ran them itself, before the model return calls, reply diff --git a/tests/fakes.py b/tests/fakes.py index 7e88afa..9df3d4a 100644 --- a/tests/fakes.py +++ b/tests/fakes.py @@ -8,13 +8,20 @@ import httpx from lib.config import Config +from lib.tools import Tools TZ = ZoneInfo("Australia/Melbourne") MAC = "AA:BB:CC:DD:EE:FF" +EVERY_SOURCE = ("Ecowitt", "AirGradient", "Pollen", "Forecast") # every source, by name HISTORY_DAYS = 548 # the fake station is 18 months old: enough for every test, and far quicker to archive than years STEP = {"5min": 300, "30min": 1800, "4hour": 14400, "1day": 86400} +class StandInTools: + """A test's stand-in for lib.tools.Tools: it answers call(name, raw, turn) itself, and call_all is the real one over it.""" + call_all = Tools.call_all + + def config(tmp_path, **over) -> Config: base = dict(telegram_token="1:x", xai_api_key="k", xai_base_url="http://x", xai_model="m", tz=TZ, ecowitt_api_key="a", ecowitt_app_key="b", airgradient_token="t", airgradient_location="42", @@ -76,10 +83,12 @@ def uv_index(ts: int) -> float: class FakeEcowitt: """Handles /device/list, /device/real_time and /device/history like api.ecowitt.net.""" - def __init__(self, busy_first: bool = False, history_days: int = HISTORY_DAYS): + def __init__(self, busy_first: bool = False, history_days: int = HISTORY_DAYS, zone: str | None = "Australia/Melbourne"): + """zone: the time zone the device list names (date_zone_id), None for none.""" self.calls: list[dict] = [] self.busy = busy_first self.history_days = history_days + self.zone = zone def __call__(self, request: httpx.Request) -> httpx.Response: p = dict(request.url.params) @@ -90,7 +99,8 @@ def __call__(self, request: httpx.Request) -> httpx.Response: path = request.url.path.rsplit("/", 1)[-1] if path == "list": return self._ok({"list": [{"mac": MAC.lower(), "name": "Fairleigh", "longitude": 145.0, "latitude": -37.8, - "createtime": int((datetime.now(timezone.utc) - timedelta(days=self.history_days)).timestamp())}]}) + "createtime": int((datetime.now(timezone.utc) - timedelta(days=self.history_days)).timestamp()), + **({"date_zone_id": self.zone} if self.zone else {})}]}) if path == "real_time": now = int(datetime.now(timezone.utc).timestamp()) return self._ok({"outdoor": {"temperature": {"time": str(now), "unit": "℃", "value": "12.3"}}}) diff --git a/tests/test_admin_only.py b/tests/test_admin_only.py index fa87839..04db4b8 100644 --- a/tests/test_admin_only.py +++ b/tests/test_admin_only.py @@ -1,5 +1,6 @@ """ADMIN_ONLY (who may use the bot), /alerts in groups, /usage, and the checks every button press goes through.""" +import asyncio import random from types import SimpleNamespace as NS @@ -107,10 +108,13 @@ def test_admin_only_is_on_unless_switched_off_and_the_listed_ids_are_read_safely monkeypatch.setenv(k, v) monkeypatch.delenv("ADMIN_ONLY", raising=False) assert Config.from_env().admin_only is True and Config.from_env().admin_chat_ids == frozenset() - for value, on in (("", True), ("on", True), ("ON", True), ("1", True), ("off", False), ("Off", False), ("0", False), ("no", False), ("false", False)): + for value, on in (("", True), ("on", True), ("ON", True), ("1", True), ("yes", True), ("off", False), ("Off", False), ("0", False), + ("no", False), ("false", False), ("ture", True), ("of", True)): # a typo keeps the default: never open by mistake monkeypatch.setenv("ADMIN_ONLY", value) assert Config.from_env().admin_only is on, value - assert _ids("1, -1001 22,,x, 3.5 -7") == frozenset({1, -1001, 22, -7}) and _ids("") == frozenset() + assert "ADMIN_ONLY: 'ture' is not on or off; keeping the default (on)" in caplog.text + ids = lambda text: _ids(text, frozenset(), "ADMIN_CHAT_IDS") + assert ids("1, -1001 22,,x, 3.5 -7") == frozenset({1, -1001, 22, -7}) and ids("") == frozenset() assert "ADMIN_CHAT_IDS: ignoring 'x', '3.5'" in caplog.text monkeypatch.setenv("ADMIN_CHAT_IDS", "5 -9") assert Config.from_env().admin_chat_ids == frozenset({5, -9}) @@ -155,6 +159,41 @@ async def test_a_failed_lookup_keeps_the_old_list_and_is_retried_soon_without_ra assert not await fresh.allowed(tg, [ADMIN]) # nothing known: nobody +class SlowTG(TG): + """A Telegram client whose admin lists take a moment to come back; `most` is the most it was asked for at once.""" + def __init__(self, admins): + super().__init__(admins) + self.at_once = self.most = 0 + + async def get_chat_administrators(self, chat_id): + self.at_once += 1 + self.most = max(self.most, self.at_once) + await asyncio.sleep(0.01) + self.at_once -= 1 + return await super().get_chat_administrators(chat_id) + + +async def test_the_groups_are_looked_up_together_and_messages_arriving_together_share_the_lookup(tmp_path): + state, tg = AlertState(tmp_path / "s.json"), SlowTG({GROUP: [ADMIN], OTHER_GROUP: [3]}) + state.add_chat(GROUP, "Home") + state.add_chat(OTHER_GROUP, "Away") + admins = Admins(state) + got = await asyncio.gather(*(admins.allowed(tg, [user]) for user in (ADMIN, 3, STRANGER, ADMIN))) + assert got == [True, True, False, True] + assert sorted(tg.fetches) == sorted([GROUP, OTHER_GROUP]) and tg.most == 2 # each list once, both at the same time + + +async def test_the_admins_of_a_group_that_removed_the_bot_no_longer_count(tmp_path): + state, tg = AlertState(tmp_path / "s.json"), TG({GROUP: [ADMIN]}) + state.add_chat(GROUP, "Home") + admins = Admins(state) + assert await admins.allowed(tg, [ADMIN]) # its list is kept for ten minutes now + state.mark_gone(GROUP, "bot kicked") + assert admins.groups() == [] and not await admins.allowed(tg, [ADMIN]) # gone: its admins don't count + state.add_chat(GROUP, "Home") # the bot is added back + assert await admins.allowed(tg, [ADMIN]) + + async def test_listed_ids_are_added_to_the_known_admins_never_replacing_them(tmp_path): state, tg = AlertState(tmp_path / "s.json"), TG({GROUP: [ADMIN], OTHER_GROUP: [3]}) admins = Admins(state, extra_ids=[STRANGER, OTHER_GROUP]) @@ -275,7 +314,7 @@ async def test_usage_is_the_help_and_alerts_button(tmp_path, kind): shown = lambda sent: [(b, [[(x.text, x.callback_data) for x in row] for row in kw["reply_markup"].inline_keyboard] if kw.get("reply_markup") else None) for b, kw in sent] assert len(from_command) == 2 and shown(from_command) == shown(from_button) - assert from_command[0][0] == templates.capabilities_text(True, False, False, False) + assert from_command[0][0] == templates.capabilities_text(["Ecowitt"]) update, refused = message("/usage", user=STRANGER, kind=kind) await bot.on_usage(update, ctx(tg)) assert [b for b, _ in refused] == [NOT_ADMIN] # gated like the other commands diff --git a/tests/test_agent.py b/tests/test_agent.py index fedbe25..b8b98bb 100644 --- a/tests/test_agent.py +++ b/tests/test_agent.py @@ -49,8 +49,9 @@ async def test_weather_questions_must_fetch_first_and_reasoning_is_passed_throug t, seen = tools() client = FakeLLM([[("weather_now", {})], "Rain unlikely."]) agent = llm.Agent(client, "m", t) - reply = await agent.run([{"role": "user", "content": "will it rain?"}], "sys", "medium", require_tool=True) - assert reply == "Rain unlikely." and client.requests[0]["tool_choice"] == "required" + reply = await agent.run([{"role": "user", "content": "will it rain?"}], "sys", "medium", tool_choice="required") + assert reply == "Rain unlikely." and [r["tool_choice"] for r in client.requests] == ["required", "auto"] + assert all(r["tools"] == t.schemas for r in client.requests) # every tool on every call: the cached prompt holds assert client.requests[0]["extra_body"]["reasoning_effort"] == "medium" and len(seen) == 1 @@ -58,11 +59,16 @@ async def test_chat_needs_no_tools_and_identical_calls_are_made_once(): t, seen = tools() client = FakeLLM(["You're welcome!"]) assert await llm.Agent(client, "m", t).run([{"role": "user", "content": "thanks"}], "s", "none", - require_tool=False) == "You're welcome!" + tool_choice="auto") == "You're welcome!" assert client.requests[0]["tool_choice"] == "auto" client = FakeLLM([[("weather_now", {}), ("weather_now", {})], "done"]) await llm.Agent(client, "m", t).run([{"role": "user", "content": "x"}], "s", "none") assert len(seen) == 1 + t, seen = tools() # the same arguments in another order are the same call + client = FakeLLM([[("weather_now", {"groups": "outdoor", "chart": True}), ("weather_now", {"chart": True, "groups": "outdoor"})], "done"]) + msgs = [{"role": "user", "content": "x"}] + await llm.Agent(client, "m", t).run(msgs, "s", "none") + assert len(seen) == 1 and "You already made this exact call" in msgs[-2]["content"] def test_history_keeps_questions_and_answers_only(): @@ -102,13 +108,13 @@ def test_charts_render_for_both_datasets(): assert charts.render(spec, ZoneInfo("Australia/Melbourne"))[:8] == b"\x89PNG\r\n\x1a\n" -async def test_reset_forgets_only_that_chat(): +async def test_reset_forgets_only_that_chat(alert_state): from lib.bot import Bot sent = [] async def reply_text(text, **kw): sent.append(text) - bot = Bot(None, None, [], None) + bot = Bot(None, None, [], alert_state) bot.chats[(1, None)].history.append({"role": "user", "content": "hi"}) bot.chats[(2, None)].history.append({"role": "user", "content": "other chat"}) bot.chats[(1, 7)].history.append({"role": "user", "content": "topic 7"}) @@ -128,7 +134,7 @@ def test_the_prompt_defines_a_rainy_day_and_how_to_phrase_the_count(): from datetime import datetime from lib import prompt from lib.ecowitt import days - text = prompt.build(datetime(2026, 9, 29, 14, 5), ["Ecowitt"]) + text = prompt.build(datetime(2026, 9, 29, 14, 5), {"Ecowitt": "Ecowitt"}) assert "rain >= 1" in text and "any rain" in text and "It rained on 507 of 1,454 days" in text assert "1 mm or more" in days.DESCRIPTION @@ -136,7 +142,7 @@ def test_the_prompt_defines_a_rainy_day_and_how_to_phrase_the_count(): def test_the_prompt_asks_for_a_footnote_about_hotter_trace_days(): from datetime import datetime from lib import prompt - text = prompt.build(datetime(2026, 9, 29, 14, 5), ["Ecowitt"]) + text = prompt.build(datetime(2026, 9, 29, 14, 5), {"Ecowitt": "Ecowitt"}) assert "trace_rain_days" in text and "footnote" in text @@ -149,7 +155,7 @@ def test_the_day_tool_schema_agrees_with_the_prompt_about_a_rainy_day(): def test_the_day_tool_rules_are_short_separate_bullets(): from datetime import datetime from lib import prompt - text = prompt.build(datetime(2026, 9, 29, 14, 5), ["Ecowitt"]) + text = prompt.build(datetime(2026, 9, 29, 14, 5), {"Ecowitt": "Ecowitt"}) block = text[text.index("- For questions that rank, compare or count DAYS"):text.index("- To ask whether rain comes WITH")] bullets = [line for line in block.splitlines() if line.startswith(" - ")] assert len(bullets) == 9 and all(len(b) < 300 for b in bullets) @@ -161,21 +167,21 @@ def test_the_system_prompt_carries_the_period_hints_only_when_there_are_some(): from datetime import datetime from lib import intent, prompt now = datetime(2026, 9, 29, 14, 5) - with_hint = prompt.build(now, ["Ecowitt"], intent.period_hints("average temp 3m", now)) + with_hint = prompt.build(now, {"Ecowitt": "Ecowitt"}, intent.period_hints("average temp 3m", now)) assert "THE PERSON'S WORDS NAME THESE PERIODS" in with_hint and '"3m" = last 3 months: 2026-07-01' in with_hint - assert "THE PERSON'S WORDS NAME" not in prompt.build(now, ["Ecowitt"], intent.period_hints("hello", now)) + assert "THE PERSON'S WORDS NAME" not in prompt.build(now, {"Ecowitt": "Ecowitt"}, intent.period_hints("hello", now)) def test_the_prompt_says_what_the_bot_can_and_cannot_do_for_the_sources_it_has(): from datetime import datetime from lib import prompt now = datetime(2026, 9, 29, 14, 5) - both = prompt.build(now, ["Ecowitt weather station", "AirGradient outdoor sensor"]) + both = prompt.build(now, {"Ecowitt": "Ecowitt weather station", "AirGradient": "AirGradient outdoor sensor"}) assert "WHAT THIS BOT CAN AND CAN'T DO" in both and "PM2.5 (µg/m³)" in both and "weather_link" in both assert "Custom alerts" in both and "solar radiation and UV index" in both and "Not available: lightning" in both and "correct the call once" in both - weather_only = prompt.build(now, ["Ecowitt weather station"]) + weather_only = prompt.build(now, {"Ecowitt": "Ecowitt weather station"}) assert "Weather station:" in weather_only and "Air quality (outdoor AirGradient)" not in weather_only - assert "Weather station:" not in prompt.build(now, ["AirGradient outdoor sensor"]) + assert "Weather station:" not in prompt.build(now, {"AirGradient": "AirGradient outdoor sensor"}) async def test_a_question_about_the_bot_itself_gets_no_tools_and_the_hint(): @@ -184,16 +190,16 @@ async def test_a_question_about_the_bot_itself_gets_no_tools_and_the_hint(): t, seen = tools() client = FakeLLM(["I track temperature, humidity..."]) reply = await llm.Agent(client, "m", t).run([{"role": "user", "content": "list our metrics"}], "sys", "none", - require_tool=False, no_tools=True) + tool_choice="none") assert reply.startswith("I track") and client.requests[0]["tool_choice"] == "none" and seen == [] now = datetime(2026, 9, 29, 14, 5) - assert "ABOUT THE BOT ITSELF" in prompt.build(now, ["Ecowitt"], [], True) and "ABOUT THE BOT ITSELF" not in prompt.build(now, ["Ecowitt"]) + assert "ABOUT THE BOT ITSELF" in prompt.build(now, {"Ecowitt": "Ecowitt"}, [], True) and "ABOUT THE BOT ITSELF" not in prompt.build(now, {"Ecowitt": "Ecowitt"}) def test_the_prompt_never_shows_a_temperature_emoji_to_copy_and_says_no_entry_means_no_emoji(): from datetime import datetime from lib import prompt - text = prompt.build(datetime(2026, 9, 29, 14, 5), ["Ecowitt weather station"]) + text = prompt.build(datetime(2026, 9, 29, 14, 5), {"Ecowitt": "Ecowitt weather station"}) assert '"outdoor.temperature": "' not in text and "a reading with no entry gets NO emoji" in text @@ -242,27 +248,23 @@ def test_a_rain_chart_caption_is_only_the_least_and_most_rain(): from datetime import datetime from lib import prompt now = datetime(2026, 9, 29, 14, 5) - text = prompt.build(now, ["Ecowitt weather station"], rain_caption=True) + text = prompt.build(now, {"Ecowitt": "Ecowitt weather station"}, rain_caption=True) assert "THIS IS A RAIN CHART" in text and "rain_outlook" not in text - assert "THIS IS A RAIN CHART" not in prompt.build(now, ["Ecowitt weather station"]) + assert "THIS IS A RAIN CHART" not in prompt.build(now, {"Ecowitt": "Ecowitt weather station"}) def test_questions_about_the_bot_are_answered_as_simple_bullet_points(): from datetime import datetime from lib import prompt - text = prompt.build(datetime(2026, 9, 29, 14, 5), ["Ecowitt weather station"], about_bot=True) + text = prompt.build(datetime(2026, 9, 29, 14, 5), {"Ecowitt": "Ecowitt weather station"}, about_bot=True) assert "simple bullet points" in text and "no sub-bullets" in text and "7 at most" in text and "No dates" in text -POLLEN_SOURCE = "Melbourne pollen forecast and thunderstorm asthma risk (melbournepollen.com.au)" -FORECAST_SOURCE = "Weather forecast for the owner's location (Open-Meteo)" - - def test_the_capabilities_mention_the_new_sources_only_when_they_are_on(): from lib import prompt - base = prompt.capabilities(["Ecowitt weather station"]) + base = prompt.capabilities(["Ecowitt"]) assert "pollen and thunderstorm asthma" in base.split("Not available:")[1] and "forecasts (only a short read" in base - both = prompt.capabilities(["Ecowitt weather station", POLLEN_SOURCE, FORECAST_SOURCE]) + both = prompt.capabilities(["Ecowitt", "Pollen", "Forecast"]) assert "tool pollen_asthma" in both and "tool weather_forecast" in both and "thunderstorm asthma risk reaching High" in both assert "pollen" not in both.split("Not available:")[1] and "forecasts" not in both.split("Not available:")[1] @@ -288,25 +290,33 @@ async def handler(args, turn=None): def test_the_capabilities_mention_forecast_revision_alerts_only_with_the_forecast_source(): from lib import prompt - assert "revised" not in prompt.capabilities(["Ecowitt weather station"]) - assert "a forecast it sent being revised" in prompt.capabilities(["Ecowitt weather station", FORECAST_SOURCE]) + assert "revised" not in prompt.capabilities(["Ecowitt"]) + assert "a forecast it sent being revised" in prompt.capabilities(["Ecowitt", "Forecast"]) + + +def test_the_capabilities_list_only_the_alerts_the_sources_send_with_their_own_thresholds(): + from lib import prompt + from lib.alerts.weather import GUST_ALERT_KMH + station, air = prompt.capabilities(["Ecowitt"]), prompt.capabilities(["AirGradient"]) + assert f"wind gusts over {GUST_ALERT_KMH} km/h" in station and "mask alerts" not in station + assert "mask alerts" in air and "rain starting" not in air and "Alerts, sent" not in prompt.capabilities([]) def test_the_capabilities_say_pollen_only_runs_october_to_december(): from lib import prompt - assert "only run from October to December" in prompt.capabilities(["Ecowitt weather station", POLLEN_SOURCE]) + assert "only run from October to December" in prompt.capabilities(["Ecowitt", "Pollen"]) def test_the_prompt_leaves_the_heads_up_to_the_bot_and_points_at_the_count_tool(): from datetime import datetime from lib import prompt - text = prompt.build(datetime(2026, 9, 29, 14, 5), ["Ecowitt weather station"]) + text = prompt.build(datetime(2026, 9, 29, 14, 5), {"Ecowitt": "Ecowitt weather station"}) assert "the bot warns the chat by itself" in text and "\u26a0" not in text and "decline" not in text.lower() assert "uv_max >= 9" in text and "count_only" in text async def test_the_chat_is_warned_once_before_the_model_is_sent_more_than_twice_a_500_point_querys_data(): - from lib.timeutil import CHARS_PER_TOKEN, WARN_TOKENS + from lib.llm import CHARS_PER_TOKEN, WARN_TOKENS from lib.tools import Turn warned, asked_after = [], [] @@ -334,7 +344,7 @@ async def tiny(args, turn=None): def test_the_prompt_says_to_use_the_whole_record_without_a_period_and_holidays_only_when_asked(): from datetime import datetime from lib import prompt - text = prompt.build(datetime(2026, 9, 29, 14, 5), ["Ecowitt weather station"]) + text = prompt.build(datetime(2026, 9, 29, 14, 5), {"Ecowitt": "Ecowitt weather station"}) assert "use the whole record" in text and "only when the person's words say holidays or weekends" in text @@ -344,7 +354,7 @@ async def test_every_call_of_a_chat_carries_the_same_conversation_id_for_the_pro await llm.Agent(client, "m", t).run([{"role": "user", "content": "q"}], "sys", "none", conv_id="ecobot-42") assert [r["extra_headers"] for r in client.requests] == [{"x-grok-conv-id": "ecobot-42"}] * 2 plain = FakeLLM(["Hi."]) - await llm.Agent(plain, "m", t).run([{"role": "user", "content": "q"}], "sys", "none", require_tool=False) + await llm.Agent(plain, "m", t).run([{"role": "user", "content": "q"}], "sys", "none", tool_choice="auto") assert "extra_headers" not in plain.requests[0] diff --git a/tests/test_airgradient.py b/tests/test_airgradient.py index a9caccb..31bd355 100644 --- a/tests/test_airgradient.py +++ b/tests/test_airgradient.py @@ -1,7 +1,12 @@ import json from datetime import datetime, timedelta +import httpx +import pytest + from lib.airgradient import AirGradient, pm25_aqi, rating +from lib.airgradient.metrics import METRICS +from lib.timeutil import local_date from lib.tools import Turn from tests.fakes import TZ, air_transport, config @@ -13,6 +18,13 @@ async def make(tmp_path): return air, calls +def test_every_metric_is_rated_on_its_own_scale(): + for name, good, poor in (("pm2_5", 9.0, 55.4), ("pm10", 54.0, 254.0), ("pm1", 9.0, 55.4), ("co2", 799.0, 1499.0), + ("voc_index", 150.0, 250.0), ("nox_index", 20.0, 150.0)): + zones = [rating(name, v).split(" ", 1)[1] for v in (good, good + 0.1, poor, poor + 0.1)] # after the emoji + assert zones == ["good", "poor", "poor", "very poor"], name + + def test_aqi_and_ratings_follow_the_us_scale(): assert pm25_aqi(9.0) == (50, "good") assert pm25_aqi(55.5)[0] == 151 and pm25_aqi(55.5)[1] == "unhealthy" @@ -22,21 +34,21 @@ def test_aqi_and_ratings_follow_the_us_scale(): async def test_current_reading_with_ratings(tmp_path): air, _ = await make(tmp_path) - async def no_previous(ts): - return None # the fake history's PM2.5 depends on the hour, so the previous reading would too - air._previous_row = no_previous + async def no_previous(ts, count): + return [] # the fake history's PM2.5 depends on the hour, so the previous reading would too + air._previous_rows = no_previous reading = json.loads(await air.handle({})) assert reading["pm2_5"]["value"] == 12.0 and reading["pm2_5"]["band"] == "moderate" - assert "_time_utc" not in reading and reading["pm2_5"]["rating"].endswith("poor") + assert set(reading) == {"sensor", "sensor_type", "time", *METRICS} and reading["pm2_5"]["rating"].endswith("poor") await air.close() async def test_a_rating_is_the_level_the_last_two_readings_both_reached(tmp_path): air, _ = await make(tmp_path) for before, band, noted in ((5.0, "good", True), (20.0, "moderate", False)): - async def previous(ts, before=before): - return {"pm2_5": before} - air._previous_row = previous + async def previous(ts, count, before=before): + return [{"pm2_5": before}] + air._previous_rows = previous reading = json.loads(await air.handle({})) assert reading["pm2_5"]["value"] == 12.0 and reading["pm2_5"]["band"] == band assert ("rating_note" in reading["pm2_5"]) is noted @@ -165,7 +177,7 @@ async def test_bucketed_air_lines_carry_a_band_and_the_peak_label_sits_on_its_to def await_days(air, days_ago): """Readings stored for the local day this many days ago (0 if none or not fetched).""" day = datetime.now(TZ).date() - timedelta(days=days_ago) - return air.store.day_count(day) or 0 + return air.store.counts([day])[day] or 0 # ---------- what the API's docs promise ---------- @@ -375,14 +387,14 @@ async def test_a_single_reading_spike_is_in_none_of_the_figures_either(tmp_path) from lib.timeutil import local_date from tests.fakes import TZ air, _ = await make(tmp_path) - start = air._now().replace(hour=0, minute=0, second=0) - timedelta(hours=6) - t0 = int(start.replace(tzinfo=TZ).timestamp()) + start = air.now().replace(microsecond=0) - timedelta(hours=1) + t0 = int(start.replace(tzinfo=TZ).timestamp()) + 600 levels = [(1, 2, 3), (1, 2, 4), (15.3, 14.2, 28.7), (3, 8, 10), (2, 4, 5), (2, 3, 4)] rows = [{"ts": t0 + 300 * i, "pm1": a, "pm2_5": b, "pm10": c} for i, (a, b, c) in enumerate(levels)] - async def held(t0_, t1_): - return rows, 0, [] - air.rows = held + async def day_rows(day, refresh=False): + return [r for r in rows if local_date(r["ts"], TZ) == day] + air._day_rows = day_rows out = await air.history(start.strftime("%Y-%m-%d %H:%M:%S"), None) assert out["pm10"]["high"] == 10 and out["pm2_5"]["high"] == 8 and out["pm1"]["high"] == 3 # not 28.7, 14.2 and 15.3 assert out["pm10"]["average"] == round((3 + 4 + 10 + 5 + 4) / 5, 1) # and not in the mean @@ -408,12 +420,50 @@ def test_the_readings_a_rise_needs_are_set_by_the_antispike_setting(): metrics.REQUIRE_CONSECUTIVE = 2 -async def test_current_gives_the_lowest_of_the_last_readings_for_the_alerts_and_not_to_the_model(tmp_path): - from lib.airgradient import metrics +async def test_the_latest_reading_comes_with_the_lowest_of_the_last_readings_for_the_alerts(tmp_path): air, _ = await make(tmp_path) - out = await air.current() - assert set(out["_sustained"]) >= {"pm2_5"} and out["_sustained"]["pm2_5"] <= out["pm2_5"]["value"] - assert "_sustained" not in json.loads(await air.handle({}, None)) + reading, at, sustained = await air.latest() + assert set(sustained) >= {"pm2_5"} and sustained["pm2_5"] <= reading["pm2_5"]["value"] and at.tzinfo is not None + assert json.loads(await air.handle({}, None)) == json.loads(json.dumps(reading)) # the tool gives the reading alone + await air.close() + + +async def test_the_reading_is_rated_alone_when_only_the_history_fails(tmp_path, caplog): + air, _ = await make(tmp_path) + + async def down(day, refresh=False): + raise RuntimeError("AirGradient past failed: HTTP 500") + air._day_rows = down + reading = json.loads(await air.handle({})) + assert reading["pm2_5"]["value"] == 12.0 and "rating_note" not in reading["pm2_5"] and "rating the latest alone" in caplog.text + assert (await air.latest()).sustained["pm2_5"] == 12.0 + await air.close() + + +@pytest.mark.parametrize("status, body", [(200, {"message": "Too many requests"}), (404, {"message": "Location not found"}), + (200, "OK")]) +async def test_a_reply_that_is_not_readings_is_an_error_and_never_stored_as_a_day_without_any(tmp_path, status, body): + air = AirGradient(config(tmp_path), transport=httpx.MockTransport(lambda request: httpx.Response(status, json=body))) + day = datetime.now(TZ).date() - timedelta(days=3) + with pytest.raises(RuntimeError): + await air._fetch_days([day]) + assert air.store.counts([day])[day] is None # not stored: asked again next time + await air.close() + + +async def test_a_chart_and_its_figures_take_out_the_same_spikes(tmp_path): + air, _ = await make(tmp_path) + now = datetime.now(TZ).replace(tzinfo=None, microsecond=0) + first = int((now - timedelta(minutes=40)).replace(tzinfo=TZ).timestamp()) + spiky = [{"ts": first + i * 300, "pm2_5": v} for i, v in enumerate((10.0, 10.0, 100.0, 30.0, 10.0, 10.0))] + + async def day_rows(day, refresh=False): + return [r for r in spiky if local_date(r["ts"], TZ) == day] + air._day_rows = day_rows + turn = Turn() + out = await air.history(f"{now - timedelta(hours=1):%Y-%m-%d %H:%M:%S}", f"{now:%Y-%m-%d %H:%M:%S}", chart=True, turn=turn) + line = turn.charts[0].panels[0].lines[0] + assert out["pm2_5"]["high"] == line.records["high"][1] == 30.0 and 100.0 not in line.y # one pass: the spike, not what was next to it await air.close() @@ -426,3 +476,24 @@ def test_the_antispike_setting_is_a_number_of_readings_default_2_and_off_is_1(mo for value, want in (("3", 3), ("1", 1), ("0", 1), ("off", 1), ("junk", 2), ("", 2)): monkeypatch.setenv("ANTISPIKE_REQUIRE_CONSECUTIVE", value) assert Config.from_env().antispike_require_consecutive == want, value + + +async def test_a_date_with_an_offset_is_read_and_a_bad_one_is_not_called_a_sensor_fault(tmp_path): + """The model sometimes sends "…Z": that is a time, not a failure to read the sensor. A date that can't be read says so.""" + from datetime import timezone as tz_ + air, _ = await make(tmp_path) + start = (datetime.now(TZ) - timedelta(days=2)).astimezone(tz_.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + out = json.loads(await air.handle({"start_date": start})) + bad = json.loads(await air.handle({"start_date": "yesterday morning"})) + await air.close() + assert "error" not in out and out["readings"] > 0 and bad["error"].startswith("bad date") + + +async def test_an_air_chart_up_to_now_is_captioned_since_its_first_day_as_a_weather_chart_is(tmp_path): + air, _ = await make(tmp_path) + now = air.now().replace(microsecond=0) + first = (now - timedelta(days=3)).date() + turn = Turn() + await air.history(f"{now - timedelta(days=3):%Y-%m-%d %H:%M:%S}", None, chart=True, turn=turn) + assert turn.charts[0].caption(now.date()) == f"PM2.5, since {first:%a} {first.day} {first:%b %Y}" # not "... - ..." as one day + await air.close() diff --git a/tests/test_alert_menu.py b/tests/test_alert_menu.py index 90ffe6e..9d5b7e0 100644 --- a/tests/test_alert_menu.py +++ b/tests/test_alert_menu.py @@ -22,9 +22,9 @@ def test_alerts_has_two_buttons_alerts_and_sound_and_a_group_has_only_the_first( assert rows(keyboard(set(), KINDS, "sound")) == rows(keyboard(set(), KINDS)) # a group cannot open Sound -def test_the_bell_toggles_are_green_when_on_and_struck_through_when_off(): +def test_the_bell_toggles_are_blue_when_on_and_struck_through_when_off(): styles = lambda m: {b.text: b.style for row in m.inline_keyboard for b in row} - assert styles(alert_keyboard("uv", set(), set(), "bell")) == {"🔔 ▾": None, "Alert": "success", strike("Sound"): None} + assert styles(alert_keyboard("uv", set(), set(), "bell")) == {"🔔 ▾": None, "Alert": "primary", strike("Sound"): None} assert styles(alert_keyboard("uv", {"uv"}, {"uv"}, "bell")) == {"🔔 ▾": None, strike("Alert"): None} @@ -47,11 +47,11 @@ def test_every_button_of_every_keyboard_parses_and_builds_back_to_the_same_data( assert parse(junk) is None, junk -def test_a_type_that_is_on_is_a_green_button_and_one_that_is_off_is_struck_through_and_plain(): +def test_a_type_that_is_on_is_a_blue_button_and_one_that_is_off_is_struck_through_and_plain(): markup = keyboard({"gusts"}, KINDS, "alerts", loud=set()) styles = {b.text: b.api_kwargs.get("style") or b.style for row in markup.inline_keyboard for b in row} - assert styles["rain"] == "success" and styles[strike("gusts")] is None and not styles["🔔 Alerts ▾"] - assert markup.inline_keyboard[1][0].to_dict()["style"] == "success" and "style" not in markup.inline_keyboard[1][1].to_dict() # and it is sent to Telegram + assert styles["rain"] == "primary" and styles[strike("gusts")] is None and not styles["🔔 Alerts ▾"] + assert markup.inline_keyboard[1][0].to_dict()["style"] == "primary" and "style" not in markup.inline_keyboard[1][1].to_dict() # and it is sent to Telegram def test_each_menu_lists_its_types_struck_through_when_off_and_keeps_only_its_own_button(): @@ -82,24 +82,21 @@ def test_the_title_lists_what_is_on_and_off(): assert title(set(), ["particulates"]).endswith("🔕 Off: nothing") and title({"particulates"}, ["particulates"]).endswith("On: nothing\n🔕 Off: particulates") -def test_state_mutes_one_type_or_all_and_old_files_load(tmp_path): +def test_state_mutes_one_type_or_all(tmp_path): state = AlertState(tmp_path / "s.json") state.add_chat(1, "a") state.add_chat(2, "b") state.set_kind(1, "rain", False) state.set_kind(1, "bogus", False) - assert state.muted(1) == {"rain"} and state.alert_chats("rain") == [2] and state.alert_chats("uv") == [1, 2] and state.alerts_on(1) + assert state.muted(1) == {"rain"} and state.alert_chats("rain") == [2] and state.alert_chats("uv") == [1, 2] and state.subscribed(1) state.set_kind(1, ALL, False) - assert state.muted(1) == set(LABELS) and not state.alerts_on(1) and state.alert_chats() == [2] + assert state.muted(1) == set(LABELS) and not state.subscribed(1) and state.alert_chats() == [2] state.set_kind(1, ALL, True) assert state.muted(1) == set() and "muted" not in state.chats[1] state.set_kind(2, "gusts", False) assert AlertState(tmp_path / "s.json").muted(2) == {"gusts"} # saved - (tmp_path / "old.json").write_text('{"chats": {"7": {"title": "x", "alerts": false}, "8": {"title": "y", "alerts": true}}, "monitor": {}}') - old = AlertState(tmp_path / "old.json") - assert old.muted(7) == set(LABELS) and old.muted(8) == set() and old.alert_chats() == [8] - state.set_alerts(1, "a", False) # the /alerts off shortcut - assert not state.alerts_on(1) + state.set_alerts(1, False) # the /alerts off shortcut + assert not state.subscribed(1) class Sent: @@ -123,7 +120,12 @@ async def test_an_alert_goes_to_the_chats_subscribed_to_its_type_and_carries_the markup = bot.sent[0][2]["reply_markup"] assert rows(markup) == [[("🔔 ▸", "al:open:bell:rain")]] # the alert's own type rides along assert bot.sent[0][1] == "It's raining" # no footer: the buttons say how to change it - assert await notifier.to_chat(2, "x", kind="rain") is False and await notifier.to_chat(2, "x", kind="uv") is True + assert await notifier("x", kind="rain", chats=[2]) == 0 and await notifier("x", kind="uv", chats=[2]) == 1 + + +async def png(): + """An alert's chart, drawn.""" + return b"png" class Query: @@ -142,6 +144,8 @@ async def edit_message_reply_markup(self, reply_markup=None): self.edits.append(("markup", reply_markup)) async def edit_message_text(self, text, reply_markup=None): + if self.fail: + raise self.fail self.edits.append(("text", text, reply_markup)) @@ -181,12 +185,21 @@ async def test_the_alerts_menu_toggles_a_type_in_place_and_back_returns_to_the_t assert rows(q.edits[0][2]) == [[("🔔 Alerts ▸", "al:open:alerts"), ("🔊 Sound ▸", "al:open:sound")]] +async def test_turning_off_every_type_the_bot_sends_unsubscribes_the_chat_from_all(tmp_path, monkeypatch): + from lib.alerts import menu + bot, state = make_bot(tmp_path, sources=("Ecowitt",)) + for kind in ("rain", "gusts", "uv", "temps"): # the weather station's types, one by one + await press(bot, Query(f"al:off:{kind}:alerts")) + monkeypatch.setitem(menu.LABELS, "newthing", "new thing") + assert state.chats[1].get("optin") and not state.subscribed(1, "newthing") and not state.subscribed(1, "particulates") + + async def test_the_alerts_message_keeps_its_summary_current_but_an_alert_keeps_its_text(tmp_path): bot, state = make_bot(tmp_path) q = Query("al:off:uv:alerts", text=f"{TITLE}\n✅ On: ...") await press(bot, q) assert q.edits[0][0] == "text" and "🔕 Off: UV" in q.edits[0][1] - q = Query("al:off:gusts:alerts", text="💨 Strong gusts: 55 km/h") + q = Query("al:off:gusts:bell:gusts", text="💨 Strong gusts: 55 km/h") await press(bot, q) assert q.edits[0][0] == "markup" @@ -233,20 +246,20 @@ async def reply_text(body, **kw): assert replies[0][0].startswith(TITLE) and "On: rain, gusts, UV, temperature crossing, particulates, pollen & asthma, forecast changes" in replies[0][0] assert rows(replies[0][1]["reply_markup"])[0] == [("🔔 Alerts ▸", "al:open:alerts"), ("🔊 Sound ▸", "al:open:sound")] await bot.on_alerts(update, NS(args=["off"])) - assert not state.alerts_on(1) and "On: nothing" in replies[1][0] + assert not state.subscribed(1) and "On: nothing" in replies[1][0] await bot.on_alerts(update, NS(args=["on"])) - assert state.alerts_on(1) and "🔕 Off: nothing" in replies[2][0] + assert state.subscribed(1) and "🔕 Off: nothing" in replies[2][0] async def test_every_monitors_alert_names_its_type(tmp_path): from lib.alerts import ParticulatesMonitor, ForecastMonitor, PollenAndAsthmaMonitor, WeatherMonitor from tests.test_alerts import FakePollen, FakeStation, T0, air_reading, gusts, rain, uv - from tests.test_forecast_alert import FakeForecast, day + from tests.test_forecast_alert import TODAY, FakeForecast, day state = AlertState(tmp_path / "s.json") state.add_chat(1, "chat") kinds = [] - async def notify(text, link=None, kind=None): + async def notify(text, link=None, kind=None, **more): kinds.append(kind) async def weather(first, then): @@ -271,14 +284,12 @@ async def weather(first, then): assert kinds == ["pollen_and_asthma"] seen = [] notifier = Notifier(Sent(), state) - real = notifier.to_chat - - async def spy(chat_id, text, link=None, kind=None): - seen.append(kind) - return await real(chat_id, text, link, kind) - notifier.to_chat = spy - state.record_forecast(1, [day(0, pct=90)]) - await ForecastMonitor(FakeForecast([day(0, pct=5)]), state, notifier).check() + + async def spy(text, **kw): + seen.append(kw["kind"]) + return await notifier(text, **kw) + state.record_forecast(1, [day(0, pct=90)], TODAY) + await ForecastMonitor(FakeForecast([day(0, pct=5)]), state, spy).check() assert seen == ["forecast"] @@ -299,28 +310,29 @@ async def test_the_bell_under_an_alert_toggles_the_alert_and_its_sound_and_stays bot, state = make_bot(tmp_path) q = Query("al:open:bell:uv", text="☀️ UV 10") await press(bot, q) - assert rows(q.edits[0][1]) == [[("🔔 ▾", "al:close:uv")], [("Alert", "al:off:uv:bell:uv")], [(strike("Sound"), "al:snd:uv:on:uv")]] + assert rows(q.edits[0][-1]) == [[("🔔 ▾", "al:close:uv")], [("Alert", "al:off:uv:bell:uv")], [(strike("Sound"), "al:snd:uv:on:uv")]] q = Query("al:snd:uv:on:uv", text="☀️ UV 10") await press(bot, q) - assert state.loud(1) == {"uv"} and q.toast == "UV alerts now make a sound" and rows(q.edits[0][1])[2] == [("Sound", "al:snd:uv:off:uv")] + assert state.loud(1) == {"uv"} and q.toast == "UV alerts now make a sound" and rows(q.edits[0][-1])[2] == [("Sound", "al:snd:uv:off:uv")] q = Query("al:off:uv:bell:uv", text="☀️ UV 10") # alert off: the sound toggle goes, the struck alert stays to turn it back on await press(bot, q) assert "uv" in state.muted(1) and q.toast == "UV alerts off in this chat" - assert rows(q.edits[0][1]) == [[("🔔 ▾", "al:close:uv")], [(strike("Alert"), "al:on:uv:bell:uv")]] + assert rows(q.edits[0][-1]) == [[("🔔 ▾", "al:close:uv")], [(strike("Alert"), "al:on:uv:bell:uv")]] q = Query("al:on:uv:bell:uv", text="☀️ UV 10") await press(bot, q) - assert "uv" not in state.muted(1) and rows(q.edits[0][1])[2] == [("Sound", "al:snd:uv:off:uv")] # the sound setting was kept + assert "uv" not in state.muted(1) and rows(q.edits[0][-1])[2] == [("Sound", "al:snd:uv:off:uv")] # the sound setting was kept q = Query("al:close:uv", text="☀️ UV 10") await press(bot, q) - assert rows(q.edits[0][1]) == [[("🔔 ▸", "al:open:bell:uv")]] + assert rows(q.edits[0][-1]) == [[("🔔 ▸", "al:open:bell:uv")]] -async def test_an_unknown_parent_in_a_button_is_ignored(tmp_path): - bot, state = make_bot(tmp_path) - q = Query("al:open:alerts:bogus", text="☀️ UV 10") - await press(bot, q) - drawn = rows(q.edits[0][1]) - assert drawn[1][0] == ("rain", "al:off:rain:alerts") # the /alerts menu, not carried on +async def test_the_bell_of_an_alert_type_the_bot_no_longer_sends_is_out_of_date(tmp_path): + from lib.bot import STALE_BUTTON + bot, state = make_bot(tmp_path, sources=("Ecowitt",)) # no pollen source any more + for data in ("al:open:bell:pollen_and_asthma", "al:off:pollen_and_asthma:bell:pollen_and_asthma"): + q = Query(data, text="🌼 Grass pollen is High") + await press(bot, q) + assert q.toast == STALE_BUTTON and q.edits == [] and state.muted(1) == set(), data def test_the_title_stays_plain_words(): @@ -332,13 +344,13 @@ async def test_the_sound_menu_toggles_every_type_in_private_chats_only_and_back_ state.set_kind(1, "gusts", False) q = Query("al:open:sound") await press(bot, q) - drawn = rows(q.edits[0][1]) + drawn = rows(q.edits[0][-1]) assert drawn[0] == [("🔊 Sound ▾", "al:close")] and drawn[1] == [(strike("rain"), "al:snd:rain:on"), (strike("UV"), "al:snd:uv:on")] assert "gusts" not in str(drawn) # an alert that is off has no sound to set assert "Alerts" not in str(drawn) # the Alerts menu is hidden while Sound is open q = Query("al:snd:rain:on") await press(bot, q) - assert state.loud(1) == {"rain"} and q.toast == "Rain alerts now make a sound" and rows(q.edits[0][1])[1][0] == ("rain", "al:snd:rain:off") + assert state.loud(1) == {"rain"} and q.toast == "Rain alerts now make a sound" and rows(q.edits[0][-1])[1][0] == ("rain", "al:snd:rain:off") await press(bot, Query("al:snd:uv:on")) assert state.loud(1) == {"rain", "uv"} q = Query("al:snd:rain:off") @@ -349,14 +361,14 @@ async def test_the_sound_menu_toggles_every_type_in_private_chats_only_and_back_ assert state.loud(1) == {"uv"} q = Query("al:close") await press(bot, q) - assert rows(q.edits[0][1]) == [[("🔔 Alerts ▸", "al:open:alerts"), ("🔊 Sound ▸", "al:open:sound")]] + assert rows(q.edits[0][-1]) == [[("🔔 Alerts ▸", "al:open:alerts"), ("🔊 Sound ▸", "al:open:sound")]] q = Query("al:snd:rain:on", chat_type="supergroup", chat_id=-5) # groups have no sound settings state.add_chat(-5, "G") await press(bot, q) assert state.loud(-5) == set() and q.edits == [] q = Query("al:open:sound", chat_type="supergroup", chat_id=-5) await press(bot, q) - assert rows(q.edits[0][1]) == [[("🔔 Alerts ▸", "al:open:alerts")]] + assert rows(q.edits[0][-1]) == [[("🔔 Alerts ▸", "al:open:alerts")]] async def test_an_alert_has_a_sound_only_in_a_chat_that_turned_it_on_and_groups_get_no_settings_button(tmp_path): @@ -426,7 +438,7 @@ async def send_photo(self, chat_id, photo, **kw): async def test_an_alert_with_a_picture_is_the_picture_with_the_text_as_its_caption_and_the_buttons(tmp_path): state, bot = AlertState(tmp_path / "s.json"), SentWithPhotos() state.add_chat(1, "one") - await Notifier(bot, state)("It's warmer outside", kind="temps", photo=b"png") + await Notifier(bot, state)("It's warmer outside", kind="temps", chart=png) assert bot.sent == [] and len(bot.photos) == 1 chat, photo, kw = bot.photos[0] assert (chat, photo, kw["caption"]) == (1, b"png", "It's warmer outside") and kw["disable_notification"] is True @@ -441,16 +453,16 @@ async def send_photo(self, chat_id, photo, **kw): state, bot = AlertState(tmp_path / "s.json"), Keeps() for chat in (1, 2, 3): state.add_chat(chat, str(chat)) - await Notifier(bot, state)("It's warmer outside", kind="temps", photo=b"png") + await Notifier(bot, state)("It's warmer outside", kind="temps", chart=png) assert [(chat, photo) for chat, photo, _ in bot.photos] == [(1, b"png"), (2, "large"), (3, "large")] -async def test_a_picture_telegram_refuses_falls_back_to_the_text_but_a_missing_chat_is_still_forgotten(tmp_path): +async def test_a_picture_telegram_refuses_falls_back_to_the_text_but_a_missing_chat_is_still_marked_gone(tmp_path): from telegram.error import BadRequest state, bot = AlertState(tmp_path / "s.json"), SentWithPhotos(refuse=BadRequest("Wrong file identifier")) state.add_chat(1, "one") - await Notifier(bot, state)("It's warmer outside", kind="temps", photo=b"png") + await Notifier(bot, state)("It's warmer outside", kind="temps", chart=png) assert bot.photos == [] and [(c, t) for c, t, _ in bot.sent] == [(1, "It's warmer outside")] gone = SentWithPhotos(refuse=BadRequest("Chat not found")) - await Notifier(gone, state)("again", kind="temps", photo=b"png") - assert gone.sent == [] and 1 not in state.chats + await Notifier(gone, state)("again", kind="temps", chart=png) + assert gone.sent == [] and state.chats[1]["gone"] and state.alert_chats() == [] diff --git a/tests/test_alerts.py b/tests/test_alerts.py index ee058fd..5bd630b 100644 --- a/tests/test_alerts.py +++ b/tests/test_alerts.py @@ -1,7 +1,11 @@ import time from datetime import datetime, timedelta, timezone -from lib.alerts import ParticulatesMonitor, AlertState, Notifier, WeatherMonitor, with_footer +import pytest + +from lib.airgradient import Latest +from lib.alerts import ParticulatesMonitor, AlertState, Notifier, WeatherMonitor +from lib.entities import with_link from tests.fakes import TZ T0 = int(datetime(2026, 9, 29, 13, 0, tzinfo=TZ).timestamp()) # 1pm Melbourne, a settled day @@ -28,7 +32,8 @@ async def live_rain(self): return self.live async def recent(self, hours): - return self.data + """The readings of the last `hours` (up to the newest one, which stands for now).""" + return [(ts, r) for ts, r in self.data if ts >= self.data[-1][0] - hours * 3600] if self.data else [] def now(self): return datetime.now(TZ).replace(tzinfo=None) @@ -63,6 +68,17 @@ async def test_rain_starts_once_then_stops_after_30_dry_minutes(tmp_path): assert len(sent) == 2 and "stopped" in sent[1] and "mm fell" in sent[1] +@pytest.mark.parametrize("wet, told", [(60, "12.0 mm fell over 5h"), + (31, "6.2 mm fell over 2h 35m")]) # its first tip is the oldest of the 3 hours looked at +async def test_the_rain_stopped_alert_counts_a_spell_longer_than_the_readings_it_looks_at(tmp_path, wet, told): + m, state, sent = monitor(tmp_path, None) + spell = rain(0, 0, *[1] * wet, *[0] * 6) # 0.2 mm a wet reading, then 30 dry minutes + for n in (3, len(spell)): + m.station.data = spell[:n] + await m.check() + assert len(sent) == 2 and sent[1] == f"\U0001f324️ The rain has stopped. {told}." + + async def test_a_dry_gap_shorter_than_the_stop_time_does_not_flap(tmp_path): m, state, sent = monitor(tmp_path, None) for data in (rain(1, 1), rain(1, 1, 0, 0), rain(1, 1, 0, 0, 0), rain(1, 1, 0, 0, 0, 1), rain(1, 1, 0, 0, 0, 1, 0)): @@ -190,13 +206,6 @@ async def test_a_crossing_soon_after_the_state_was_learned_waits_out_the_cooldow assert len(sent) == 1 and "cooler for 30 min" in sent[0] -async def test_crossing_state_saved_by_an_earlier_version_still_works(tmp_path): - m, state, sent = monitor(tmp_path, temps(25, 20, T0 + 2 * 3600)) - state.monitor["cross"] = {"side": "cooler", "since": T0} - await m.check() - assert len(sent) == 1 and "cooler for 2h" in sent[0] and state.monitor["cross"]["alerted_state"] == "warmer" - - async def test_small_temperature_differences_are_noise(tmp_path): m, state, sent = monitor(tmp_path, temps(20.1, 20.0)) await m.check() @@ -205,8 +214,8 @@ async def test_small_temperature_differences_are_noise(tmp_path): # ---------- air ---------- def air_reading(pm25, pm10=10.0, age=timedelta(minutes=5)): - return {"pm2_5": {"value": pm25, "aqi_us": 160}, "pm10": {"value": pm10}, - "_time_utc": datetime.now(timezone.utc) - age} + """What air.latest() gives: the reading, when it was taken, and (none here: the reading's own) the lowest of the last readings.""" + return Latest({"pm2_5": {"value": pm25, "aqi_us": 160}, "pm10": {"value": pm10}}, datetime.now(timezone.utc) - age, {}) class FakeAir: @@ -265,22 +274,22 @@ def test_alert_state_survives_restarts_and_opt_out(tmp_path): state = AlertState(path) state.add_chat(1, "Home") state.add_chat(2, "Work") - state.set_alerts(2, "Work", False) + state.set_alerts(2, False) state.monitor["rain"] = {"alerted_state": "raining", "alerted_time": 5, "since": 5} state.save() again = AlertState(path) - assert again.alert_chats() == [1] and again.monitor["rain"]["alerted_state"] == "raining" and not again.alerts_on(2) + assert again.alert_chats() == [1] and again.monitor["rain"]["alerted_state"] == "raining" and not again.subscribed(2) -def test_footer_marks_up_link_and_mute_hint(): - text, entities = with_footer("\U0001f327️ Rain", ("live chart", "https://x"), "/alerts off to mute") - assert text.endswith("live chart · /alerts off to mute") +def test_the_footer_is_the_links_label_in_italic_placed_in_utf16_units(): + text, entities = with_link("\U0001f327️ Rain \n", ("live chart", "https://x")) + assert text == "\U0001f327️ Rain\nlive chart" and with_link("Rain \n", None) == ("Rain", []) italic, link = entities - assert italic.offset == link.offset == len("\U0001f327️ Rain".encode("utf-16-le")) // 2 + 1 - assert link.length == len("live chart") and link.url == "https://x" + assert italic.type == "italic" and italic.offset == link.offset == len("\U0001f327️ Rain".encode("utf-16-le")) // 2 + 1 + assert italic.length == link.length == len("live chart") and link.url == "https://x" -async def test_alerts_are_silent_and_dead_chats_are_forgotten(tmp_path): +async def test_alerts_are_silent_and_a_chat_that_blocked_the_bot_is_marked_gone(tmp_path): from telegram.error import Forbidden state = AlertState(tmp_path / "s.json") state.add_chat(1, "ok") @@ -293,7 +302,131 @@ async def send_message(self, chat_id, text, **kw): if chat_id == 2: raise Forbidden("bot was blocked") await Notifier(Bot(), state)("hi") - assert all(kw["disable_notification"] for _, kw in calls) and 2 not in state.chats and 1 in state.chats + assert all(kw["disable_notification"] for _, kw in calls) and state.chats[2]["gone"] and state.alert_chats() == [1] + + +async def test_an_alerts_chart_is_drawn_once_and_only_when_some_chat_gets_the_alert(tmp_path): + state = AlertState(tmp_path / "s.json") + for chat in (1, 2): + state.add_chat(chat, str(chat)) + state.set_kind(2, "uv", False) + drawn, sent = [], [] + + async def chart(): + drawn.append(1) + return b"png" + + class Bot: + async def send_photo(self, chat_id, photo, **kw): + sent.append((chat_id, photo)) + notifier = Notifier(Bot(), state) + assert await notifier("Gusts", kind="gusts", chart=chart) == 2 and drawn == [1] and [p for _, p in sent] == [b"png"] * 2 + assert await notifier("UV", kind="uv", chart=chart, chats=[2]) == 0 and drawn == [1] # chat 2 has UV alerts off: not drawn + assert await notifier("UV", kind="uv", chart=chart, chats=[1, 2]) == 1 and drawn == [1, 1] and sent[-1] == (1, b"png") + + +async def test_a_gone_chat_keeps_its_settings_gets_nothing_and_is_back_when_it_writes(tmp_path): + from telegram.error import BadRequest, Forbidden + state = AlertState(tmp_path / "s.json") + for chat in (1, 2, 3): + state.add_chat(chat, str(chat)) + state.set_kind(2, "rain", False) + calls, refusals = [], {2: Forbidden("bot was blocked by the user"), 3: BadRequest("Message thread not found")} + + class Bot: + async def send_message(self, chat_id, text, **kw): + calls.append(chat_id) + if chat_id in refusals: + raise refusals[chat_id] + await Notifier(Bot(), state)("hi", kind="uv") + assert state.chats[2]["gone"] and state.muted(2) == {"rain"} and state.present() == [1, 3] + assert not state.chats[3].get("gone") # only "chat not found" means gone: the chat is still there + calls.clear() + await Notifier(Bot(), state)("again", kind="uv") + assert calls == [1, 3] # nothing is sent to a gone chat + state.add_chat(2, "2") # it writes again + assert state.subscribed(2, "uv") and not state.subscribed(2, "rain") and AlertState(state.path).present() == [1, 2, 3] + + +def test_a_migrated_groups_forecast_moves_with_it(tmp_path): + from datetime import date + state = AlertState(tmp_path / "s.json") + state.add_chat(-5, "group") + today = date(2026, 10, 3) + state.record_forecast(-5, [{"date": today, "max_c": 20.0, "rain_chance_pct": 90}], today) + state.migrate_chat(-5, -1005) + assert state.forecast_sent(-1005)[today]["rain_chance_pct"] == 90 and state.forecast_sent(-5) == {} and state.forecast_chats() == [-1005] + + +def test_a_damaged_state_file_is_kept_aside_not_overwritten(tmp_path, caplog): + path = tmp_path / "s.json" + path.write_text('{"chats": {"1": {"title": "Home"}}, "monitor": {"rain": {"alerted_st') # cut short + state = AlertState(path) + assert state.chats == {} + state.add_chat(2, "Work") # the next save + aside = list(tmp_path.glob("s.json.damaged-*")) + assert len(aside) == 1 and aside[0].read_text().startswith('{"chats": {"1"') and "moved it to" in caplog.text + assert AlertState(path).chats == {2: {"title": "Work"}} + + +def test_a_save_is_on_disk_before_it_replaces_the_old_file(tmp_path, monkeypatch): + from lib.alerts import notify + calls, replace = [], notify.os.replace + monkeypatch.setattr(notify.os, "fsync", lambda fd: calls.append("fsync")) + monkeypatch.setattr(notify.os, "replace", lambda a, b: (calls.append("replace"), replace(a, b))) + AlertState(tmp_path / "s.json").add_chat(1, "Home") + assert calls == ["fsync", "replace"] + + +async def test_one_chat_failing_never_stops_the_alert_reaching_the_others(tmp_path, monkeypatch): + from telegram.error import Forbidden + state = AlertState(tmp_path / "s.json") + for chat in (1, 2, 3): + state.add_chat(chat, f"chat {chat}") + got = [] + + class Bot: + async def send_message(self, chat_id, text, **kw): + if chat_id == 1: + raise Forbidden("bot was blocked") + got.append(chat_id) + + def full(): + raise OSError(28, "No space left on device") + monkeypatch.setattr(state, "save", full) # forgetting chat 1 can't be saved + await Notifier(Bot(), state)("hi") + assert got == [2, 3] + + +@pytest.mark.filterwarnings("ignore:Deprecated since version v22.2") # python-telegram-bot 22: retry_after will be a timedelta +async def test_a_short_flood_wait_or_a_dropped_connection_is_tried_once_more_and_a_timeout_is_not(tmp_path, monkeypatch): + from telegram.error import NetworkError, RetryAfter, TimedOut + from lib.alerts import notify + waits = [] + + async def sleep(seconds): + waits.append(seconds) + monkeypatch.setattr(notify.asyncio, "sleep", sleep) + flood, long_flood = RetryAfter(5), RetryAfter(300) + state = AlertState(tmp_path / "s.json") + state.add_chat(1, "chat") + + class Bot: + def __init__(self, *errors): + self.errors, self.tries = list(errors), 0 + + async def send_message(self, chat_id, text, **kw): + self.tries += 1 + if self.errors: + raise self.errors.pop(0) + cases = [((flood,), True, 2, [5]), ((NetworkError("Connection reset by peer"),), True, 2, [3]), + ((NetworkError("down"), NetworkError("down")), False, 2, [3]), # once more, not twice + ((TimedOut(),), False, 1, []), # it may have arrived + ((long_flood,), False, 1, [])] # not waited out + for errors, sent, tries, waited in cases: + waits.clear() + bot = Bot(*errors) + assert (bool(await Notifier(bot, state)("hi", chats=[1])), bot.tries, waits) == (sent, tries, waited), errors def uv(*values, start=T0): @@ -467,12 +600,12 @@ async def test_no_rain_alerts_in_quiet_hours_and_one_summary_after_them(tmp_path m, state, sent = monitor(tmp_path, None) m.quiet = (0, 6) one_am = int(datetime(2026, 9, 30, 1, 0, tzinfo=TZ).timestamp()) - for data in (night_rain(one_am, 0, 0, 0), night_rain(one_am, 0, 0, 1), night_rain(one_am, 0, 0, 1, 1, 1), - night_rain(one_am, 0, 0, 1, 1, 1, *[0] * 13)): # it starts, rains, stops after 60 dry minutes - m.station.data = data + night = night_rain(one_am, 0, 0, 1, 1, 1, *[0] * 57) # 1am to 6:05am + for n in (3, 4, 6, 18, 60): # it starts, rains, stops after 60 dry minutes; 5:55 + m.station.data = night[:n] await m.check() assert sent == [] # all in quiet hours - m.station.data = night_rain(int(datetime(2026, 9, 30, 5, 55, tzinfo=TZ).timestamp()), 0, 0, 0) # 5:55 to 6:05 + m.station.data = night # 6:05 await m.check() assert len(sent) == 1 and sent[0].startswith("\U0001f327️ Overnight rain: 1.2 mm, from about 1:10am to 1:25am.") await m.check() @@ -482,6 +615,31 @@ async def test_no_rain_alerts_in_quiet_hours_and_one_summary_after_them(tmp_path assert len(sent) == 2 and "started raining" in sent[1] # by day, rain alerts are back +async def test_quiet_hours_across_midnight_keep_the_evening_rain_for_the_summary(tmp_path): + m, state, sent = monitor(tmp_path, None) + m.quiet = (22, 6) + evening = night_rain(int(datetime(2026, 9, 29, 22, 0, tzinfo=TZ).timestamp()), *[0] * 12, *[1] * 6, *[0] * 6) # 11pm, 2.4 mm + morning = night_rain(int(datetime(2026, 9, 30, 0, 0, tzinfo=TZ).timestamp()), *[0] * 12, *[1] * 3, *[0] * 59) # the gauge + night = evening + morning # reset at midnight; 1am, 1.2 mm; dry to 6:05am + for n in range(1, len(night) + 1): # a check every 5 minutes + m.station.data = night[:n] + await m.check() + assert sent == ["\U0001f327️ Overnight rain: 3.6 mm, from about 11:00pm to 1:15am."] + + +async def test_a_note_left_from_a_night_the_bot_missed_gives_way_to_the_next_nights(tmp_path): + m, state, sent = monitor(tmp_path, None) + m.quiet = (22, 6) + m.station.data = night_rain(int(datetime(2026, 9, 29, 23, 0, tzinfo=TZ).timestamp()), 0, 1) + await m.check() # rain noted at 11:05pm, then the bot is down until the next night + night = night_rain(int(datetime(2026, 9, 30, 22, 30, tzinfo=TZ).timestamp()), 0, 1, *[0] * 89) # to 6am + m.station.data = night[:2] + await m.check() + m.station.data = night + await m.check() + assert sent == ["\U0001f327️ Overnight rain: 0.4 mm, from about 10:35pm to 10:40pm."] + + async def test_a_dry_night_has_no_summary_and_quiet_hours_can_be_off(tmp_path): m, state, sent = monitor(tmp_path, None) m.quiet = (0, 6) @@ -512,25 +670,22 @@ def test_unsubscribing_from_all_keeps_future_alert_types_off_until_subscribed(tm assert not state.subscribed(1) # nothing on at all state.set_kind(1, "all", True) # subscribe to all: new types come on again assert state.subscribed(1, "newer") and state.subscribed(1, "rain") - for kind in list(menu.LABELS): # turning every type off one by one is not "unsubscribe from all" - state.set_kind(3, kind, False) + offered = ["rain", "gusts", "uv", "temps"] # a bot with the weather station alone + for kind in offered[:-1]: + state.set_kind(3, kind, False, offered) monkeypatch.setitem(menu.LABELS, "newest", "newest") - assert state.subscribed(3, "newest") + assert state.subscribed(3, "newest") and state.subscribed(3, "temps") # some still on: the normal mode + state.set_kind(3, "temps", False, offered) # every type the bot sends, off one by one: "unsubscribe from all" + monkeypatch.setitem(menu.LABELS, "newer still", "newer still") + assert not state.subscribed(3) and not state.subscribed(3, "newer still") + state.set_kind(3, "rain", True, offered) + again = notify.AlertState(state.path) # and it survives a restart + assert again.subscribed(3, "rain") and not again.subscribed(3, "uv") and not again.subscribed(3, "newer still") -def test_chats_that_already_unsubscribed_from_all_load_as_opt_in(tmp_path, monkeypatch): - import json - from lib.alerts import menu, notify - path = tmp_path / "s.json" - path.write_text(json.dumps({"chats": {"1": {"title": "a", "muted": list(menu.LABELS)}, "2": {"title": "b", "alerts": False}, - "3": {"title": "c", "muted": ["rain"]}}})) - state = notify.AlertState(path) - monkeypatch.setitem(menu.LABELS, "newthing", "new thing") - assert not state.subscribed(1, "newthing") and not state.subscribed(2, "newthing") and state.subscribed(3, "newthing") - state.set_kind(1, "uv", True) - state.save() - again = notify.AlertState(path) # and it survives a restart - assert again.subscribed(1, "uv") and not again.subscribed(1, "rain") and not again.subscribed(1, "newthing") +async def picture_of(kw): + """The picture an alert sent with these arguments would carry: what its chart draws (None: the text alone).""" + return await kw["chart"]() if kw.get("chart") else None def asking(mon, photo=b"png"): @@ -550,7 +705,7 @@ async def test_the_crossing_alert_carries_the_temperature_chart_when_there_is_on async def notify(text, link=None, **kw): sent.append(text) - photos.append(kw.get("photo")) + photos.append(await picture_of(kw)) m.notify = notify asked = asking(m) await m.check() # learns: outdoor cooler @@ -569,7 +724,7 @@ async def test_the_gusts_alert_carries_the_wind_chart(tmp_path): seen = [] async def notify(text, link=None, **kw): - seen.append((text, kw.get("photo"))) + seen.append((text, await picture_of(kw))) m.notify = notify asked = asking(m) await m.check() # the first look sets the mark @@ -581,20 +736,20 @@ async def notify(text, link=None, **kw): plain = [] async def notify2(text, link=None, **kw): - plain.append(kw) + plain.append((kw["kind"], await picture_of(kw))) m2.notify = notify2 asking(m2, photo=None) await m2.check() m2.station.data = [(T0, {"wind.wind_gust": 10}), (T0 + 300, {"wind.wind_gust": 12}), (T0 + 600, {"wind.wind_gust": 70})] await m2.check() - assert plain == [{"kind": "gusts"}] # no chart: text only + assert plain == [("gusts", None)] # no chart: text only async def test_both_particulates_alerts_carry_the_particulates_chart(tmp_path): state, seen = AlertState(tmp_path / "s.json"), [] async def notify(text, link=None, **kw): - seen.append((text, kw.get("photo"))) + seen.append((text, await picture_of(kw))) mon = ParticulatesMonitor(FakeAir(), state, notify) asked = asking(mon) for pm in (90, 90, 30, 30): @@ -618,7 +773,7 @@ async def notify(text, link=None, **kw): for data in (rain(0, 0, 0), rain(0, 0, 1)): m.station.data = data await m.check() - assert seen and all("photo" not in kw for kw in seen) + assert seen and all("chart" not in kw for kw in seen) async def test_the_alert_charts_are_the_typed_one_day_charts_drawn_from_each_sources_history(tmp_path, archived_cache): @@ -649,7 +804,7 @@ async def test_the_particulates_chart_draws_pm1_pm25_and_pm10_together(tmp_path) transport, _ = air_transport(oldest=datetime.now(timezone.utc) - timedelta(days=5)) air = AirGradient(config(tmp_path), transport=transport) await air.start() - read = intent.read("Particulates chart 1d", now_local(TZ), True, True) + read = intent.read("Particulates chart 1d", now_local(TZ)) turn = Turn(chart_asked=read.chart_asked, text="Particulates chart 1d") await air.handle(read.fast[1], turn) (panel,) = turn.charts[0].panels @@ -663,7 +818,7 @@ async def test_the_uv_alert_carries_the_uv_chart_once_per_spell(tmp_path): seen = [] async def notify(text, link=None, **kw): - seen.append((text, kw.get("photo"))) + seen.append((text, await picture_of(kw))) m.notify = notify asked = asking(m) await m.check() # the first look sets the mark @@ -678,13 +833,27 @@ async def notify(text, link=None, **kw): got = [] async def notify2(text, link=None, **kw): - got.append(kw) + got.append((kw["kind"], await picture_of(kw))) plain.notify = notify2 asking(plain, photo=None) await plain.check() plain.station.data = uv(3, 5, 11, 12) await plain.check() - assert got == [{"kind": "uv"}] # no chart: text only + assert got == [("uv", None)] # no chart: text only + + +def test_an_alert_rearms_only_once_its_condition_has_stopped_for_the_whole_wait(): + from lib.alerts import confirm + m = {"active": True, "calm_since": None} # it has fired + for now in (T0, T0 + 60): + confirm.rearm(m, False, now, 100) + assert m["active"] and m["calm_since"] == T0 + confirm.rearm(m, True, T0 + 90, 100) # the condition again: the calm starts over + for now in (T0 + 120, T0 + 200): + confirm.rearm(m, False, now, 100) + assert m["active"] # only 80 s of calm since T0 + 120 + confirm.rearm(m, False, T0 + 220, 100) + assert not m["active"] and m["calm_since"] is None # it can fire again def test_the_confirmation_rule_counts_distinct_readings_in_a_row(): @@ -764,11 +933,8 @@ async def notify(text, link=None, **kw): mon = ParticulatesMonitor(FakeAir(), state, notify) stamp = datetime.now(timezone.utc) - def reading(minutes, latest, before): - r = air_reading(latest) - r["_time_utc"] = stamp + timedelta(minutes=minutes) - r["_sustained"] = {"pm2_5": min(latest, before)} # what air.current() gives: the lowest of the last readings - return r + def reading(minutes, latest, before): # the lowest of the last readings, as air.latest() gives it + return Latest(air_reading(latest).reading, stamp + timedelta(minutes=minutes), {"pm2_5": min(latest, before)}) await mon.check(reading(0, 90, 5)) # a spike: the reading before it was clean await mon.check(reading(30, 90, 5)) # and again at the next check, still alone assert not sent and state.monitor["particulates"]["above"] == 0 diff --git a/tests/test_bot.py b/tests/test_bot.py index 0ae83a5..d3a0151 100644 --- a/tests/test_bot.py +++ b/tests/test_bot.py @@ -7,7 +7,7 @@ from lib import intent from lib.bot import Bot from lib.tools import Turn -from tests.fakes import TZ +from tests.fakes import TZ, StandInTools class RaisingAgent: @@ -16,12 +16,12 @@ async def run(self, *a, **k): class AnsweringAgent: - async def run(self, messages, system, effort, first_call=None, require_tool=True, no_tools=False, turn=None, **k): + async def run(self, messages, system, effort, *, first_call=None, turn=None, **k): self.first_call = first_call return "It is 12 degrees." -async def ask(agent, text="how hot is it"): +async def ask(alert_state, agent, text="how hot is it"): replies = [] async def reply_text(body, **kw): @@ -33,29 +33,29 @@ async def send_chat_action(*a, **k): chat=NS(type="private", title=None), from_user=NS(full_name="Rob"), reply_to_message=None) update = NS(effective_message=msg, effective_chat=msg.chat, effective_user=NS(username="rob", full_name="Rob")) context = NS(bot=NS(send_chat_action=send_chat_action, id=99)) - await Bot(NS(tz=TZ), agent, [], None).respond(update, context, text) + await Bot(NS(tz=TZ), agent, [], alert_state).respond(update, context, text) return replies -async def test_error_details_stay_in_the_log_not_the_chat(): - replies = await ask(RaisingAgent()) +async def test_error_details_stay_in_the_log_not_the_chat(alert_state): + replies = await ask(alert_state, RaisingAgent()) assert len(replies) == 1 and "went wrong" in replies[0] and "try again" in replies[0].lower() assert "SECRET" not in replies[0] and "Bearer" not in replies[0] and "acme" not in replies[0] assert "RuntimeError" in replies[0] # just the kind of problem -async def test_a_failing_shortcut_falls_back_to_the_model(monkeypatch): +async def test_a_failing_shortcut_falls_back_to_the_model(alert_state, monkeypatch): def boom(*a, **k): raise OverflowError("date value out of range") monkeypatch.setattr(intent, "fast_call", boom) agent = AnsweringAgent() - assert await ask(agent, "weather last 99999 years") == ["It is 12 degrees."] + assert await ask(alert_state, agent, "weather last 99999 years") == ["It is 12 degrees."] assert agent.first_call is None -def test_a_reply_carries_the_message_it_replies_to_in_private_chats_too(): +def test_a_reply_carries_the_message_it_replies_to_in_private_chats_too(alert_state): from types import SimpleNamespace as NS - bot = Bot(NS(tz=TZ), None, [], None) + bot = Bot(NS(tz=TZ), None, [], alert_state) mine = NS(text="Absolute pressure\n• Low: 992.9 hPa at 2:20pm on Sat 5 Sep", caption=None, from_user=NS(id=99, full_name="Bot")) theirs = NS(text="lovely day", caption=None, from_user=NS(id=7, full_name="Ann")) private = NS(chat=NS(type="private"), from_user=NS(full_name="Rob"), reply_to_message=mine) @@ -67,7 +67,7 @@ def test_a_reply_carries_the_message_it_replies_to_in_private_chats_too(): assert bot._content(NS(**{**group.__dict__, "reply_to_message": None}), "hi", 99) == "Rob: hi" -async def test_a_hung_question_is_given_up_on_so_the_next_one_in_the_chat_is_answered(monkeypatch, caplog): +async def test_a_hung_question_is_given_up_on_so_the_next_one_in_the_chat_is_answered(alert_state, monkeypatch, caplog): import asyncio from lib import bot @@ -78,7 +78,7 @@ async def run(self, *a, **k): monkeypatch.setattr(bot, "TURN_SECONDS", 0.3) monkeypatch.setattr(bot, "WATCHDOG_SECONDS", 0.1) with caplog.at_level("WARNING"): - replies = await ask(Hanging()) + replies = await ask(alert_state, Hanging()) assert len(replies) == 1 and "took too long" in replies[0] assert "Still working" in caplog.text and "Gave up" in caplog.text @@ -108,7 +108,7 @@ async def send_chat_action(self, *a, **k): class StreamingAgent: - async def run(self, messages, system, effort, first_call=None, require_tool=True, no_tools=False, turn=None, on_text=None, tool_names=None, on_heavy=None, conv_id=None): + async def run(self, messages, system, effort, *, on_text=None, **k): await asyncio.sleep(0.05) # "thinking" on_text("It is") await asyncio.sleep(0.05) @@ -117,7 +117,7 @@ async def run(self, messages, system, effort, first_call=None, require_tool=True return "It is 12 degrees." -async def ask_private(agent, bot, chat_type="private"): +async def ask_private(alert_state, agent, bot, chat_type="private"): replies = [] async def reply_text(body, **kw): @@ -125,21 +125,21 @@ async def reply_text(body, **kw): msg = NS(chat_id=1, message_thread_id=None, is_topic_message=False, reply_text=reply_text, chat=NS(type=chat_type, title=None), from_user=NS(full_name="Rob"), reply_to_message=None) update = NS(effective_message=msg, effective_chat=msg.chat, effective_user=NS(username="rob", full_name="Rob")) - await Bot(NS(tz=TZ), agent, [], None).respond(update, NS(bot=bot), "how hot is it") + await Bot(NS(tz=TZ), agent, [], alert_state).respond(update, NS(bot=bot), "how hot is it") return replies -async def test_a_private_chat_shows_a_thinking_draft_that_streams_the_answer_then_sends_it(monkeypatch): +async def test_a_private_chat_shows_a_thinking_draft_that_streams_the_answer_then_sends_it(alert_state, monkeypatch): monkeypatch.setattr("lib.bot.DRAFT_MIN_GAP", 0.01) bot = DraftBot() - replies = await ask_private(StreamingAgent(), bot) + replies = await ask_private(alert_state, StreamingAgent(), bot) assert replies == ["It is 12 degrees."] assert bot.drafts[0][1] is None # empty: "Thinking..." assert [t for _, t in bot.drafts][-1] == "It is 12 degrees." and len({d for d, _ in bot.drafts}) == 1 assert bot.typing >= 1 # "typing..." runs alongside the draft -async def test_an_idle_draft_is_refreshed_before_telegram_drops_it(monkeypatch): +async def test_an_idle_draft_is_refreshed_before_telegram_drops_it(alert_state, monkeypatch): monkeypatch.setattr("lib.bot.DRAFT_REFRESH_SECONDS", 0.02) class Slow: @@ -147,27 +147,27 @@ async def run(self, *a, on_text=None, **k): await asyncio.sleep(0.15) return "ok" bot = DraftBot() - await ask_private(Slow(), bot) + await ask_private(alert_state, Slow(), bot) assert len(bot.drafts) >= 4 and all(t is None for _, t in bot.drafts) -async def test_a_refused_draft_falls_back_to_typing_and_the_reply_still_arrives(): +async def test_a_refused_draft_falls_back_to_typing_and_the_reply_still_arrives(alert_state): bot = DraftBot(fail=True) - assert await ask_private(StreamingAgent(), bot) == ["It is 12 degrees."] + assert await ask_private(alert_state, StreamingAgent(), bot) == ["It is 12 degrees."] assert bot.drafts == [] and bot.typing >= 1 -async def test_groups_never_get_drafts(): +async def test_groups_never_get_drafts(alert_state): bot = DraftBot() - assert await ask_private(AnsweringAgent(), bot, chat_type="supergroup") == ["It is 12 degrees."] + assert await ask_private(alert_state, AnsweringAgent(), bot, chat_type="supergroup") == ["It is 12 degrees."] assert bot.drafts == [] -async def test_a_generation_stopped_update_is_logged(caplog): +async def test_a_generation_stopped_update_is_logged(alert_state, caplog): update = NS(effective_message=NS(api_kwargs={"message_generation_stopped": {"draft_id": 1}}, chat_id=1), effective_chat=NS(type="private", id=1, title=None), effective_user=NS(username="rob", full_name="Rob")) with caplog.at_level("INFO", logger="lib.bot"): - await Bot(NS(tz=TZ), None, [], None).on_other(update, None) + await Bot(NS(tz=TZ), None, [], alert_state).on_other(update, None) assert "generation stopped" in caplog.text and "message_generation_stopped" in caplog.text @@ -188,10 +188,10 @@ async def reply_text(body, **kw): assert sent == [("text", "Fri was the hottest day."), ("photo", "Temperature")] -async def test_a_report_is_written_in_code_from_the_fast_path_calls_and_the_model_is_not_asked(): +async def test_a_report_is_written_in_code_from_the_fast_path_calls_and_the_model_is_not_asked(alert_state): import json - class Tools: + class Tools(StandInTools): def __init__(self): self.called = [] @@ -216,7 +216,7 @@ async def reply_text(body, **kw): async def send_chat_action(*a, **k): pass - bot = Bot(NS(tz=TZ), agent, sources, None) + bot = Bot(NS(tz=TZ), agent, sources, alert_state) await bot.respond(update, NS(bot=NS(send_chat_action=send_chat_action, id=99)), "report") await bot.respond(update, NS(bot=NS(send_chat_action=send_chat_action, id=99)), "weather now") assert agent.tools.called == ["weather_now", "air_quality", "weather_now"] @@ -226,20 +226,20 @@ async def send_chat_action(*a, **k): assert history[-1] == {"role": "assistant", "content": replies[1]} and history[-4]["content"] == "report" -async def test_a_question_that_times_out_is_asked_again_one_reasoning_step_lower(monkeypatch): +async def test_a_question_that_times_out_is_asked_again_one_reasoning_step_lower(alert_state, monkeypatch): from lib import bot as botmod monkeypatch.setattr(botmod, "TURN_SECONDS", 0.05) monkeypatch.setattr(botmod, "RETRY_SECONDS", 0.5) efforts = [] class SlowThenQuick: - async def run(self, messages, system, effort, first_call=None, require_tool=True, no_tools=False, turn=None, on_text=None, tool_names=None, on_heavy=None, conv_id=None): + async def run(self, messages, system, effort, **k): efforts.append(effort) messages.append({"role": "assistant", "content": "(unfinished)"}) if effort == "medium": await asyncio.sleep(10) return f"answered at {effort}" - bot = Bot(NS(tz=TZ), SlowThenQuick(), [], None) + bot = Bot(NS(tz=TZ), SlowThenQuick(), [], alert_state) working = [{"role": "user", "content": "q"}] read = intent.read("what do you think about the weather today?", datetime(2026, 10, 2, 12, 0)) assert read.effort == "medium" @@ -247,7 +247,7 @@ async def run(self, messages, system, effort, first_call=None, require_tool=True assert efforts == ["medium", "low"] and working == [{"role": "user", "content": "q"}, {"role": "assistant", "content": "(unfinished)"}] -async def test_a_question_with_no_lower_step_or_a_second_timeout_gives_up(monkeypatch): +async def test_a_question_with_no_lower_step_or_a_second_timeout_gives_up(alert_state, monkeypatch): import pytest from lib import bot as botmod monkeypatch.setattr(botmod, "TURN_SECONDS", 0.05) @@ -258,7 +258,7 @@ class Hangs: async def run(self, messages, system, effort, **k): efforts.append(effort) await asyncio.sleep(10) - bot = Bot(NS(tz=TZ), Hangs(), [], None) + bot = Bot(NS(tz=TZ), Hangs(), [], alert_state) for text, expected in (("hello there", ["none"]), ("what do you think about the weather today?", ["medium", "low"])): efforts.clear() with pytest.raises(TimeoutError): @@ -266,12 +266,12 @@ async def run(self, messages, system, effort, **k): assert efforts == expected -async def test_a_plain_chart_request_is_drawn_and_captioned_in_code_and_the_model_is_not_asked(monkeypatch): +async def test_a_plain_chart_request_is_drawn_and_captioned_in_code_and_the_model_is_not_asked(alert_state, monkeypatch): import json from lib import bot as botmod from lib.specs import Chart, Line, Panel - class Tools: + class Tools(StandInTools): calls = [] async def call(self, name, raw, turn=None): @@ -298,20 +298,20 @@ async def reply_photo(photo, caption=None, **kw): async def send_chat_action(*a, **k): pass - await Bot(NS(tz=TZ), Agent(), sources, None).respond(update, NS(bot=NS(send_chat_action=send_chat_action, id=99)), "Humidity chart 7d") + await Bot(NS(tz=TZ), Agent(), sources, alert_state).respond(update, NS(bot=NS(send_chat_action=send_chat_action, id=99)), "Humidity chart 7d") name, args = Agent.tools.calls[0] assert name == "weather_history" and args["chart"] and args["groups"] == "outdoor,indoor" assert sent == ["Humidity"] # the chart's title is its whole caption -async def test_a_chart_made_in_code_starts_no_refresh_and_other_questions_do(monkeypatch): +async def test_a_chart_made_in_code_starts_no_refresh_and_other_questions_do(alert_state, monkeypatch): """The chart's own reads go first in line; a refresh started beside them would only queue requests ahead of the next ones.""" import json from lib import bot as botmod from lib.specs import Chart, Line, Panel pokes = [] - class Tools: + class Tools(StandInTools): async def call(self, name, raw, turn=None): turn.charts.append(Chart("t", "", [Panel("Temperature", "°C", [Line("Outdoor", [1, 2], [1.0, 2.0])])])) return json.dumps({"period": "Fri 25 Sep 2026 - Thu 01 Oct 2026", "series": { @@ -333,14 +333,14 @@ async def reply(*a, **k): async def send_chat_action(*a, **k): pass - bot, context = Bot(NS(tz=TZ), Agent(), sources, None), NS(bot=NS(send_chat_action=send_chat_action, id=99)) + bot, context = Bot(NS(tz=TZ), Agent(), sources, alert_state), NS(bot=NS(send_chat_action=send_chat_action, id=99)) await bot.respond(update, context, "Temperature chart 7d") assert pokes == [] await bot.respond(update, context, "what do you think about the weather today?") assert pokes == [1] -async def test_a_chart_reply_logs_how_long_each_step_took(tmp_path, monkeypatch, caplog): +async def test_a_chart_reply_logs_how_long_each_step_took(alert_state, tmp_path, monkeypatch, caplog): """One line per chart reply: reading the data, the calculation, drawing, the PNG (and its size) and sending it.""" import re from lib.ecowitt import Ecowitt, api @@ -368,7 +368,7 @@ async def reply_photo(photo, caption=None, **kw): async def send_chat_action(*a, **k): pass with caplog.at_level("DEBUG"): - await Bot(NS(tz=TZ), Agent(), sources, None).respond(update, NS(bot=NS(send_chat_action=send_chat_action, id=99)), + await Bot(NS(tz=TZ), Agent(), sources, alert_state).respond(update, NS(bot=NS(send_chat_action=send_chat_action, id=99)), "Temperature chart 1d") await eco.close() lines = [r.getMessage() for r in caplog.records if r.getMessage().startswith("Chart ms:")] @@ -377,11 +377,11 @@ async def send_chat_action(*a, **k): assert re.fullmatch(rf"Chart ms: data \d+, calc \d+, draw \d+, PNG \d+ \({kb} KB\), send \d+", lines[0]) -async def _ask_with(tool_result, text, monkeypatch=None): +async def _ask_with(alert_state, tool_result, text, monkeypatch=None): """Ask `text` of a bot whose one tool returns `tool_result`; returns (replies, whether the model was asked).""" asked = [] - class Tools: + class Tools(StandInTools): async def call(self, name, raw, turn=None): return tool_result @@ -402,29 +402,29 @@ async def reply_text(body, **kw): async def send_chat_action(*a, **k): pass - await Bot(NS(tz=TZ), Agent(), sources, None).respond(update, NS(bot=NS(send_chat_action=send_chat_action, id=99)), text) + await Bot(NS(tz=TZ), Agent(), sources, alert_state).respond(update, NS(bot=NS(send_chat_action=send_chat_action, id=99)), text) return replies, bool(asked) -async def test_a_plain_reading_question_is_answered_in_code_without_the_model(): +async def test_a_plain_reading_question_is_answered_in_code_without_the_model(alert_state): import json result = json.dumps({"outdoor": {"temperature": "12.3 ℃", "feels_like": "11.0 ℃"}, "indoor": {"temperature": "18.3 ℃"}}) - replies, asked = await _ask_with(result, "how hot is it") + replies, asked = await _ask_with(alert_state, result, "how hot is it") assert not asked and replies == ["• Outdoor: 12.3 °C, feels like 11.0 °C\n• Indoor: 18.3 °C"] -async def test_a_lookup_that_cannot_be_written_falls_back_to_the_model(): +async def test_a_lookup_that_cannot_be_written_falls_back_to_the_model(alert_state): import json - replies, asked = await _ask_with(json.dumps({"error": "station not answering"}), "how hot is it") + replies, asked = await _ask_with(alert_state, json.dumps({"error": "station not answering"}), "how hot is it") assert asked and replies == ["from the model"] -async def test_about_the_bot_is_answered_in_code_without_the_model(): - replies, asked = await _ask_with("{}", "what can you do") +async def test_about_the_bot_is_answered_in_code_without_the_model(alert_state): + replies, asked = await _ask_with(alert_state, "{}", "what can you do") assert not asked and replies[0].startswith("• Temperature, humidity") -async def test_typing_starts_before_the_model_is_asked_and_runs_alongside_the_draft(): +async def test_typing_starts_before_the_model_is_asked_and_runs_alongside_the_draft(alert_state): seen = {} class Agent: @@ -433,11 +433,11 @@ async def run(self, *a, **k): await asyncio.sleep(0.05) return "ok" bot = DraftBot() - assert await ask_private(Agent(), bot) == ["ok"] + assert await ask_private(alert_state, Agent(), bot) == ["ok"] assert seen["typing_when_asked"] >= 1 and bot.drafts -async def test_the_forecast_is_fetched_with_the_first_step_of_a_rain_ahead_question(): +async def test_the_forecast_is_fetched_with_the_first_step_of_a_rain_ahead_question(alert_state): got = [] class Agent: @@ -455,11 +455,11 @@ async def reply_text(body, **kw): async def send_chat_action(*a, **k): pass - await Bot(NS(tz=TZ), Agent(), sources, None).respond(update, NS(bot=NS(send_chat_action=send_chat_action, id=99)), "will it rain?") + await Bot(NS(tz=TZ), Agent(), sources, alert_state).respond(update, NS(bot=NS(send_chat_action=send_chat_action, id=99)), "will it rain?") assert [name for name, _ in got[0]] == ["weather_now", "weather_history", "weather_forecast"] and replies == ["Possibly."] -async def _message_aged(seconds): +async def _message_aged(alert_state, seconds): from datetime import datetime, timedelta, timezone answered = [] @@ -478,22 +478,22 @@ async def send_chat_action(*a, **k): date=datetime.now(timezone.utc) - timedelta(seconds=seconds), chat=NS(type="private", title=None, id=1), from_user=NS(full_name="Rob"), reply_to_message=None) update = NS(effective_message=msg, effective_chat=msg.chat, effective_user=NS(username="rob", full_name="Rob")) - bot = Bot(NS(tz=TZ), Agent(), [], None) + bot = Bot(NS(tz=TZ), Agent(), [], alert_state) await bot.on_message(update, NS(bot=NS(send_chat_action=send_chat_action, id=99, username="b"))) return replies -async def test_a_message_sent_while_the_bot_was_starting_is_answered(caplog): - assert await _message_aged(120) == ["ok"] +async def test_a_message_sent_while_the_bot_was_starting_is_answered(alert_state, caplog): + assert await _message_aged(alert_state, 120) == ["ok"] -async def test_a_message_that_waited_hours_is_ignored_and_logged(caplog): +async def test_a_message_that_waited_hours_is_ignored_and_logged(alert_state, caplog): with caplog.at_level("INFO", logger="lib.bot"): - assert await _message_aged(2 * 3600) == [] + assert await _message_aged(alert_state, 2 * 3600) == [] assert "Ignored a message 120 minutes old" in caplog.text -async def test_a_heavy_question_gets_a_heads_up_in_the_chat_before_the_answer_and_the_answer_still_comes(): +async def test_a_heavy_question_gets_a_heads_up_in_the_chat_before_the_answer_and_the_answer_still_comes(alert_state): class Agent: async def run(self, messages, system, effort, on_heavy=None, **k): await on_heavy(24000) @@ -509,33 +509,33 @@ async def reply_text(body, **kw): async def send_chat_action(*a, **k): pass - await Bot(NS(tz=TZ), Agent(), sources, None).respond(update, NS(bot=NS(send_chat_action=send_chat_action, id=99)), "is it too cold to run") + await Bot(NS(tz=TZ), Agent(), sources, alert_state).respond(update, NS(bot=NS(send_chat_action=send_chat_action, id=99)), "is it too cold to run") assert replies == ["\u26a0\ufe0f That's pulling in about 24k tokens of data, so it will take a little longer. Working on it...", "The answer."] -async def test_a_days_over_a_limit_question_is_counted_in_code_with_no_model(): +async def test_a_days_over_a_limit_question_is_counted_in_code_with_no_model(alert_state): import json result = json.dumps({"period": "Wed 5 Oct 2022 - Sat 3 Oct 2026", "days_checked": 1454, "matching_days": 137, "condition": "UV index ≥ 10"}) - replies, asked = await _ask_with(result, "days over UVI 10 by year") + replies, asked = await _ask_with(alert_state, result, "days over UVI 10 by year") assert not asked and replies == ["137 of 1,454 days with UV index ≥ 10 (Wed 5 Oct 2022 – Sat 3 Oct 2026)."] -async def test_rain_by_month_is_worked_out_in_code_with_no_model(): +async def test_rain_by_month_is_worked_out_in_code_with_no_model(alert_state): import json result = json.dumps({"period": "Fri 4 Oct 2024 - Sat 3 Oct 2026", "days_checked": 731, "matching_days": 731, "stat": {"what": "Total rain", "of": "rain", "unit": "mm"}, "value": 1234.5}) - replies, asked = await _ask_with(result, "rain by month for 2 years") + replies, asked = await _ask_with(alert_state, result, "rain by month for 2 years") assert not asked and replies == ["Total rain: 1234.5 mm over 731 days (Fri 4 Oct 2024 – Sat 3 Oct 2026)."] -async def test_every_model_call_sends_the_full_tool_list_not_a_per_question_one(): +async def test_the_model_is_asked_with_the_tool_choice_the_message_calls_for(alert_state): seen = [] class Agent: async def run(self, messages, system, effort, **k): - seen.append(k.get("tool_names", "all")) + seen.append(k["tool_choice"]) return "ok" - for text in ("is it too cold to run", "which air quality day was worst when it also rained"): + for text in ("will it rain tomorrow?", "thanks!"): replies = [] async def reply_text(body, **kw): @@ -547,16 +547,16 @@ async def reply_text(body, **kw): async def send_chat_action(*a, **k): pass - await Bot(NS(tz=TZ), Agent(), sources, None).respond(update, NS(bot=NS(send_chat_action=send_chat_action, id=99)), text) - assert seen == ["all", "all"] + await Bot(NS(tz=TZ), Agent(), sources, alert_state).respond(update, NS(bot=NS(send_chat_action=send_chat_action, id=99)), text) + assert seen == ["required", "auto"] -async def test_weather_all_week_and_a_plain_chart_are_sent_with_just_their_title(monkeypatch): +async def test_weather_all_week_and_a_plain_chart_are_sent_with_just_their_title(alert_state, monkeypatch): import json from lib import bot as botmod from lib.specs import Chart, Line, Panel - class Tools: + class Tools(StandInTools): async def call(self, name, raw, turn=None): turn.charts.append(Chart("t", "", [Panel("Temperature", "°C", [Line("Outdoor", [1, 2], [1.0, 2.0])])])) return json.dumps({"period": "Fri 25 Sep 2026 - Thu 01 Oct 2026", "series": { @@ -581,15 +581,15 @@ async def reply_photo(photo, caption=None, **kw): async def send_chat_action(*a, **k): pass - await Bot(NS(tz=TZ), Agent(), sources, None).respond(update, NS(bot=NS(send_chat_action=send_chat_action, id=99)), text) + await Bot(NS(tz=TZ), Agent(), sources, alert_state).respond(update, NS(bot=NS(send_chat_action=send_chat_action, id=99)), text) assert sent == ["t"], text -async def test_the_report_carries_a_refresh_button_that_redraws_it_in_place(): +async def test_the_report_carries_a_refresh_button_that_redraws_it_in_place(alert_state): import json from lib import report_menu - class Tools: + class Tools(StandInTools): reading = "12.3 ℃" async def call(self, name, raw, turn=None): @@ -620,7 +620,9 @@ async def reply_text(body, **kw): async def send_chat_action(*a, **k): pass context = NS(bot=NS(send_chat_action=send_chat_action, id=99)) - bot = Bot(NS(tz=TZ), Agent(), sources, None) + bot = Bot(NS(tz=TZ), Agent(), sources, alert_state) + alert_state.add_chat(1, "Rob") + alert_state.set_keyboard(1, bot.keyboard_version) # its buttons are current, so a reply carries only its own await bot.respond(update, context, "report") await bot.respond(update, context, "weather now") assert [b.callback_data for b in sent[0][1].inline_keyboard[0]] == [report_menu.REFRESH] and sent[1][1] is None diff --git a/tests/test_chart_labels.py b/tests/test_chart_labels.py index 7af43f9..0e4e4d9 100644 --- a/tests/test_chart_labels.py +++ b/tests/test_chart_labels.py @@ -14,6 +14,8 @@ def banded(): assert banded().unbanded(7).low is None and banded().unbanded(SHORT_DAYS).high is None assert banded().unbanded(7, keep=True).low is not None # wind: the mean shaded up to the gusts assert banded().unbanded(SHORT_DAYS + 1).low is not None # a quarter keeps its daily range + line = banded() + assert line.unbanded(7).low is None and line.low is not None # a new line: the one it came from keeps its range def labels_of(chart): @@ -109,3 +111,41 @@ def test_two_lows_that_end_together_at_the_right_edge_get_pills_on_opposite_side Line("Other", xs, floor, reading="humidity")], reading="humidity")]) sides = dict(annotations_of(chart)) assert {sides["45.3 %"], sides["48.0 %"]} == {"top", "bottom"}, sides # never the same side + + +def test_a_temperature_low_just_above_zero_is_labelled(): + """Only a reading that can't go below zero (rain, wind, sun, air) has its axis floored at 0 and its low on the floor left + unlabelled; a cold night of 0.4° is a record worth its label.""" + ys = [12.0, 8.0, 0.4, 6.0, 15.0, 20.0, 18.0] + chart = Chart("T", "s", [Panel("Temperature", "°C", [one_line("temperature", ys).panels[0].lines[0]], reading="temperature")]) + assert "0.4°" in labels_of(chart) + assert "0" not in labels_of(one_line("wind", [3.0, 0.0, 5.0, 9.0])) # a calm low on the floor still says nothing + + +def test_a_day_of_25_hours_is_drawn_25_hours_long_with_its_ticks_on_local_hours(): + """5 Apr 2026 in Melbourne: 2am to 3am happens twice. The axis is real time, so a line through it never runs backwards (on + local clock times it drew that hour twice), and the hour ticks still fall on local 0, 3, 6 ... o'clock.""" + from datetime import datetime + from zoneinfo import ZoneInfo + import matplotlib.dates as mdates + import matplotlib.pyplot as plt + import numpy as np + tz = ZoneInfo("Australia/Melbourne") + start = int(datetime(2026, 4, 5, tzinfo=tz).timestamp()) + xs = charts._nums(range(start, start + 25 * 3600, 1800)) + assert np.all(np.diff(xs) > 0) and round((xs[-1] - xs[0]) * 24, 3) == 24.5 + fig, ax = plt.subplots() + try: + ax.set_xlim(xs[0], xs[-1]) + charts._time_axis(ax, xs[-1] - xs[0], tz) + ticks = [mdates.num2date(t, tz) for t in ax.xaxis.get_major_locator()()] + finally: + plt.close(fig) + assert ticks and {t.hour for t in ticks} <= {0, 3, 6, 9, 12, 15, 18, 21} and ticks[0].day == 5 + + +def test_every_weather_reading_and_air_metric_has_a_colour_of_its_own(): + from lib import charts + from lib.series import AIR, WEATHER + lines = [name for name in WEATHER if name != "rain"] # rain is drawn as bars, in its own blue + assert not [name for name in (*lines, *AIR) if name not in charts.READING_COLOURS] diff --git a/tests/test_compose.py b/tests/test_compose.py index 812a875..e32008d 100644 --- a/tests/test_compose.py +++ b/tests/test_compose.py @@ -56,6 +56,21 @@ async def test_a_reading_with_its_zones_over_rain_is_one_stacked_chart(tmp_path, await eco.close() +async def test_the_air_panels_share_one_load_and_a_weather_chart_asks_the_sensor_nothing(tmp_path, archived_cache): + comp, eco = await composer(tmp_path, archived_cache) + loads, rows = [], comp.air.rows + + async def counted(t0, t1): + loads.append((t0, t1)) + return await rows(t0, t1) + comp.air.rows = counted + out, _ = await plot(comp, panels=[{"series": "pm2_5"}, {"series": "pm10", "style": "rating"}, {"series": "rain"}]) + assert "chart" in out and len(loads) == 1 + out, _ = await plot(comp, panels=[{"series": "temperature"}]) + assert "chart" in out and len(loads) == 1 + await eco.close() + + async def test_the_traffic_light_rating_is_the_share_of_each_days_readings(tmp_path, archived_cache): comp, eco = await composer(tmp_path, archived_cache) out, specs = await plot(comp, panels=[{"series": "pm2_5", "style": "rating"}, {"series": "rain", "style": "bars"}]) @@ -124,6 +139,13 @@ def test_air_that_clears_after_every_rain_is_a_yes_and_air_that_ignores_rain_is_ assert analyse_air(air, rain, limits, TZ, "PM2.5")["verdict"].startswith("Yes, the other way") +def test_air_that_reads_zero_when_dry_and_events_that_change_nothing_are_counted_as_such(): + air, rain = synthetic(60, lambda washed: 0.0) # a clean sensor: zero before and after every rain + out = analyse_air(air, rain, (9.0, 55.4), TZ, "PM2.5") + assert out["events"]["cleaner_after"] == out["events"]["worse_after"] == 0 and out["events"]["count"] >= 15 + assert out["findings"][0].endswith("when dry (good)") and f"{out['events']['count']} about the same" in out["findings"][1] + + def test_too_few_rain_events_or_no_overlap_says_so(): air, rain = synthetic(7, lambda washed: 10.0 if washed else 30.0) # two events in a week assert analyse_air(air, rain, (9.0, 55.4), TZ, "PM2.5")["verdict"].startswith("Not enough") diff --git a/tests/test_days.py b/tests/test_days.py index b51f40d..062fd95 100644 --- a/tests/test_days.py +++ b/tests/test_days.py @@ -229,6 +229,16 @@ async def test_one_known_day_can_be_looked_up_with_no_conditions(station): assert row["rain"] == truth(day)[1] and row["temp_max"] == round(truth(day)[0], 1) +async def test_with_nothing_named_days_rank_hottest_first_and_a_figure_per_month_lists_no_days(station): + eco, _ = station + first, last = dates(eco, 30, 1) + days = [first + timedelta(days=k) for k in range((last - first).days + 1)] + out = await ask(eco, start_date=str(first), end_date=str(last), limit=3) + assert [r["date"] for r in out["days"]] == [label(d) for d in sorted(days, key=lambda d: -truth(d)[0])[:3]] + monthly = await ask(eco, start_date=str(first), end_date=str(last), stat="sum", of="rain", group_by="month") + assert monthly["by_month"] and "days" not in monthly # the figures alone, count_only or not + + async def test_days_can_be_counted_by_any_reading_not_only_temperature_rain_and_gusts(station): eco, fake = station first, last = dates(eco, 10, 1) @@ -363,5 +373,35 @@ def test_the_field_list_is_in_the_tool_schema_once_and_a_wrong_sort_or_of_field_ props = days.PARAMETERS["properties"] assert "enum" in props["where"]["items"]["properties"]["field"] and "enum" not in props["sort_by"] and "enum" not in props["of"] from lib.daytable import validate - units = {n: f[3] for n, f in days.FIELDS.items()} + units = days.FIELD_UNITS assert "sort_by must be one of" in validate({"sort_by": "nonsense"}, units) and "needs `of`" in validate({"stat": "sum", "of": "nonsense"}, units) + assert "of must be one of" in validate({"stat": "count", "of": "nonsense"}, units) # a count needs no `of`, but a wrong one is caught + + +async def test_a_wrong_of_with_a_plain_count_is_an_error_not_a_crash(station): + eco, _ = station + first, last = dates(eco, 10, 1) + out = await ask(eco, start_date=str(first), end_date=str(last), of="nonsense") + assert "of must be one of" in out["error"] + + +def test_the_description_names_every_field_and_which_have_a_lowest(): + """The model learns the fields from the description. A reading whose lowest says nothing (0 every night or calm hour) has no _min.""" + import re + from lib.ecowitt import days + both, max_only = (re.search(rf"(\w+(?:, \w+)*) \(_max {kind}\)", days.DESCRIPTION)[1].split(", ") for kind in ("and _min", "only")) + assert all(f"{n}_max" in days.FIELDS and f"{n}_min" in days.FIELDS for n in both) + assert all(f"{n}_max" in days.FIELDS and f"{n}_min" not in days.FIELDS for n in max_only) + assert {n.removesuffix("_max").removesuffix("_min").removesuffix("_avg") for n in days.FIELDS} == {"temp", "rain", "wind_gust", *both, *max_only} + + +async def test_a_figure_per_month_with_no_matching_day_promises_no_chart(station): + """Highest temperature per month on days over 99°: every month is None, so there is no bar to draw and the model must not be + told a chart is attached.""" + from lib.tools import Turn + eco, _ = station + first, last = dates(eco, 70, 1) + turn = Turn() + out = json.loads(await eco.tools[2].handler(dict(start_date=str(first), end_date=str(last), count_only=True, group_by="month", + stat="max", of="temp_max", where=[{"field": "temp_max", "op": ">", "value": 99}]), turn)) + assert out["by_month"] and all(v is None for v in out["by_month"].values()) and "chart" not in out and turn.charts == [] diff --git a/tests/test_ecowitt.py b/tests/test_ecowitt.py index 2ce42a0..7bc76c4 100644 --- a/tests/test_ecowitt.py +++ b/tests/test_ecowitt.py @@ -26,6 +26,38 @@ async def test_finds_the_station(station): assert "Fairleigh" in eco.describe() and "created" in eco.describe() +async def test_a_station_that_gives_no_longitude_has_none_not_melbournes(tmp_path): + import httpx + + class NoLongitude(FakeEcowitt): + def __call__(self, request): + response = super().__call__(request) + if request.url.path.endswith("/list"): + data = response.json() + del data["data"]["list"][0]["longitude"] + return httpx.Response(200, json=data) + return response + transport, _ = ecowitt_transport(NoLongitude()) + eco = Ecowitt(config(tmp_path), transport=transport) + await eco.start() + await eco.close() + assert eco.latitude == -37.8 and eco.longitude is None # so the forecast is offered no half location + + +@pytest.mark.parametrize("zone, used", [("Australia/Perth", "Australia/Perth"), (None, "Australia/Melbourne"), + ("Mars/Olympus", "Australia/Melbourne")]) +async def test_the_station_names_its_own_time_zone(tmp_path, caplog, zone, used): + """The device list's date_zone_id is the station's zone, whatever the configured one (Melbourne here); one Python doesn't + know, or none, leaves the configured zone.""" + caplog.set_level("INFO") + transport, _ = ecowitt_transport(FakeEcowitt(zone=zone)) + eco = Ecowitt(config(tmp_path), transport=transport) + await eco.start() + await eco.close() + assert str(eco.tz) == used and str(eco.created.tzinfo) == used and f"(AA:BB:CC:DD:EE:FF), {used}" in caplog.text + assert ("Ecowitt names a time zone Python doesn't know, 'Mars/Olympus'" in caplog.text) == (zone == "Mars/Olympus") + + @pytest.mark.parametrize("msg", ["System is busy.", "Operation too frequent"]) async def test_busy_and_too_frequent_are_retried(monkeypatch, msg): import httpx @@ -84,6 +116,8 @@ async def test_history_gives_extremes_with_times_and_days(station): s = result["series"]["outdoor.temperature"] assert set(s) >= {"low", "high", "low_when", "high_when", "low_date", "high_date", "daily"} assert len(s["daily"]) == 3 + assert list(s["daily"]) == [f"{d:%a} {d.day} {d:%b}" for d in (start, start + timedelta(days=1), end)] # "Sat 3 Oct", never "Sat 03 Oct" + assert result["period"] == f"{start:%a} {start.day} {start:%b %Y} - {end:%a} {end.day} {end:%b %Y}" assert float(s["high"]) > float(s["low"]) assert s["high_when"].startswith(("at ", "around ")) # refined to a 5-minute or 30-minute reading @@ -283,6 +317,8 @@ def monthly(out): spec = turn.charts[0] after = monthly(out) assert len(after) >= 6 and all({"low_when", "low_date", "high_when", "high_date"} <= set(m) for m in after.values()) + assert all(float(m["low"]) < float(m["high"]) for m in after.values()) # each month's records refined in place + assert all(f"{datetime.strptime(month, '%b %Y'):%b %Y}" == month for month in after) # a row per month: "Oct 2026" assert "monthly_note" not in json.loads(out) and fake.calls == [] assert "daily averages, range shaded" in spec.subtitle # 200 days: a point a day from the cached 30-minute data line = spec.panels[0].lines[0] @@ -456,6 +492,97 @@ async def test_http_errors_are_refused_only_when_ecowitt_answers_with_a_reason(t aio.sleep = orig +async def test_the_api_keys_never_reach_an_error_message(monkeypatch): + """Ecowitt takes its keys in the URL: a gateway page or a network error must not carry them into the error, which the + model is shown and the log keeps.""" + import httpx + real_sleep = asyncio.sleep + monkeypatch.setattr(ecowitt_api.asyncio, "sleep", lambda s: real_sleep(0)) # a 502 is asked again: without the waits + + def gateway(request): + return httpx.Response(502, text="Bad gateway") + + def unreachable(request): + raise httpx.ConnectError(f"can't reach {request.url}", request=request) + for handler, said in ((gateway, "HTTP 502"), (unreachable, "can't reach")): + api = ecowitt_api.EcowittAPI("SECRET-API", "SECRET-APP", httpx.MockTransport(handler)) + with pytest.raises(ecowitt_api.EcowittError) as err: + await api.devices() + await api.close() + assert said in str(err.value) and "SECRET" not in str(err.value), str(err.value) + + +async def test_a_failed_history_request_tells_the_model_and_the_log_no_key(tmp_path, caplog, monkeypatch): + import httpx + real_sleep = asyncio.sleep + monkeypatch.setattr(ecowitt_api.asyncio, "sleep", lambda s: real_sleep(0)) + fake = FakeEcowitt() + transport = httpx.MockTransport(lambda r: httpx.Response(502, text="Bad gateway") + if r.url.path.endswith("/history") else fake(r)) + eco = Ecowitt(config(tmp_path, ecowitt_api_key="SECRET-API", ecowitt_app_key="SECRET-APP"), transport=transport) + await eco.start() + day = datetime.now(eco.tz).date() - timedelta(days=3) + out = await eco.tools[1].handler({"groups": "outdoor", "start_date": f"{day} 00:00:00", "end_date": f"{day} 23:59:59"}, Turn()) + await eco.close() + assert "HTTP 502" in out and "SECRET" not in out and "SECRET" not in caplog.text + + +async def test_a_server_error_is_asked_again_and_the_turn_is_free_while_it_waits(monkeypatch): + """A 503 is asked again after a pause, like "System is busy."; meanwhile the turn is free, so another request (a question's) + goes ahead instead of waiting behind the pause.""" + import httpx + order, first = [], {"list": True} + + def handler(request): + path = request.url.path.rsplit("/", 1)[-1] + order.append(path) + if path == "list" and first.pop("list", False): + return httpx.Response(503, text="Service unavailable") + return httpx.Response(200, json={"code": 0, "data": {"list": [{"mac": "x"}]} if path == "list" else {}}) + real_sleep = asyncio.sleep + monkeypatch.setattr(ecowitt_api.asyncio, "sleep", lambda s: real_sleep(0.05 if s >= 3 else 0)) + api = ecowitt_api.EcowittAPI("a", "b", httpx.MockTransport(handler)) + + async def later(): + await real_sleep(0.01) # once the first request has had its 503 + return await api.realtime("x", "outdoor", urgent=True) + devices, _ = await asyncio.gather(api.devices(), later()) + await api.close() + assert len(devices) == 1 and order == ["list", "real_time", "list"] + + +async def test_during_an_outage_the_archive_asks_once_a_range_and_stops_after_three(tmp_path, monkeypatch): + """A timeout says nothing about a group: no range is asked again group by group (8 requests where there was one), and three + failures in a row end the run; the rest wait for the next one.""" + import httpx + from lib.ecowitt import Archive, archive as archive_mod + monkeypatch.setattr(archive_mod, "PACE_SECONDS", 0) + monkeypatch.setattr(ecowitt_api, "MIN_GAP_SECONDS", 0) + real, asked = ecowitt_transport()[1], [] + + def handler(request): + if request.url.path.endswith("/history"): + asked.append(request.url.params["call_back"]) + raise httpx.ConnectTimeout("timed out") + return real(request) + eco = Ecowitt(config(tmp_path), transport=httpx.MockTransport(handler)) + await eco.start() + fetched, failed = await Archive(eco).run_once() + await eco.close() + assert (fetched, failed) == (0, 3) and len(asked) == 3 and all("," in groups for groups in asked) + + +def test_an_old_range_fetched_does_not_push_out_the_fresh_copy_of_the_last_hours(): + import time as _time + from lib.ecowitt.store import HotStore + hot, now = HotStore(), int(_time.time()) + hot.put("M", "5min", "outdoor", now - 3 * 3600, now, {"temperature": "recent"}) + hot.put("M", "5min", "outdoor", now - 90 * 86400, now - 89 * 86400, {"temperature": "old"}) # the archive, going back + assert hot.get("M", "5min", "outdoor", now - 3600, now) == {"temperature": "recent"} + hot.put("M", "5min", "outdoor", now - 3600, now + 1, {"temperature": "newer"}) # a refresh replaces it + assert hot.get("M", "5min", "outdoor", now - 1800, now) == {"temperature": "newer"} + + # ---------- wind direction is circular ---------- def test_direction_summary_crosses_north_without_a_wide_swing(): from lib.ecowitt.direction import point, summarise, vector_mean @@ -649,7 +776,7 @@ async def wind_chart(days, **extra): def test_the_rolling_mean_smooths_a_staircase_without_moving_timestamps_or_crossing_gaps(): - from lib.timeutil import rolling_mean + from lib.lines import rolling_mean assert rolling_mean({0: 5.0, 300: 5.0, 600: 5.0}, 450) == {0: 5.0, 300: 5.0, 600: 5.0} assert rolling_mean({0: 1.0, 300: float('nan'), 600: 3.0}, 450) == {0: 1.0, 600: 3.0} # a bad reading is left out, it does not poison the rest stairs = {i * 300: 10 + 0.1 * (i // 3) + (0.1 if i % 2 else 0.0) for i in range(30)} # 0.1-degree steps with jitter @@ -686,12 +813,12 @@ async def chart(): async def test_a_long_period_is_read_from_the_cache_at_30_minutes_when_it_is_held_and_fits_the_row_budget(tmp_path, archived_cache, monkeypatch): from lib.ecowitt import query as history seen = [] - real = history.HistoryQuery._chunks + real = history.HistoryQuery._read async def spy(self, cycle, t, until): seen.append(cycle) await real(self, cycle, t, until) - monkeypatch.setattr(history.HistoryQuery, "_chunks", spy) + monkeypatch.setattr(history.HistoryQuery, "_read", spy) eco, fake = await archived_station(tmp_path, archived_cache) today = datetime.now(eco.tz).date() args = {"groups": "outdoor", "start_date": f"{today - timedelta(days=200)} 00:00:00", "end_date": f"{today - timedelta(days=1)} 00:00:00"} @@ -857,6 +984,17 @@ def test_rain_that_comes_with_a_pressure_fall_is_found_and_rain_that_does_not_is assert out["correlation_change_vs_rain_next_3h"] > fell_correlation and analyse({}, {}, 1.0, TZ) == {} +def test_rain_and_time_shares_are_of_the_same_slots(): + from lib.analysis.pairs import analyse + from tests.fakes import TZ + base = 1_790_000_000 // 1800 * 1800 + rain = {base + k * 1800: 1.0 for k in range(40)} # rain in every slot, half of them with no reading 3 h before + driver = {t: 0.0 for t in list(rain)[:6]} | {t: 5.0 for t in list(rain)[20:]} + table = analyse(driver, rain, 1.0, TZ)["by_change_before"] + assert sum(int(g["share_of_rain"].rstrip("%")) for g in table.values()) == 100 + assert all(g["rain_vs_fair_share"] == "1.0x" for g in table.values() if g["slots"]) # it rained in every slot: fair shares + + async def test_weather_link_reads_the_cache_only_and_charts_rain_under_the_reading(tmp_path, archived_cache): from lib.charts import render eco, fake = await archived_station(tmp_path, archived_cache) @@ -870,9 +1008,34 @@ async def test_weather_link_reads_the_cache_only_and_charts_rain_under_the_readi assert len(specs) == 1 and specs[0].title == "Pressure, Rain" and specs[0].panels[0].bars.width == 6 * 3600 assert render(specs[0], eco.tz)[:4] == b"\x89PNG" assert "error" in json.loads(await eco.tools[3].handler({"driver": "nonsense"})) + wind = Turn() # shaded as plot_chart shades it: up to the gusts (48 km/h at most) + await eco.tools[3].handler({"driver": "wind", "start_date": f"{today - timedelta(days=20)}", "end_date": f"{today - timedelta(days=2)}"}, wind) + line = wind.charts[0].panels[0].lines[0] + assert line.records["high"][1] == max(line.high) == 48.0 await eco.close() +def test_a_reading_is_shaded_alike_on_every_chart_ecowitts_own_range_else_the_5_minute_readings(tmp_path): + """reading_slots, which plot_chart and weather_link both draw from: Ecowitt's own 30-minute lows and highs where it gives them, + else the range of the cached 5-minute readings in each slot; wind's band reaches its gusts.""" + from lib.ecowitt.link import reading_slots + from lib.ecowitt.store import HistoryCache + cache = HistoryCache(tmp_path / "c.sqlite", {"u": "C"}) + t = int(datetime.now().timestamp() - 10 * 86400) // 1800 * 1800 + + def put(cycle, group, **fields): + data = {group: {f: {"unit": "", "list": {str(ts): str(v) for ts, v in points.items()}} for f, points in fields.items()}} + cache.store("M", cycle, [group], data, t, t + 1799) + put("30min", "pressure", relative={t: 1015.0}) + put("5min", "pressure", relative={t + 300 * i: 1014.0 + i for i in range(6)}) + assert reading_slots(cache, "M", "pressure", t, t + 1799) == ({t: 1015.0}, {t: 1014.0}, {t: 1019.0}) + put("30min", "outdoor", humidity={t: 60.0}, humidity_low={t: 52.0}, humidity_high={t: 71.0}) + put("5min", "outdoor", humidity={t + 300 * i: 40.0 + i for i in range(6)}) + assert reading_slots(cache, "M", "humidity", t, t + 1799) == ({t: 60.0}, {t: 52.0}, {t: 71.0}) + put("30min", "wind", wind_speed={t: 8.0}, wind_speed_low={t: 2.0}, wind_speed_high={t: 12.0}, wind_gust={t: 20.0}, wind_gust_high={t: 31.0}) + assert reading_slots(cache, "M", "wind", t, t + 1799) == ({t: 8.0}, {t: 2.0}, {t: 31.0}) + + def test_the_pair_chart_band_is_ecowitts_own_lows_and_highs_where_the_cache_has_them(): from lib.ecowitt.link import driver_series from tests.fakes import TZ @@ -900,7 +1063,7 @@ def test_a_days_mean_counts_a_stretch_held_at_5_minutes_no_more_than_the_same_st def test_the_bucket_widens_as_the_period_grows_and_never_goes_below_the_readings(): - from lib.timeutil import bucket_width + from lib.lines import bucket_width day = 86400 assert [bucket_width(d * day, 300) for d in (1, 1.7, 2)] == [300, 300, 1800] # 5-minute readings, until they exceed the budget assert [bucket_width(d * day, 1800) for d in (2, 10, 11, 20, 21, 41, 42, 83, 84, 1460)] == [ @@ -909,18 +1072,18 @@ def test_the_bucket_widens_as_the_period_grows_and_never_goes_below_the_readings def test_bucket_summary_is_the_mean_and_true_extremes_per_epoch_aligned_bucket(): - from lib.timeutil import bucket_summary, bucketed + from lib.lines import bucket_summary, bucketed from tests.fakes import TZ t0 = int(datetime(2026, 5, 1, tzinfo=TZ).timestamp()) // 7200 * 7200 points = [(t0 + i * 1800, float(i), i - 0.5, i + 0.5, False) for i in range(8)] # two 2-hour buckets of four 30-minute readings out = bucket_summary(points, 7200) assert out == {t0: (1.5, -0.5, 3.5), t0 + 7200: (5.5, 3.5, 7.5)} - xs, mean, low, high = bucketed(points, TZ, 7200) + xs, mean, low, high = bucketed(points, 7200) assert xs == [t0 + 3600, t0 + 10800] and mean == [1.5, 5.5] and low == [-0.5, 3.5] and high == [3.5, 7.5] def test_daily_summary_is_one_mean_and_true_extremes_per_local_day(): - from lib.timeutil import daily_summary + from lib.lines import daily_summary from tests.fakes import TZ t0 = int(datetime(2026, 5, 1, tzinfo=TZ).timestamp()) day1 = [(t0 + i * 1800, 10.0, 8.0, 12.0, False) for i in range(48)] @@ -946,13 +1109,18 @@ def test_the_pair_chart_renders_for_a_day_a_month_and_a_year(): assert spec.panels[0].bars.width in (3600, 6 * 3600, 86400) and render(spec, TZ)[:4] == b"\x89PNG" -async def test_weather_link_says_which_series_is_missing(tmp_path): +@pytest.mark.parametrize("driver, series, falling", [(None, "pressure (pressure.relative)", "down 1 hPa"), + ("humidity", "humidity (outdoor.humidity)", "down 5 %"), + ("wind", "wind (wind.wind_speed)", "down 5 km/h")]) +async def test_weather_link_says_which_series_is_missing(tmp_path, driver, series, falling): transport, fake = ecowitt_transport() eco = Ecowitt(config(tmp_path), transport=transport) # nothing cached at all await eco.start() today = datetime.now(eco.tz).date() - out = json.loads(await eco.tools[3].handler({"start_date": str(today - timedelta(days=9)), "end_date": str(today - timedelta(days=2))})) - assert "pressure (pressure.relative) or rainfall (rainfall.daily)" in out["note"] and "chart" not in out + out = json.loads(await eco.tools[3].handler({"start_date": str(today - timedelta(days=9)), "end_date": str(today - timedelta(days=2)), + **({"driver": driver} if driver else {})})) + assert f"{series} or rainfall (rainfall.daily)" in out["note"] and "chart" not in out + assert f"falling = {falling} or more" in out["how_to_read"] # the change that counts, in the reading's unit await eco.close() @@ -1123,16 +1291,17 @@ async def test_thirty_minute_readings_without_a_range_get_one_from_the_cached_fi cache.close() -async def test_a_chart_field_may_be_a_readings_name_or_its_field(tmp_path): +async def test_a_chart_field_may_be_a_readings_name_or_its_field_however_it_is_written(tmp_path): transport, _ = ecowitt_transport() eco = Ecowitt(config(tmp_path), transport=transport) await eco.start() today = datetime.now(eco.tz).date() - for said in ("pressure", "relative", "dew_point"): - turn = Turn() - await eco.tools[1].handler({"groups": "outdoor,pressure", "chart": True, "chart_field": said, + for said, title in (("pressure", "Pressure"), ("relative", "Pressure"), ("dew_point", "Dew point"), ("Dew Point", "Dew point"), + ("wind gust", "Wind")): + turn = Turn() # the reading's group is added for it + await eco.tools[1].handler({"groups": "outdoor", "chart": True, "chart_field": said, "start_date": f"{today - timedelta(days=9)} 00:00:00", "end_date": f"{today - timedelta(days=1)} 23:59:59"}, turn) - assert turn.charts and turn.charts[0].title == ("Dew point" if said == "dew_point" else "Pressure"), said + assert turn.charts and turn.charts[0].title == title, said await eco.close() @@ -1282,7 +1451,7 @@ async def test_rain_older_than_the_30_minute_readings_is_drawn_from_the_daily_re q.f, q.tz, q.turn = f, TZ, Turn() q.start, q.end = datetime.combine(days[0], datetime.min.time()), datetime.combine(days[-1], datetime.max.time()) q.span = q.end - q.start - q.rain_from_daily = False + q.rain_from_daily, q.store = False, {} daily = await q._rain_bars() assert q.rain_from_daily and len(daily.x) >= 59 and round(sum(daily.y), 1) == 120.0 and daily.per == "day" q.turn = Turn(text="rain by month for 2 months") @@ -1328,7 +1497,7 @@ async def test_a_single_reading_chart_is_labelled_with_the_reading_not_its_group eco, _ = await archived_station(tmp_path, archived_cache) async def lines(text): - read = intent.read(text, now_local(eco.tz), True, True) + read = intent.read(text, now_local(eco.tz)) turn = Turn(chart_asked=read.chart_asked, chart_field=read.chart_field, chart_fields=read.chart_fields, text=text) await eco.tools[1].handler(read.fast[1], turn) (panel,) = turn.charts[0].panels @@ -1338,3 +1507,65 @@ async def lines(text): assert sorted(await lines("Temperature chart 1d")) == ["Indoor", "Outdoor"] # these two tell the lines apart, so they keep their names assert await lines("Wind chart 1d") == ["Wind"] await eco.close() + + +async def test_the_newest_rain_is_in_the_bars_though_the_cache_has_not_got_it_yet(tmp_path): + """The cache holds 30-minute readings only up to the settle horizon (now less 2.5 hours); the newer ones are the query's own + (from memory or Ecowitt) and their rain must be in the bars and the total, not left out.""" + from lib.ecowitt.fetch import Fetcher + from lib.ecowitt.query import HistoryQuery + from lib.ecowitt.store import HistoryCache + from tests.fakes import TZ + cache = HistoryCache(tmp_path / "c.sqlite", {}) + day = datetime.now(TZ).date() - timedelta(days=3) + t0 = int(datetime.combine(day, time()).replace(tzinfo=TZ).timestamp()) + cache.store("M", "30min", ["rainfall"], {"rainfall": {"daily": {"unit": "mm", "list": {str(t0 + i * 1800): "0.0" for i in range(4)}}}}, + t0, t0 + 4 * 1800 - 1) # midnight to 2am, dry + f = Fetcher(None, cache, None, "M", [], TZ) + q = HistoryQuery.__new__(HistoryQuery) + q.f, q.tz, q.turn, q.rain_from_daily = f, TZ, Turn(), False + q.start, q.end = datetime.combine(day, time()), datetime.combine(day, time(3)) + q.span = q.end - q.start + q.store = {"rainfall.daily": {"unit": "mm", "pts": {t0 + 7200 + i * 300: {"cycle": "5min", "value": (0.5 * i, f"{0.5 * i}")} + for i in range(11)}}} # 2am to 2:50am, 5 mm + bars = await q._rain_bars() + assert round(sum(bars.y), 1) == 5.0 and bars.y[2] == 5.0 # all of it in the 2am bar + + +async def test_wind_direction_over_a_past_week_keeps_its_last_day(tmp_path, archived_cache): + """A week 200 days back is cached at 30 minutes only, and Ecowitt keeps 5-minute readings for 90 days: the week's last day comes + from the 30-minute readings like the others, not asked for at 5 minutes (as today's is) and dropped.""" + import shutil + import httpx + from tests.fakes import TZ + + class Retaining(FakeEcowitt): # as Ecowitt: no 5-minute readings older than 90 days + def _history(self, p): + start = datetime.strptime(p["start_date"], "%Y-%m-%d %H:%M:%S").replace(tzinfo=TZ) + return {} if p["cycle_type"] == "5min" and (datetime.now(TZ) - start).days > 89 else FakeEcowitt._history(p) + cfg = config(tmp_path) + shutil.copy(archived_cache, cfg.cache_path) + eco = Ecowitt(cfg, transport=httpx.MockTransport(Retaining())) + await eco.start() + last = datetime.now(eco.tz).date() - timedelta(days=200) + out = json.loads(await eco.tools[1].handler({"groups": "wind", "start_date": f"{last - timedelta(days=6)} 00:00:00", + "end_date": f"{last} 23:59:59"})) + await eco.close() + assert len(out["series"]["wind.wind_direction"]["daily"]) == 7 + + +async def test_the_wind_rose_s_failed_requests_are_reported_with_the_rest(tmp_path, archived_cache, monkeypatch): + from types import SimpleNamespace + from lib.ecowitt import query + eco, _ = await archived_station(tmp_path, archived_cache) + counted = query.HistoryQuery._summarise_direction + + async def with_a_failure(self): + await counted(self) + self.wind = SimpleNamespace(errors=["5min 16 Oct 2026 - 16 Oct 2026: HTTP 502"], seconds=0.0) + monkeypatch.setattr(query.HistoryQuery, "_summarise_direction", with_a_failure) + today = datetime.now(eco.tz).date() + out = json.loads(await eco.tools[1].handler({"groups": "wind", "start_date": f"{today - timedelta(days=3)} 00:00:00", + "end_date": f"{today - timedelta(days=2)} 23:59:59"})) + await eco.close() + assert "HTTP 502" in " ".join(out.get("missing", [])) and "warning" in out diff --git a/tests/test_forecast.py b/tests/test_forecast.py index e0f9da8..1871e35 100644 --- a/tests/test_forecast.py +++ b/tests/test_forecast.py @@ -115,9 +115,12 @@ async def test_the_forecast_is_fetched_only_in_the_day_and_not_again_while_young await forecast.close() -async def test_the_station_location_is_used_when_none_is_configured_and_without_one_the_source_does_not_start(tmp_path): +async def test_the_station_location_is_used_when_none_is_configured_and_without_one_the_source_does_not_start(tmp_path, caplog): cfg = config(tmp_path, forecast=True) assert Forecast(cfg, (-37.8, 144.9)).location == (-37.8, 144.9) + half = config(tmp_path, forecast=True, forecast_lat=-33.9) # one half alone is not a place: not paired with the station's + assert Forecast(half, (-37.8, 144.9)).location == (-37.8, 144.9) and "FORECAST_LAT and FORECAST_LON go together" in caplog.text + assert Forecast(config(tmp_path, forecast=True, forecast_lat=-33.9, forecast_lon=151.2), (-37.8, 144.9)).location == (-33.9, 151.2) with pytest.raises(RuntimeError, match="FORECAST_LAT"): await Forecast(cfg).start() diff --git a/tests/test_forecast_alert.py b/tests/test_forecast_alert.py index f768fd4..709435b 100644 --- a/tests/test_forecast_alert.py +++ b/tests/test_forecast_alert.py @@ -52,7 +52,7 @@ def test_substantial_means_rain_flips_or_a_max_temperature_over_2_degrees_differ async def test_a_day_sent_as_rainy_that_turns_dry_is_told_once_to_that_chat(tmp_path): mon, forecast, state, bot = setup(tmp_path, [day(0, pct=95, summary="Thunderstorm."), day(1)]) - state.record_forecast(1, [day(0, pct=95, summary="Thunderstorm."), day(1)]) + state.record_forecast(1, [day(0, pct=95, summary="Thunderstorm."), day(1)], TODAY) await mon.check() assert bot.sent == [] # unchanged forecast.days = [day(0, pct=10, summary="Sunny."), day(1)] @@ -66,7 +66,7 @@ async def test_a_day_sent_as_rainy_that_turns_dry_is_told_once_to_that_chat(tmp_ async def test_dry_to_rain_and_max_temperature_changes_alert_and_a_further_change_alerts_again(tmp_path): mon, forecast, state, bot = setup(tmp_path, [day(0), day(1, max_c=20.0)]) - state.record_forecast(1, [day(0), day(1, max_c=20.0)]) + state.record_forecast(1, [day(0), day(1, max_c=20.0)], TODAY) forecast.days = [day(0, pct=80, summary="Showers."), day(1, max_c=22.1)] await mon.check() assert len(bot.sent) == 1 # two changed days: one message @@ -78,24 +78,45 @@ async def test_dry_to_rain_and_max_temperature_changes_alert_and_a_further_chang async def test_small_changes_past_days_muted_chats_and_chats_never_sent_a_forecast_are_left_alone(tmp_path): mon, forecast, state, bot = setup(tmp_path, [day(0), day(1)]) - state.record_forecast(1, [day(0), day(1)]) + state.record_forecast(1, [day(0), day(1)], TODAY) forecast.days = [day(0, max_c=21.9, pct=40), day(1, max_c=18.5)] await mon.check() assert bot.sent == [] # inside the thresholds - state.record_forecast(1, [day(-1, pct=90), day(0), day(1)]) + state.record_forecast(1, [day(-1, pct=90), day(0), day(1)], TODAY) forecast.days = [day(0, pct=70), day(1)] # yesterday is not in the forecast any more; today flips await mon.check() assert len(bot.sent) == 1 and "Yesterday" not in bot.sent[0][1] and bot.sent[0][1].count("• ") == 1 - state.record_forecast(2, [day(0, pct=90)]) - state.set_alerts(2, "chat 2", False) + state.record_forecast(2, [day(0, pct=90)], TODAY) + state.set_alerts(2, False) forecast.days = [day(0, pct=5), day(1)] await mon.check() assert [chat for chat, _ in bot.sent] == [1, 1] and state.forecast_sent(2)[TODAY]["rain_chance_pct"] == 90 # muted: not told, not reset +async def test_a_revised_day_leaves_the_other_days_sent_with_it_to_be_told(tmp_path): + week = [day(i) for i in range(7)] + mon, forecast, state, bot = setup(tmp_path, week) + state.record_forecast(1, week, TODAY) + forecast.days = [*week[:2], day(2, pct=80), *week[3:]] # Monday turns rainy + await mon.check() + forecast.days = [week[0], day(1, pct=80), day(2, pct=80), *week[3:]] # then Sunday does + await mon.check() + assert len(bot.sent) == 2 and "• Monday: was" in bot.sent[0][1] and "• Sunday: was" in bot.sent[1][1] + assert sorted(state.forecast_sent(1)) == [d["date"] for d in week] + + +async def test_an_answer_about_later_days_leaves_today_to_be_told(tmp_path): + mon, forecast, state, bot = setup(tmp_path, [day(0, pct=90), day(1)]) + state.record_forecast(1, [day(0, pct=90), day(1)], TODAY) # the morning report + state.record_forecast(1, [day(1)], TODAY) # "what about tomorrow?" + forecast.days = [day(0, pct=5), day(1)] + await mon.check() + assert len(bot.sent) == 1 and "• Today: was" in bot.sent[0][1] + + async def test_what_was_sent_survives_a_restart_and_is_not_repeated(tmp_path): mon, forecast, state, bot = setup(tmp_path, [day(0, pct=90)]) - state.record_forecast(1, [day(0, pct=90)]) + state.record_forecast(1, [day(0, pct=90)], TODAY) again = AlertState(tmp_path / "s.json") assert again.forecast_sent(1)[TODAY]["rain_chance_pct"] == 90 forecast.days = [day(0, pct=5)] @@ -155,7 +176,7 @@ async def send_chat_action(*a, **k): await forecast.close() -async def test_to_chat_goes_to_one_chat_and_forgets_a_blocked_one(tmp_path): +async def test_an_alert_for_one_chat_goes_to_it_alone_and_a_blocked_one_is_marked_gone(tmp_path): from telegram.error import Forbidden state = AlertState(tmp_path / "s.json") state.add_chat(1, "a") @@ -168,8 +189,8 @@ async def send_message(self, chat_id, text, **kw): raise Forbidden("bot was blocked by the user") sent.append((chat_id, text)) notifier = Notifier(Bot(), state) - assert await notifier.to_chat(1, "hello") is True and [c for c, _ in sent] == [1] and sent[0][1] == "hello" - assert await notifier.to_chat(2, "hello") is False and 2 not in state.chats + assert await notifier("hello", chats=[1]) == 1 and [c for c, _ in sent] == [1] and sent[0][1] == "hello" + assert await notifier("hello", chats=[2]) == 0 and state.chats[2]["gone"] def test_a_revision_compares_the_one_millimetre_chance_when_both_days_have_it_and_the_older_figure_otherwise(): diff --git a/tests/test_intent.py b/tests/test_intent.py index 884f992..7a79503 100644 --- a/tests/test_intent.py +++ b/tests/test_intent.py @@ -1,12 +1,13 @@ from datetime import datetime, timedelta from lib import intent +from tests.fakes import EVERY_SOURCE NOW = datetime(2026, 9, 29, 14, 5) # a Tuesday -def call(text, ecowitt=True, air=True): - return intent.fast_call(text, NOW, ecowitt, air) +def call(text, sources=("Ecowitt", "AirGradient")): + return intent.fast_call(text, NOW, sources) def test_only_looking_ahead_and_descriptions_reason_a_little(): @@ -31,6 +32,13 @@ def test_chat_needs_no_data(): assert intent.needs_data("how hot is it") +def test_every_reading_named_needs_its_data_but_chat_words_like_cool_or_feeling_do_not(): + for asked in ("vpd now", "what's the uvi", "dewpoint 90d", "solar radiation today", "ultraviolet", "precipitation this week"): + assert intent.needs_data(asked), asked + for chat in ("cool, thanks", "feeling better now", "that's apparent"): + assert not intent.needs_data(chat), chat + + def test_highs_and_lows_go_fast(): name, args, _ = call("what was this week's high and low?") assert name == "weather_history" @@ -66,8 +74,8 @@ def test_air_now_and_air_chart(): def test_sources_are_optional(): - assert call("how's the air?", air=False) is None - assert call("this week's highs", ecowitt=False) is None + assert call("how's the air?", ["Ecowitt"]) is None + assert call("this week's highs", ["AirGradient"]) is None def test_weather_plus_a_period_is_a_summary_request(): @@ -89,7 +97,7 @@ def test_weather_now_fetches_every_reading_the_station_has_and_asks_for_the_repo assert (name, label) == ("weather_now", "weather now") and args == {"groups": intent.NOW_GROUPS}, text assert intent.read(text, NOW).weather_now assert not intent.read("weather last 7 days", NOW).weather_now and not intent.read("report", NOW).weather_now - assert not intent.read("weather now", NOW, ecowitt=False).weather_now + assert not intent.read("weather now", NOW, ["AirGradient"]).weather_now def test_24_hour_weather_charts_over_a_rolling_window(): @@ -345,14 +353,14 @@ def test_ecowitt_means_the_weather_and_ag_or_airgradient_means_the_air_quality() def test_the_report_fetches_everything_in_code_before_the_model_sees_it(): - r = intent.read("report", NOW, True, True, True, True) + r = intent.read("report", NOW, EVERY_SOURCE) assert r.report and r.fast == ("weather_now", {"groups": intent.NOW_GROUPS}, "report") assert r.more == [("air_quality", {}), ("pollen_asthma", {"cached": True}), ("weather_forecast", {"days": 3, "cached": True})] plain = intent.read("sitrep", NOW) assert [plain.fast[0], *(t for t, _ in plain.more)] == ["weather_now", "air_quality"] - only_air = intent.read("status", NOW, ecowitt=False) + only_air = intent.read("status", NOW, ["AirGradient"]) assert only_air.fast[0] == "air_quality" and only_air.more == [] - assert intent.read("report", NOW, ecowitt=False, air=False).fast is None + assert intent.read("report", NOW, []).fast is None assert not intent.read("report for last week", NOW).report @@ -398,7 +406,7 @@ def test_now_alone_is_not_a_weather_question_but_naming_a_reading_with_it_is(): ("temperature today", "extremes", ["temperature"]), ("is it hotter than yesterday", "", []), ("why is it so humid", "", []), ]) def test_plain_lookups_are_answered_in_code_and_everything_else_is_left_to_the_model(text, kind, named): - read = intent.read(text, datetime(2026, 10, 2, 12), True, True, True, True) + read = intent.read(text, datetime(2026, 10, 2, 12), EVERY_SOURCE) assert (read.lookup, read.lookup_arg) == (kind, named) @@ -406,9 +414,9 @@ def test_plain_lookups_are_answered_in_code_and_everything_else_is_left_to_the_m ("hottest day that also rained", [], {"days"}, {"air", "link"}), ("did it rain on Sat 5 Sep", [], {"days"}, {"air"}), ("is there a correlation between pressure and rainfall", [], {"link"}, {"air"}), - ("is the air ok for a run", [], {"air"}, {"days", "link", "compose"}), - ("plot air quality against rainfall", [], {"air", "compose"}, set()), - ("will it rain later", [], {"outlook", "forecast"}, {"air"}), + ("is the air ok for a run", [], {"air"}, {"days", "link"}), + ("plot air quality against rainfall", [], {"air"}, set()), + ("will it rain later", [], {"outlook"}, {"air"}), ("most common wind direction", [], {"wind"}, {"air"}), ("describe what it was like on Monday", [], {"describe"}, set()), ("add an alert when winds reach 100", [], {"bot"}, set()), @@ -426,38 +434,38 @@ def test_topics_pick_the_guidance_and_tools_a_question_needs(text, before, has, ("will it rain this week", 7), ("rain later?", 3)]) def test_a_question_about_rain_ahead_has_the_reading_the_last_3_hours_and_the_forecast_fetched_for_the_model(text, days): now = datetime(2026, 10, 3, 12) - read = intent.read(text, now, True, True, False, True) + read = intent.read(text, now, ["Ecowitt", "AirGradient", "Forecast"]) groups = "outdoor,pressure,rainfall,rainfall_piezo,wind" assert read.fast is None and read.extra == [ ("weather_now", {"groups": groups}), ("weather_history", {"groups": groups, "start_date": "2026-10-03 09:00:00", "end_date": "2026-10-03 12:00:00"}), ("weather_forecast", {"days": days, "cached": True})] - no_forecast = intent.read(text, now, True, True, False, False).extra + no_forecast = intent.read(text, now).extra assert [c[0] for c in no_forecast] == ["weather_now", "weather_history"] # forecast off: just the station - assert intent.read(text, now, False, True, False, True).extra == [("weather_forecast", {"days": days, "cached": True})] # no station + assert intent.read(text, now, ["AirGradient", "Forecast"]).extra == [("weather_forecast", {"days": days, "cached": True})] # no station @_pytest.mark.parametrize("text, name, day", [("what was yesterday like?", "yesterday", "2026-10-02"), ("describe today", "today", "2026-10-03")]) def test_describing_today_or_yesterday_has_that_days_readings_fetched_for_the_model(text, name, day): - read = intent.read(text, datetime(2026, 10, 3, 12), True, True, False, False) + read = intent.read(text, datetime(2026, 10, 3, 12)) (tool, args), = read.extra assert tool == "weather_history" and args["groups"] == "outdoor,indoor,rainfall,wind" and args["start_date"].startswith(day) @_pytest.mark.parametrize("text", ["what was it like on Mon 10 Aug", "what was it like at 3pm today", "how hot was yesterday", "describe last week"]) def test_a_named_date_a_moment_a_longer_period_or_a_plain_question_has_nothing_fetched_ahead(text): - assert intent.read(text, datetime(2026, 10, 3, 12), True, True, False, True).extra == [] + assert intent.read(text, datetime(2026, 10, 3, 12), ["Ecowitt", "AirGradient", "Forecast"]).extra == [] @_pytest.mark.parametrize("text", ["how much rain fell today", "how much rain this week", "did it rain yesterday", "forecast", "rain chart 7d"]) def test_questions_about_rain_so_far_do_not_fetch_the_forecast(text): - assert intent.read(text, datetime(2026, 10, 3, 12), True, True, False, True).extra == [] + assert intent.read(text, datetime(2026, 10, 3, 12), ["Ecowitt", "AirGradient", "Forecast"]).extra == [] @_pytest.mark.parametrize("text", ["how many times did it rain this year", "worst air quality day per month", "days over 35 and also rained", "days over UVI 10 on weekends"]) def test_counting_and_totalling_questions_are_never_turned_into_a_plain_chart(text): - read = intent.read(text, datetime(2026, 10, 3, 12), True, True, True, True) + read = intent.read(text, datetime(2026, 10, 3, 12), EVERY_SOURCE) assert read.fast is None and not read.chart_in_code and read.lookup == "" @@ -473,7 +481,7 @@ def test_counting_and_totalling_questions_are_never_turned_into_a_plain_chart(te ("how many days was PM2.5 over 25 in 2024", "air_days", "pm2_5_max", ">", 25, None), ]) def test_one_reading_against_one_limit_is_counted_in_code(text, tool, field, op, value, group): - read = intent.read(text, datetime(2026, 10, 3, 12), True, True, False, False) + read = intent.read(text, datetime(2026, 10, 3, 12)) name, args, _ = read.fast assert read.lookup == "days" and name == tool and args["count_only"] is True and args.get("group_by") == group assert args["where"] == [{"field": field, "op": op, "value": float(value)}] @@ -481,11 +489,11 @@ def test_one_reading_against_one_limit_is_counted_in_code(text, tool, field, op, def test_the_period_is_the_whole_record_unless_one_is_named(): now = datetime(2026, 10, 3, 12) - whole = intent.read("days over UVI 10 by year", now, True, True).fast[1] + whole = intent.read("days over UVI 10 by year", now).fast[1] assert whole["start_date"] == "2022-10-05" and whole["end_date"] == "2026-10-03" - named = intent.read("days over UVI 10 by month this year", now, True, True).fast[1] + named = intent.read("days over UVI 10 by month this year", now).fast[1] assert named["start_date"] == "2026-01-01" and named["group_by"] == "month" - year = intent.read("how many days was PM2.5 over 25 in 2024", now, True, True).fast[1] + year = intent.read("how many days was PM2.5 over 25 in 2024", now).fast[1] assert year["start_date"] == "2024-01-01" and year["end_date"] == "2024-12-31" @@ -501,7 +509,7 @@ def test_the_period_is_the_whole_record_unless_one_is_named(): ("average co2 per year", "air_days", "avg", "co2_avg", "year"), ]) def test_one_reading_by_month_or_year_is_a_figure_per_period_worked_out_in_code(text, tool, stat, of, group): - read = intent.read(text, datetime(2026, 10, 3, 12), True, True, False, False) + read = intent.read(text, datetime(2026, 10, 3, 12)) name, args, _ = read.fast assert read.lookup == "days" and (name, args["stat"], args["of"], args["group_by"]) == (tool, stat, of, group) and args["count_only"] is True @@ -509,23 +517,71 @@ def test_one_reading_by_month_or_year_is_a_figure_per_period_worked_out_in_code( @_pytest.mark.parametrize("text", ["pm2.5 and co2 by month", "rain by month on weekends", "air quality by month", "will it rain by month", "days over 30 degrees and rain per month", "rain days per month"]) def test_anything_beyond_one_reading_by_month_or_year_is_left_to_the_model(text): - read = intent.read(text, datetime(2026, 10, 3, 12), True, True, False, False) + read = intent.read(text, datetime(2026, 10, 3, 12)) assert read.lookup == "" and (read.fast is None or read.fast[0] not in ("weather_days", "air_days")) -def test_weather_all_week_is_flagged_as_a_chart_with_no_caption_and_a_named_chart_is_not(): - now = datetime(2026, 10, 3, 12) - assert intent.read("weather all week", now, True, True).chart_all and intent.read("ecowitt all week", now, True, True).chart_all - assert not intent.read("plot temperature and rain", now, True, True).chart_all and not intent.read("temperature chart 7d", now, True, True).chart_all +def test_weather_all_week_is_every_reading_and_a_named_chart_is_not(): + assert intent.wants_chart_all("weather all week") and intent.wants_chart_all("ecowitt all week") + assert not intent.wants_chart_all("plot temperature and rain") and not intent.wants_chart_all("temperature chart 7d") @_pytest.mark.parametrize("ask", ["weather chart 90d", "chart all 90d", "chart 90d", "chart"]) def test_a_chart_that_names_no_reading_is_every_reading_not_temperature(ask): read = intent.read(ask, NOW) - assert read.chart_all and read.chart_fields[:3] == ["temperature", "humidity", "solar"] and read.chart_fields[-1] == "wind" + assert intent.wants_chart_all(ask) and read.chart_fields[:3] == ["temperature", "humidity", "solar"] and read.chart_fields[-1] == "wind" def test_a_named_reading_or_an_air_chart_is_not_every_reading_and_indoors_has_only_what_the_indoor_sensor_measures(): for ask in ("temperature chart 90d", "rain chart 7d", "chart air 90d", "plot pm2.5 90d"): assert not intent.wants_chart_all(ask) and intent.chart_fields(ask) == [] assert intent.chart_fields("chart all 90d indoors") == ["temperature", "humidity", "dew_point"] + + +# ---------- what a phone types, and words that only look like other things ---------- +def test_a_phone_s_curly_apostrophe_reads_as_the_plain_one(): + for plain in ("what's the weather", "what's the temperature", "show what you've got", "what's the forecast"): + curly = plain.replace("'", "’") + assert intent.read(curly, NOW, EVERY_SOURCE).fast == intent.read(plain, NOW, EVERY_SOURCE).fast, curly + assert intent.read(curly, NOW, EVERY_SOURCE).fast is not None, curly + + +def test_all_time_and_every_day_are_periods_not_every_air_reading(): + for text in ("pm2.5 chart all time", "pm2.5 every day this week", "highest pm2.5 of all-time"): + assert intent.air_named(text) == ["pm2_5"], text + for text in ("aq all week", "air quality all metrics 1d", "chart everything in the air 7d"): # every reading still means every one + assert intent.air_named(text) == ["pm2_5", "pm10", "pm1", "co2", "voc_index", "nox_index"], text + assert call("count days over PM2.5 of 90 per year all time")[0] == "air_days" + + +def test_co2_with_a_subscript_two_is_the_air_reading(): + assert call("co₂ chart 7d")[1]["metrics"] == ["co2"] and call("chart CO₂ 1d")[0] == "air_quality" + assert intent.needs_data("co₂") and intent.needs_data("pm1") and intent.mentions_air("co₂") + + +def test_indoor_dew_point_and_feels_like_come_from_the_indoor_sensor(): + for text in ("indoor dew point chart", "indoor feels like chart 7d", "indoor dew point 90d"): + assert call(text)[1]["groups"] == "indoor", text + assert call("dew point chart")[1]["groups"] == "outdoor" # unasked, they stay the outdoor sensor's + assert call("indoor and outdoor dew point chart 7d")[1]["groups"] == "outdoor,indoor" + + +def test_the_lowest_uv_per_month_is_the_lowest_daily_high_not_the_highest(): + tool, args, _ = intent.period_figures_call("lowest uv by month", NOW, ["Ecowitt"]) + assert (tool, args["stat"], args["of"], args["group_by"]) == ("weather_days", "min", "uv_max", "month") + assert intent.period_figures_call("uv by month", NOW, ["Ecowitt"])[1]["stat"] == "max" # a UV index is its day's high + assert intent.period_figures_call("lowest wind by month", NOW, ["Ecowitt"]) is None # wind has no lowest to ask for + + +def test_the_air_table_is_the_one_in_series_and_panels_imports_on_its_own(): + import subprocess + import sys + from lib.airgradient.metrics import LABELS, MARK_LOW, RATINGS + from lib.series import AIR + assert list(LABELS) == list(AIR) == [m for _, m in intent.AIR_METRICS] and MARK_LOW == ("co2",) and RATINGS["pm2_5"] == (9.0, 55.4) + run = subprocess.run([sys.executable, "-c", "import lib.panels"], capture_output=True, text=True) + assert run.returncode == 0, run.stderr # no circular import through airgradient + + +def test_the_models_first_step_must_fetch_for_data_may_just_reply_to_chat_and_gets_no_tools_about_the_bot(): + assert [intent.read(t, NOW).tool_choice for t in ("will it rain tomorrow?", "thanks!", "what can you do")] == ["required", "auto", "none"] diff --git a/tests/test_irrigation.py b/tests/test_irrigation.py index fed4ba2..bf52029 100644 --- a/tests/test_irrigation.py +++ b/tests/test_irrigation.py @@ -4,7 +4,7 @@ import hashlib import hmac import json -from datetime import datetime +from datetime import datetime, timedelta from types import SimpleNamespace as NS import httpx @@ -124,6 +124,29 @@ def down(request): assert SECRET not in str(e.value) and "ConnectError" in str(e.value) +TOKEN = {"success": True, "result": {"access_token": "tok", "expire_time": 7200}} + + +@pytest.mark.parametrize("token, status", [({"success": True}, None), # a token reply with no token + ({"success": True, "result": ["tok"]}, None), + ([TOKEN], None), # a reply that isn't an object + (TOKEN, {"success": True, "result": {"switch": True}}), # a status that isn't a list + (TOKEN, "OK")]) +async def test_a_reply_of_an_unexpected_shape_is_an_irrigation_error(tmp_path, token, status): + def serve(request): + return httpx.Response(200, json=token if request.url.path.startswith("/v1.0/token") else status) + device = Tuya(cfg(tmp_path), transport=httpx.MockTransport(serve)) + with pytest.raises(IrrigationError): + await device.status() + await device.close() + + +async def test_status_points_with_no_value_or_no_code_are_read_safely(tmp_path): + device, fake = tuya(tmp_path, status=[{"code": "switch"}, "battery", {"value": 3}, {"code": "countdown", "value": 60}]) + assert await device.status() == {"switch": None, "countdown": 60} + await device.close() + + def test_the_controller_is_on_only_with_all_three_settings_and_the_hour_is_read_safely(monkeypatch, tmp_path): assert cfg(tmp_path).irrigation and not config(tmp_path).irrigation and not config(tmp_path, tuya_device_id="d", tuya_client_id="c").irrigation env = {"TELEGRAM_BOT_TOKEN": "t", "XAI_API_KEY": "k", "ECOWITT_API_KEY": "a", "ECOWITT_APP_KEY": "b"} @@ -171,6 +194,8 @@ def test_the_buttons_open_one_section_at_a_time_and_every_callback_is_short(): assert rows(menu.keyboard(True, True, running=True)) == collapsed + [[("Water off ❌", "ir:sw:off")]] # only while the valve is on assert rows(menu.keyboard(True, True, paused=True, running=True))[-1] == [("Water off ❌", "ir:sw:off")] assert "ir:sw:on" not in [data for row in water for _, data in row] + assert [len(r) for r in water] == [1, 4] and rows(menu.keyboard(True, True, "delay"))[0] == [("⏸ Pause ▾", "ir:close")] # an open menu hides the other buttons + assert rows(menu.keyboard(True, True, "water", running=True))[1] == [("Water off ❌", "ir:sw:off")] # but Off stays assert rows(menu.keyboard(True, True, "delay"))[-1] == [("24h", "ir:delay:24h"), ("48h", "ir:delay:48h"), ("72h", "ir:delay:72h")] paused = rows(menu.keyboard(True, True, paused=True)) # paused: the button unpauses, in one press assert paused[0] == [("▶️ Unpause", "ir:delay:cancel")] and paused[1] == collapsed[1] # no Water button while paused @@ -183,8 +208,7 @@ def test_the_buttons_open_one_section_at_a_time_and_every_callback_is_short(): (False, True, (strike("battery alerts"), "ir:batt:on"), ("pause alerts", "ir:pause:off")), (True, True, ("battery alerts", "ir:batt:off"), ("pause alerts", "ir:pause:off"))): alerts = rows(menu.keyboard(battery, pause, "alerts")) - assert alerts[:2] == collapsed[:1] + [[("🔔 Alerts ▾", "ir:close"), ("🔄 Refresh", "ir:refresh")]] - assert alerts[2:] == [[battery_button], [pause_button]] # each names the action it takes + assert alerts == [[("🔔 Alerts ▾", "ir:close")], [battery_button], [pause_button]] # the open menu's own button is its heading; the rest are hidden # each names the action it takes for section in (None, *menu.SECTIONS): for battery in (True, False): for pause in (True, False): @@ -196,7 +220,7 @@ def test_the_irrigation_button_takes_wind_s_place_only_when_configured_and_chang assert [b.text for row in plain for b in row][-1] == "\U0001f4a8 Wind" and templates.IRRIGATION not in [b.text for row in plain for b in row] labels = [b.text for row in with_irrigation for b in row] assert labels[-1] == templates.IRRIGATION and "\U0001f4a8 Wind" not in labels and len(labels) == 9 and with_irrigation[:2] == plain[:2] - assert templates.version() == templates.VERSION != templates.version(True) + assert templates.version() != templates.version(True) assert templates.sentence(templates.IRRIGATION) is None # not a question for the model @@ -255,6 +279,7 @@ class Forecast: def __init__(self, *chances): self.days = [{"date": datetime(2026, 10, 8 + i).date(), "rain_1mm_pct": c} for i, c in enumerate(chances)] + [ {"date": datetime(2026, 10, 10).date(), "rain_1mm_pct": 90}] # the day after tomorrow never counts + self.fetched_at = datetime(2026, 10, 8, 17, 45, tzinfo=TZ).timestamp() def now(self): return datetime(2026, 10, 8, 18, 0) @@ -305,6 +330,30 @@ async def test_a_delay_already_set_is_never_shortened_or_repeated(tmp_path): assert device.commands == [] and sent == [] # nothing changed: nothing to tell +async def test_the_bots_own_pause_from_yesterday_is_started_again_and_one_someone_set_is_not(tmp_path): + device = Device() + m, state, sent = daily(tmp_path, device, Station(0, 3)) + await m.check(NOW) # paused at 6pm + await m.check(NOW + timedelta(days=1)) # 6pm the next day: still on, for seconds + assert device.commands == [("delay", "24h"), ("delay", "cancel"), ("delay", "24h")] + assert sent[1][0] == "☔ Irrigation paused for another 24 hours: 3 mm of rain in the last 24 hours." + m.station = Station(0, 0) + await m.check(NOW + timedelta(days=2)) # dry: it lapses + m.station = Station(0, 3) + await m.check(NOW + timedelta(days=4)) # a 24h someone set (the bot's is days old) + assert len(device.commands) == 3 and len(sent) == 2 + + +async def test_a_forecast_more_than_a_day_old_does_not_decide(tmp_path, caplog): + stale = Forecast(90, 90) + stale.fetched_at -= 25 * 3600 + device = Device() + m, state, sent = daily(tmp_path, device, Station(0, 0), stale) + with caplog.at_level("INFO"): + await m.check(NOW) + assert device.commands == [] and sent == [] and "over a day old: only the measured rain decides" in caplog.text + + async def test_a_battery_under_5_percent_is_told_once_a_day_to_the_irrigation_alert_type(tmp_path): device = Device(battery_percentage=4) m, state, sent = daily(tmp_path, device) @@ -338,6 +387,7 @@ def test_a_check_is_due_once_a_day_after_the_hour_and_the_next_is_the_following_ assert not m.due(datetime(2026, 10, 8, 23, 0)) and not m.due(datetime(2026, 10, 9, 3, 0)) and m.due(datetime(2026, 10, 9, 18, 0)) assert m.seconds_until_next(datetime(2026, 10, 8, 17, 0)) == 3600 and m.seconds_until_next(datetime(2026, 10, 8, 18, 0)) == 86400 assert m.seconds_until_next(datetime(2026, 10, 8, 23, 30)) == 18.5 * 3600 + assert m.seconds_until_next(datetime(2026, 10, 3, 18, 0)) == 23 * 3600 # the clocks go forward that night: 23 real hours, not 24 async def test_run_checks_when_due_retries_after_a_failure_and_then_waits_for_tomorrow(tmp_path, monkeypatch): @@ -458,6 +508,25 @@ async def reply_text(body, **kw): assert sent[1][0] == "Sorry, I couldn't reach the irrigation controller." and len(sent) == 2 +async def test_an_odd_reply_from_the_controller_is_a_toast_not_a_press_left_hanging(tmp_path): + def odd(request): + return httpx.Response(200, json=["not", "a", "reply"]) + bot, state, device = irrigation_bot(tmp_path, Tuya(cfg(tmp_path), transport=httpx.MockTransport(odd))) + q = await press(bot, "ir:refresh") + assert q.toast == "Couldn't reach the irrigation controller" and q.edits == [] + await device.close() + + +async def test_a_run_whose_reply_is_lost_is_still_watched(tmp_path): + class Lost(Remote): + async def water(self, minutes): + await super().water(minutes) # the valve opened + raise IrrigationError("Tuya request failed (ReadTimeout)") # but the reply never came + bot, state, device = irrigation_bot(tmp_path, Lost()) + q = await press(bot, "ir:water:10") + assert q.toast == "Couldn't reach the irrigation controller" and bot.watchdog.off_at is not None + + async def test_opening_and_closing_a_section_only_changes_the_buttons(tmp_path): bot, _, device = irrigation_bot(tmp_path) q = await press(bot, "ir:open:water") @@ -491,14 +560,14 @@ async def test_watering_a_delay_and_the_switch_act_on_the_controller_and_refresh async def test_the_alerts_menu_toggles_each_alert_type_for_this_chat_alone_and_stays_open(tmp_path): bot, state, _ = irrigation_bot(tmp_path) q = await press(bot, "ir:open:alerts") - assert q.edits[0][0] == "markup" and [r[0][0] for r in rows(q.edits[0][1])[2:]] == ["battery alerts", "pause alerts"] + assert q.edits[0][0] == "markup" and [r[0][0] for r in rows(q.edits[0][1])[1:]] == ["battery alerts", "pause alerts"] q = await press(bot, "ir:batt:off") assert state.muted(1) == {"irrigation_battery"} and q.toast == "Irrigation battery alerts off in this chat" - assert rows(q.edits[0][2])[2:] == [[(strike("battery alerts"), "ir:batt:on")], [("pause alerts", "ir:pause:off")]] # still open + assert rows(q.edits[0][2])[1:] == [[(strike("battery alerts"), "ir:batt:on")], [("pause alerts", "ir:pause:off")]] # still open assert q.edits[0][1].endswith("pause: not paused (alerts: on)\nbattery: 100% 🔋 (alerts: off)") # the status text follows q = await press(bot, "ir:pause:off") assert state.muted(1) == {"irrigation_battery", "irrigation_pause"} and q.toast == "Irrigation pause alerts off in this chat" - assert q.edits[0][1].endswith("pause: not paused (alerts: off)\nbattery: 100% 🔋 (alerts: off)") and rows(q.edits[0][2])[3] == [(strike("pause alerts"), "ir:pause:on")] + assert q.edits[0][1].endswith("pause: not paused (alerts: off)\nbattery: 100% 🔋 (alerts: off)") and rows(q.edits[0][2])[2] == [(strike("pause alerts"), "ir:pause:on")] q = await press(bot, "ir:batt:on") assert state.muted(1) == {"irrigation_pause"} and q.toast == "Irrigation battery alerts on in this chat" # the other type is untouched q = await press(bot, "ir:pause:on") @@ -506,7 +575,7 @@ async def test_the_alerts_menu_toggles_each_alert_type_for_this_chat_alone_and_s for kind, prefix in (("irrigation_battery", "batt"), ("irrigation_pause", "pause")): # a button shows the state, pressing it flips it for _ in range(2): alerts = rows(menu.keyboard(state.subscribed(1, "irrigation_battery"), state.subscribed(1, "irrigation_pause"), "alerts")) - text, data = next(b for row in alerts[2:] for b in row if b[1].startswith(f"ir:{prefix}:")) + text, data = next(b for row in alerts[1:] for b in row if b[1].startswith(f"ir:{prefix}:")) await press(bot, data) assert state.subscribed(1, kind) is (text != text.replace("\u0336", "")), (text, data) other = await press(bot, "ir:water:5") @@ -543,7 +612,7 @@ async def test_the_handler_is_registered_and_the_alert_list_includes_irrigation_ assert len(added) == len(added_plain) # one entry point for every button, with or without a controller assert "ir:" in bot.buttons() and "ir:" not in plain.buttons() and {"al:", "pd:"} <= set(plain.buttons()) assert "irrigation_battery" in bot._alert_kinds() and "irrigation_battery" not in plain._alert_kinds() - assert bot.keyboard_version == templates.version(True) != plain.keyboard_version == templates.VERSION + assert bot.keyboard_version == templates.version(True) != plain.keyboard_version == templates.version() @pytest.mark.parametrize("text, yes", [ @@ -810,7 +879,7 @@ def test_a_status_message_says_whether_it_is_paused(): assert menu.paused_in(menu.status_text({"weather_delay": "72h"}, True, True)) is True assert menu.paused_in(menu.status_text({"weather_delay": "cancel"}, True, True)) is False assert menu.paused_in(menu.status_text({}, True, True)) is False and menu.paused_in("") is False and menu.paused_in(None) is False - assert menu.paused_in("mode: idle\npause time: 24h (alerts: on)") is True and menu.paused_in("pause time: inactive (alerts: on)") is False # an older message + assert menu.paused_in("mode: idle\npause: 24h (alerts: on)") is True async def test_while_paused_the_button_is_unpause_and_one_press_ends_the_pause(tmp_path): diff --git a/tests/test_lines.py b/tests/test_lines.py index 99a2fd4..bf7d5df 100644 --- a/tests/test_lines.py +++ b/tests/test_lines.py @@ -32,6 +32,13 @@ def test_a_bands_range_is_kept_at_every_width(): assert hourly.width == 3600 and hourly.high is not None +def test_a_slots_range_always_holds_its_value(): + (r,) = slot_readings({T0: 5.0}, lows={T0: 6.0}, highs={T0: 4.0}) # a low above the value, a high below it + assert (r.low, r.high) == (5.0, 5.0) + (r,) = slot_readings({T0: 5.0}, lows={T0: 3.0}, highs={T0: 8.0}) + assert (r.low, r.high) == (3.0, 8.0) + + def test_a_slow_five_minute_line_is_smoothed_but_ends_on_the_latest_reading(): rs = [(T0 + i * 300, 10.0 + (i % 2), None, None, 300) for i in range(50)] plain, smooth = build_line(rs, TZ, 50 * 300), build_line(rs, TZ, 50 * 300, smooth=True) diff --git a/tests/test_periods.py b/tests/test_periods.py index 3b19450..348a829 100644 --- a/tests/test_periods.py +++ b/tests/test_periods.py @@ -6,7 +6,7 @@ from lib import intent, periods from lib.bot import Bot from lib.specs import Chart, Line, Panel -from tests.fakes import TZ +from tests.fakes import TZ, StandInTools NOW = datetime(2026, 10, 2, 12, 0) @@ -36,7 +36,7 @@ def test_only_the_newest_questions_are_remembered(): charted = periods.Charted() for i in range(periods.REMEMBERED + 5): charted.remember(1, i, f"q{i}") - assert len(charted) == periods.REMEMBERED and (1, 0) not in charted and (1, periods.REMEMBERED + 4) in charted + assert len(charted.store) == periods.REMEMBERED and charted.get(1, 0) is None and charted.get(1, periods.REMEMBERED + 4) def test_the_questions_behind_charts_survive_a_restart(tmp_path): @@ -47,7 +47,7 @@ def test_the_questions_behind_charts_survive_a_restart(tmp_path): assert again.charted.get(1, 101) == "Temperature chart 7d" -class Tools: +class Tools(StandInTools): def __init__(self): self.calls = [] @@ -57,7 +57,7 @@ async def call(self, name, raw, turn=None): return json.dumps({"period": "Fri 25 Sep 2026 - Thu 01 Oct 2026", "series": {"outdoor.temperature": {"unit": "℃", "low": "1", "high": "2"}}}) -def make_bot(monkeypatch): +def make_bot(alert_state, monkeypatch): class Agent: tools = Tools() @@ -65,7 +65,16 @@ async def run(self, *a, **k): raise AssertionError("no model for a plain chart") monkeypatch.setattr(botmod, "render_chart", lambda spec, tz, times=None: b"png") sources = [NS(name="Ecowitt", wants=lambda t: True, poke=lambda: None, describe=lambda: "Ecowitt")] - return Bot(NS(tz=TZ), Agent(), sources, None) + return Bot(NS(tz=TZ), Agent(), sources, alert_state) + + +async def test_closing_the_bot_cancels_its_uploads_still_running(alert_state, monkeypatch): + import asyncio + bot = make_bot(alert_state, monkeypatch) + upload = asyncio.create_task(asyncio.Event().wait()) + bot.background.add(upload) + await bot.close() + assert upload.cancelled() def chat_message(sent, edits=None): @@ -86,18 +95,18 @@ async def send_chat_action(*a, **k): CONTEXT = NS(bot=NS(send_chat_action=send_chat_action, id=99)) -async def test_a_chart_is_sent_with_the_other_periods_under_it_and_its_question_is_remembered(monkeypatch): - bot, sent = make_bot(monkeypatch), [] +async def test_a_chart_is_sent_with_the_other_periods_under_it_and_its_question_is_remembered(alert_state, monkeypatch): + bot, sent = make_bot(alert_state, monkeypatch), [] msg = chat_message(sent) update = NS(effective_message=msg, effective_chat=msg.chat, effective_user=NS(username="rob", full_name="Rob")) await bot.respond(update, CONTEXT, "Temperature chart 30d") caption, markup = sent[0] assert caption == "Temperature" and labels(markup) == [[("1D", "pd:1"), ("7D", "pd:7"), ("\u25cf 1M", "pd:30"), ("3M", "pd:90"), ("1Y", "pd:365")]] - assert bot.charted[(1, 101)] == "Temperature chart 30d" + assert bot.charted.get(1, 101) == "Temperature chart 30d" -async def test_pressing_a_period_redraws_the_chart_in_place_with_that_period_left_out(monkeypatch): - bot, edits = make_bot(monkeypatch), [] +async def test_pressing_a_period_redraws_the_chart_in_place_with_that_period_left_out(alert_state, monkeypatch): + bot, edits = make_bot(alert_state, monkeypatch), [] msg = chat_message([], edits) bot.charted.remember(1, 50, "Temperature chart 30d") answers = [] @@ -111,11 +120,11 @@ async def answer(text=None): assert name == "weather_history" and args["start_date"].startswith("2026-") and answers == ["Drawing 3M..."] caption, markup = edits[0] assert caption == "Temperature" and [t for t, _ in labels(markup)[0]] == ["1D", "7D", "1M", "\u25cf 3M", "1Y"] - assert bot.charted[(1, 50)] == "Temperature chart 90d" + assert bot.charted.get(1, 50) == "Temperature chart 90d" -async def test_a_button_on_a_chart_from_before_a_restart_says_so(monkeypatch): - bot = make_bot(monkeypatch) +async def test_a_button_on_a_chart_from_before_a_restart_says_so(alert_state, monkeypatch): + bot = make_bot(alert_state, monkeypatch) answers = [] async def answer(text=None): @@ -133,9 +142,8 @@ class Replies: async def reply_photo(self, photo, **kw): sent.append(kw["reply_markup"]) return NS(message_id=7) - shown = [] - carried = await botmod.deliver(Replies(), "", [b"p"], markup=keyboard(), titles=["T"], period_row=periods.keyboard(), sent_photo=shown) - assert carried is False and labels(sent[0])[0][0][0] == "1D" and shown[0].message_id == 7 + got = await botmod.deliver(Replies(), "", [b"p"], markup=keyboard(), titles=["T"], period_row=periods.keyboard()) + assert got.carried is False and labels(sent[0])[0][0][0] == "1D" and got.photo.message_id == 7 def test_a_chart_drawn_lately_comes_back_from_the_cache_until_it_is_too_old_or_pushed_out(): @@ -152,8 +160,8 @@ def test_a_chart_drawn_lately_comes_back_from_the_cache_until_it_is_too_old_or_p assert len(cache.items) == periods.ImageCache.CHART_IMAGES and "q0" not in cache.items -async def test_toggling_back_to_a_period_just_drawn_uses_the_cached_image_and_asks_nothing_again(monkeypatch): - bot, sent, edits = make_bot(monkeypatch), [], [] +async def test_toggling_back_to_a_period_just_drawn_uses_the_cached_image_and_asks_nothing_again(alert_state, monkeypatch): + bot, sent, edits = make_bot(alert_state, monkeypatch), [], [] msg = chat_message(sent, edits) update = NS(effective_message=msg, effective_chat=msg.chat, effective_user=NS(username="rob", full_name="Rob")) await bot.respond(update, CONTEXT, "Temperature chart 7d") @@ -169,8 +177,8 @@ async def answer(text=None): assert [e[0] for e in edits] == ["Temperature"] * 4 # the repeated 30d press edited nothing and bot.charted[(1, 101)] == "Temperature chart 7d" -async def test_a_chart_once_sent_goes_again_as_telegrams_copy_not_a_new_upload(monkeypatch): - bot, uploads, edits = make_bot(monkeypatch), [], [] +async def test_a_chart_once_sent_goes_again_as_telegrams_copy_not_a_new_upload(alert_state, monkeypatch): + bot, uploads, edits = make_bot(alert_state, monkeypatch), [], [] async def reply_photo(photo, caption=None, **kw): uploads.append(photo) @@ -192,8 +200,8 @@ async def answer(text=None): assert uploads == [b"png"] and edits == ["upload", "7d", "edit 1"] # 30d drawn and uploaded once; then both by Telegram's copy -async def test_redrawing_a_chart_from_a_button_shows_no_typing_or_thinking(monkeypatch): - bot = make_bot(monkeypatch) +async def test_redrawing_a_chart_from_a_button_shows_no_typing_or_thinking(alert_state, monkeypatch): + bot = make_bot(alert_state, monkeypatch) actions = [] async def chat_action(*a, **k): @@ -221,8 +229,8 @@ async def reply_photo(photo, caption=None, **kw): return NS(effective_message=msg, effective_chat=msg.chat, effective_user=NS(username="rob", full_name="Rob")) -async def test_the_button_charts_are_drawn_ahead_and_a_press_is_sent_one_at_once(monkeypatch, caplog): - bot, sent = make_bot(monkeypatch), [] +async def test_the_button_charts_are_drawn_ahead_and_a_press_is_sent_one_at_once(alert_state, monkeypatch, caplog): + bot, sent = make_bot(alert_state, monkeypatch), [] with caplog.at_level("INFO"): await bot.predraw() calls = len(bot.agent.tools.calls) @@ -242,9 +250,9 @@ async def test_the_button_charts_are_drawn_ahead_and_a_press_is_sent_one_at_once assert not any("drawn ahead" in r.getMessage() for r in caplog.records) # only the first round is logged -async def test_a_chart_drawn_ahead_is_drawn_again_once_it_is_stale_and_never_answers_other_words(monkeypatch): +async def test_a_chart_drawn_ahead_is_drawn_again_once_it_is_stale_and_never_answers_other_words(alert_state, monkeypatch): clock = [0.0] - bot, sent = make_bot(monkeypatch), [] + bot, sent = make_bot(alert_state, monkeypatch), [] bot.images = periods.ImageCache(lambda: clock[0]) await bot.predraw() calls = len(bot.agent.tools.calls) @@ -257,9 +265,9 @@ async def test_a_chart_drawn_ahead_is_drawn_again_once_it_is_stale_and_never_ans assert len(bot.agent.tools.calls) == calls + 1 -async def test_charts_are_not_drawn_ahead_while_a_question_is_being_answered(monkeypatch): +async def test_charts_are_not_drawn_ahead_while_a_question_is_being_answered(alert_state, monkeypatch): import asyncio - bot = make_bot(monkeypatch) + bot = make_bot(alert_state, monkeypatch) bot.answering = 1 bot.idle.clear() drawing = asyncio.create_task(bot.predraw()) @@ -274,11 +282,11 @@ async def test_charts_are_not_drawn_ahead_while_a_question_is_being_answered(mon -def preloading_bot(monkeypatch, refuse=None, clock=None): +def preloading_bot(alert_state, monkeypatch, refuse=None, clock=None): """A bot with TELEGRAM_PRELOAD_CHAT set, and the (chat, photo, silent) of every upload to it. The periods are uploaded side by side, in no fixed order, so each period's picture is its question's words and its file id "file " and those words; context.refused.error is what the uploads raise (none: they go through).""" - bot, uploads, refused = make_bot(monkeypatch), [], NS(error=refuse) + bot, uploads, refused = make_bot(alert_state, monkeypatch), [], NS(error=refuse) bot.cfg = NS(tz=TZ, telegram_preload_chat=-1001) if clock: bot.images = periods.ImageCache(lambda: clock[0]) @@ -307,8 +315,8 @@ async def settle(bot): await asyncio.gather(*bot.background) -async def test_the_other_periods_of_a_chart_are_uploaded_ahead_and_a_press_edits_by_file_id(monkeypatch): - bot, uploads, context = preloading_bot(monkeypatch) +async def test_the_other_periods_of_a_chart_are_uploaded_ahead_and_a_press_edits_by_file_id(alert_state, monkeypatch): + bot, uploads, context = preloading_bot(alert_state, monkeypatch) sent, edits = [], [] async def edit_media(media, reply_markup=None): @@ -329,15 +337,15 @@ async def answer(text=None): assert len(bot.agent.tools.calls) == calls and edits[0] == "file Temperature chart 90d" # Telegram's copy of the 3M, no new question or upload -async def test_nothing_is_uploaded_ahead_without_the_setting_and_a_refused_upload_stops_it_with_one_warning(monkeypatch, caplog): - bot, uploads, context = preloading_bot(monkeypatch) +async def test_nothing_is_uploaded_ahead_without_the_setting_and_a_refused_upload_stops_it_with_one_warning(alert_state, monkeypatch, caplog): + bot, uploads, context = preloading_bot(alert_state, monkeypatch) bot.cfg = NS(tz=TZ) update = pressed(bot, []) await bot.respond(update, context, "Temperature chart 7d") await settle(bot) assert uploads == [] and not bot.background from telegram.error import BadRequest - bot, uploads, context = preloading_bot(monkeypatch, refuse=BadRequest("Chat not found")) + bot, uploads, context = preloading_bot(alert_state, monkeypatch, refuse=BadRequest("Chat not found")) with caplog.at_level("WARNING"): await bot.respond(pressed(bot, []), context, "Temperature chart 7d") await settle(bot) @@ -346,13 +354,13 @@ async def test_nothing_is_uploaded_ahead_without_the_setting_and_a_refused_uploa assert [r for r in caplog.records if "TELEGRAM_PRELOAD_CHAT" in r.getMessage()].__len__() == 1 and uploads == [] and bot.preload_off -async def test_a_flood_limit_or_a_timeout_skips_those_uploads_and_the_next_chart_uploads_again(monkeypatch, caplog): +async def test_a_flood_limit_or_a_timeout_skips_those_uploads_and_the_next_chart_uploads_again(alert_state, monkeypatch, caplog): import warnings from telegram.error import RetryAfter, TimedOut with warnings.catch_warnings(): warnings.simplefilter("ignore") # python-telegram-bot 22 warns whenever it builds a RetryAfter flood = RetryAfter(30) - bot, uploads, context = preloading_bot(monkeypatch) + bot, uploads, context = preloading_bot(alert_state, monkeypatch) for error, asked in ((flood, "Temperature chart 7d"), (TimedOut(), "Temperature chart 30d")): context.refused.error = error with caplog.at_level("WARNING"): @@ -366,8 +374,8 @@ async def test_a_flood_limit_or_a_timeout_skips_those_uploads_and_the_next_chart assert sorted(uploads) == uploads_of(1, 365) # the 7D and the 1M were sent as the questions' own charts -async def test_a_period_that_cannot_be_drawn_is_left_out_and_the_others_are_uploaded(monkeypatch): - bot, uploads, context = preloading_bot(monkeypatch) +async def test_a_period_that_cannot_be_drawn_is_left_out_and_the_others_are_uploaded(alert_state, monkeypatch): + bot, uploads, context = preloading_bot(alert_state, monkeypatch) draw = bot._draw_ahead async def draw_ahead(text, now): @@ -381,9 +389,9 @@ async def draw_ahead(text, now): assert bot.images.has_copy(bot.images.key("Temperature chart 30d", 30, NOW)) and not bot.images.has_copy(bot.images.key("Temperature chart 90d", 90, NOW)) -async def test_charts_are_uploaded_ahead_at_once_even_while_a_question_is_being_answered(monkeypatch): +async def test_charts_are_uploaded_ahead_at_once_even_while_a_question_is_being_answered(alert_state, monkeypatch): import asyncio - bot, uploads, context = preloading_bot(monkeypatch) + bot, uploads, context = preloading_bot(alert_state, monkeypatch) bot.answering = 1 bot.idle.clear() bot.preload_soon(context.bot, "Temperature chart 7d") @@ -391,9 +399,9 @@ async def test_charts_are_uploaded_ahead_at_once_even_while_a_question_is_being_ assert sorted(uploads) == uploads_of(1, 30, 90, 365) -async def test_each_period_is_uploaded_as_soon_as_it_is_drawn_not_when_the_slowest_is(monkeypatch): +async def test_each_period_is_uploaded_as_soon_as_it_is_drawn_not_when_the_slowest_is(alert_state, monkeypatch): import asyncio - bot, uploads, context = preloading_bot(monkeypatch) + bot, uploads, context = preloading_bot(alert_state, monkeypatch) slow, three, draw, send = asyncio.Event(), asyncio.Event(), bot._draw_ahead, context.bot.send_photo async def draw_ahead(text, now): @@ -415,9 +423,9 @@ async def send_photo(chat, photo, **kw): assert sorted(uploads) == uploads_of(1, 30, 90, 365) -async def test_a_chart_uploaded_ahead_is_used_for_30_minutes_and_then_drawn_afresh(monkeypatch): +async def test_a_chart_uploaded_ahead_is_used_for_30_minutes_and_then_drawn_afresh(alert_state, monkeypatch): clock = [0.0] - bot, uploads, context = preloading_bot(monkeypatch, clock=clock) + bot, uploads, context = preloading_bot(alert_state, monkeypatch, clock=clock) await bot.respond(pressed(bot, []), context, "Temperature chart 7d") await settle(bot) key = bot.images.key("Temperature chart 30d", 30, NOW) diff --git a/tests/test_pollen.py b/tests/test_pollen.py index 0201a2c..2a919a8 100644 --- a/tests/test_pollen.py +++ b/tests/test_pollen.py @@ -61,11 +61,31 @@ async def test_the_lines_have_a_level_emoji_and_never_name_the_district(tmp_path await pollen.close() -async def test_a_forecast_for_another_day_and_a_missing_asthma_forecast_are_said_so(tmp_path): +async def test_a_forecast_for_another_day_and_a_missing_asthma_forecast_are_said_so(tmp_path, caplog): pollen, _ = make(tmp_path, lambda: page(grass="Moderate", day=SEASON_DAY + timedelta(days=1), season=False)) - await pollen.start() + with caplog.at_level("WARNING"): + await pollen.start() assert pollen.lines() == ["Grass pollen: 🟠 Moderate (forecast for Wed 11 Nov)"] - assert "asthma_note" in json.loads(await pollen.handle({})) + assert json.loads(await pollen.handle({}))["asthma_note"] == "The page had no thunderstorm asthma level for this district when it was last read." + assert "no thunderstorm asthma level for Central on the page" in caplog.text # in season the page should have one + await pollen.close() + + +def test_a_heading_with_words_after_it_or_in_another_case_is_still_found(): + reworded = page(grass="Low", asthma="High").replace("

Thunderstorm Asthma Forecast

", + "

Thunderstorm asthma forecast for Tuesday 10 November

") + got = parse_melbourne_pollen(reworded) + assert got["thunderstorm_asthma"] == {"Central": "High", "Wimmera": "High"} and got["asthma_updated"] == "1 Oct 2026 4:00pm" + assert got["district_grass"] == {"Central": "Low", "Mallee": "Moderate"} # the grass list still ends at it + + +async def test_the_district_is_the_sites_whatever_its_case_and_one_it_lacks_is_warned_about(tmp_path, caplog): + pollen, _ = make(tmp_path, pollen_district="north east") + assert pollen.district == "North East" + await pollen.close() + with caplog.at_level("WARNING"): + pollen, _ = make(tmp_path, pollen_district="Centrl") + assert pollen.district == "Centrl" and "POLLEN_DISTRICT 'Centrl' is not one of the site's districts" in caplog.text await pollen.close() @@ -113,10 +133,10 @@ def test_pollen_words_need_data_and_a_bare_pollen_question_is_fetched_without_th from lib import intent now = datetime(2026, 10, 1, 12, 0) for text in ("pollen", "Hay fever", "thunderstorm asthma risk"): - assert intent.read(text, now, True, True, True).fast[0] == "pollen_asthma" - assert intent.read("pollen", now, True, True, False).fast is None + assert intent.read(text, now, ["Ecowitt", "AirGradient", "Pollen"]).fast[0] == "pollen_asthma" + assert intent.read("pollen", now).fast is None for text in ("is the pollen bad today?", "any asthma risk"): - assert intent.read(text, now, True, True, True).needs_data + assert intent.read(text, now, ["Ecowitt", "AirGradient", "Pollen"]).needs_data def make_on(tmp_path, day, hour=12, html=None): @@ -156,10 +176,24 @@ async def test_last_seasons_page_is_not_shown_and_the_report_leaves_the_block_ou assert january.data is not None and january.lines() == [] and january.current()["grass"] is None and calls == [] result = await january.handle({"cached": True}) assert "error" in json.loads(result) - assert "Pollen" not in report.report({"pollen_asthma": result}) # no block in the report + assert "Pollen" not in report.report({"pollen_asthma": result}, date(2027, 1, 5)) # no block in the report await january.close() +async def test_last_seasons_page_is_never_shown_on_the_first_of_october_and_is_fetched_again_even_at_night(tmp_path): + december = lambda: page(grass="High", asthma="High", day=date(2025, 12, 31)) + pollen, _ = make_on(tmp_path, date(2025, 12, 31), html=december) + await pollen.start() + await pollen.close() + october, calls = make_on(tmp_path, date(2026, 10, 1), hour=3, html=december) # a restart at 3 am, the saved page December's + assert october.data is not None and october.current()["grass"] is None and october.current()["asthma"] is None + await october.warm(True) # outside the sync hours: fetched all the same + assert len(calls) == 1 and october.current()["grass"] is None # the site still has December's page (not modified) + october.data["melbourne_date"], october.fetched_at = None, datetime(2025, 12, 31, 12, tzinfo=TZ).timestamp() + assert october.page is None # with no date on the page, when it was fetched says + await october.close() + + async def test_the_first_warm_on_the_first_of_october_fetches_without_a_restart(tmp_path): clock = {"day": date(2026, 9, 30)} pollen, calls = make(tmp_path) diff --git a/tests/test_rain.py b/tests/test_rain.py index eaf80b9..4da5535 100644 --- a/tests/test_rain.py +++ b/tests/test_rain.py @@ -63,3 +63,22 @@ def test_a_stacked_chart_over_a_rolling_day_with_no_rain_has_no_axis_left_of_its bars = rain_bars({}, tz, date(2026, 10, 7), date(2026, 10, 8), until=until, since=since) panel = Panel("Humidity", "%", [Line("Outdoor", xs, [60.0 + (i % 5) for i in range(len(xs))], reading="humidity")], bars=bars, reading="humidity") assert min(panel.xs) == bars.x[0] and since - min(panel.xs) < 3600 # at most the hour the data starts in + + +def test_rain_bars_stay_on_local_days_and_hours_across_a_daylight_saving_change(): + """5 Apr 2026 is 25 hours long in Melbourne and 4 Oct 23: the bars keep starting at midnight (and 6am, noon, 6pm), so rain just + after midnight is the new day's, not the day before's.""" + from datetime import date, datetime + from zoneinfo import ZoneInfo + from lib.compose import rating_shares + from lib.rain import rain_bars + from lib.timeutil import day_bounds + tz = ZoneInfo("Australia/Melbourne") + after_midnight = day_bounds(date(2026, 4, 10), tz)[0] + 1800 + days = rain_bars({after_midnight: 3.0}, tz, date(2026, 3, 1), date(2026, 4, 30)) + assert len(days.x) == 61 and all(datetime.fromtimestamp(x, tz).hour == 0 for x in days.x) + assert datetime.fromtimestamp(days.x[days.y.index(3.0)], tz).date() == date(2026, 4, 10) + hours = rain_bars({}, tz, date(2026, 10, 1), date(2026, 10, 10)) + assert {datetime.fromtimestamp(x, tz).hour for x in hours.x} == {0, 6, 12, 18} and len(hours.x) == 40 + shares = rating_shares({after_midnight: 5.0}, (12.0, 35.0), tz, date(2026, 3, 1), date(2026, 4, 30), "PM2.5 rating") + assert datetime.fromtimestamp(shares.x[0], tz) == datetime(2026, 4, 10, tzinfo=tz) diff --git a/tests/test_report.py b/tests/test_report.py index 42c553c..84dbfdd 100644 --- a/tests/test_report.py +++ b/tests/test_report.py @@ -1,9 +1,10 @@ import json +from datetime import date from lib import report -# What `python scripts/report_inputs.py` printed on the real station (rain, night) -NOW = ('{"time":"Fri 02 Oct 2026 02:21","outdoor":{"temperature":"12.3 ℃","feels_like":"12.3 ℃","app_temp":"12.7 ℃","dew_point":"11.3 ℃",' +# What `python scripts/report_inputs.py` printed on the real station (rain, night), its time as the station now writes it +NOW = ('{"time":"Fri 2 Oct 2026 2:21am","outdoor":{"temperature":"12.3 ℃","feels_like":"12.3 ℃","app_temp":"12.7 ℃","dew_point":"11.3 ℃",' '"vpd":"0.085 kPa","humidity":"94 %"},"indoor":{"temperature":"18.3 ℃","humidity":"63 %","dew_point":"11.1 ℃","feels_like":"18.3 ℃",' '"app_tempin":"18.7 ℃"},"solar_and_uvi":{"solar":"0.0 W/m²","uvi":"0"},"rainfall":{"rain_rate":"1.8 mm/hr","daily":"6.3 mm",' '"event":"49.8 mm","1_hour":"1.3 mm","weekly":"49.8 mm","monthly":"49.8 mm","yearly":"5105.7 mm"},"wind":{"wind_speed":"0.0 km/h",' @@ -20,6 +21,8 @@ FORECAST = ('{"lines": ["Today: 🌦️ Showers. 11–16°C, 97% chance of rain", "Saturday: ⛈️ Thunderstorm. 12–20°C, 95% chance of rain"], ' '"place": "Melbourne", "source": "Open-Meteo", "time": "Fri 2 Oct 2026 6:05pm"}') +TODAY = date(2026, 10, 2) # the day the results above are from + WEATHER = """\ • Outdoor: 🧥 12.3 °C, 💦 94 %, dew point 11.3 °C, VPD 0.085 kPa • Indoor: 🧥 18.3 °C, 63 % @@ -28,7 +31,7 @@ def test_the_report_is_laid_out_in_code_with_emoji_before_the_readings_they_belong_to(): - got = report.report({"weather_now": NOW, "air_quality": AIR, "pollen_asthma": POLLEN, "weather_forecast": FORECAST}) + got = report.report({"weather_now": NOW, "air_quality": AIR, "pollen_asthma": POLLEN, "weather_forecast": FORECAST}, TODAY) assert got == f"""Weather station (2:21am) {WEATHER} @@ -62,15 +65,16 @@ def test_a_dry_calm_day_has_no_rain_line_noise(): assert lines == ["• Outdoor: 21.0 °C, 50 %", "• Rain today: 0.0 mm (month total 12.0 mm)", "• Wind: 12.0 km/h from 270°, gust 20.0 km/h"] drizzle = dry.replace('"0.0 mm/hr"', '"0.6 mm/hr"').replace('"emoji":{}', '"emoji":{"rainfall.rain_rate":"🌧️"}') assert "; 🌧️ raining (0.6 mm/h)" in report.weather_now(drizzle) + assert "raining" not in report.weather_now(dry.replace('"0.0 mm/hr"', '"0.00 mm/hr"')) # however the zero is written def test_a_failed_or_missing_source_is_left_out_or_said_so_and_the_forecast_without_a_tag_has_a_plain_heading(): got = report.report({"weather_now": "Error calling tool: boom", "air_quality": AIR, "pollen_asthma": '{"error": "No pollen levels are available right now."}', - "weather_forecast": '{"lines": ["Today: ☀️ Sunny. 20–25°C"]}'}) + "weather_forecast": '{"lines": ["Today: ☀️ Sunny. 20–25°C"]}'}, TODAY) assert got.startswith("Weather station\n• not available right now\n\nAir quality (2:22am)\n• PM2.5:") assert "Pollen" not in got and got.endswith("Forecast\n• Today: ☀️ Sunny. 20–25°C") - assert report.report({}) == "No readings are available right now." + assert report.report({}, TODAY) == "No readings are available right now." def test_sun_uv_and_wind_are_left_out_when_zero_and_shown_when_not(): @@ -182,15 +186,20 @@ def test_a_reading_lookup_prints_only_what_was_asked(): assert report.reading_lines(now, ["humidity"], ["outdoor"]) == ["• Outdoor: 💦 94 %"] assert report.reading_lines(now, ["wind"], []) == ["• Wind: calm"] assert report.reading_lines(now, ["pressure"], []) == ["• Pressure: 🗜️ 1025.0 hPa"] + assert report.reading_lines(now, ["dew_point"], ["indoor"]) == ["• Dew point (indoor): 11.1 °C"] # "indoor dew point" + assert report.reading_lines(now, ["feels_like"], ["outdoor", "indoor"])[0].startswith("• Feels like: ") # unasked: outdoor assert report.reading_lines(now, ["rain"], [])[0].startswith("• Rain today: ☔ 6.3 mm") and "raining (1.8 mm/h)" in report.reading_lines(now, ["rain"], [])[0] + windy = json.loads(NOW.replace('"wind_speed":"0.0 km/h"', '"wind_speed":"14.0 km/h"')) + assert [line for line in report.weather_lines(windy) if line.startswith(("• Wind", "• Rain"))] == \ + report.reading_lines(windy, ["rain", "wind"], []) # the same words as the report's def test_air_pollen_and_forecast_lookups_print_the_tools_own_lines(): - assert report.lookup("air", AIR, ["pm10"], []) == "• PM10: 3.0 µg/m³ 🟢 good" - assert report.lookup("air", AIR, [], []).splitlines()[0] == "• PM2.5: 1.4 µg/m³ 🟢 good (AQI 8)" - assert report.lookup("pollen", POLLEN, [], []) == "Pollen & asthma (Melbourne, 6:00pm)\n• Grass pollen: 🟢 Low\n• Thunderstorm asthma risk: 🟢 Low" - assert report.lookup("forecast", FORECAST, [], []).startswith("Forecast (Melbourne, 6:05pm)\n• Today: ") - assert report.lookup("pollen", json.dumps({"error": "off season"}), [], []) is None + assert report.lookup("air", AIR, ["pm10"], [], TODAY) == "• PM10: 3.0 µg/m³ 🟢 good" + assert report.lookup("air", AIR, [], [], TODAY).splitlines()[0] == "• PM2.5: 1.4 µg/m³ 🟢 good (AQI 8)" + assert report.lookup("pollen", POLLEN, [], [], TODAY) == "Pollen & asthma (Melbourne, 6:00pm)\n• Grass pollen: 🟢 Low\n• Thunderstorm asthma risk: 🟢 Low" + assert report.lookup("forecast", FORECAST, [], [], TODAY).startswith("Forecast (Melbourne, 6:05pm)\n• Today: ") + assert report.lookup("pollen", json.dumps({"error": "off season"}), [], [], TODAY) is None def test_a_short_period_highs_and_lows_lookup_gives_each_sides_extremes_with_their_times(): @@ -198,7 +207,7 @@ def test_a_short_period_highs_and_lows_lookup_gives_each_sides_extremes_with_the "outdoor.temperature": {"unit": "℃", "low": "9.0", "low_when": "at 5am", "low_date": "Fri 2 Oct 2026", "high": "16.2", "high_when": "at 2pm", "high_date": "Fri 2 Oct 2026"}, "outdoor.humidity": {"unit": "%", "low": "40", "high": "95"}}}) - assert report.lookup("extremes", result, [], []) == ("Fri 02 Oct – Fri 02 Oct 2026\n" + assert report.lookup("extremes", result, [], [], TODAY) == ("Fri 02 Oct – Fri 02 Oct 2026\n" "• Temperature: low 9.0 °C, Fri 2 Oct at 5am · high 16.2 °C, Fri 2 Oct at 2pm") @@ -207,34 +216,41 @@ def test_an_average_or_highs_lookup_covers_the_readings_named(): "outdoor.temperature": {"unit": "℃", "low": "9.0", "high": "16.2", "average": "12.4"}, "outdoor.humidity": {"unit": "%", "low": "40", "high": "95", "low_when": "at 3pm", "low_date": "Thu 1 Oct 2026", "high_when": "at 5am", "high_date": "Thu 1 Oct 2026"}, "wind.wind_speed": {"unit": "km/h", "low": "0", "high": "14"}, "wind.wind_gust": {"unit": "km/h", "low": "0", "high": "31", "high_when": "at 2pm", "high_date": "Thu 1 Oct 2026"}}}) - assert report.lookup("extremes", result, ["temperature"], []).splitlines()[1] == "• Temperature: average 12.4 °C (low 9.0, high 16.2)" - assert report.lookup("extremes", result, ["humidity"], []).splitlines()[1] == "• Humidity: low 40 %, Thu 1 Oct at 3pm · high 95 %, Thu 1 Oct at 5am" - assert [line.split(":")[0] for line in report.lookup("extremes", result, ["wind"], []).splitlines()[1:]] == ["• Wind", "• Wind gust"] + assert report.lookup("extremes", result, ["temperature"], [], TODAY).splitlines()[1] == "• Temperature: average 12.4 °C (low 9.0, high 16.2)" + assert report.lookup("extremes", result, ["humidity"], [], TODAY).splitlines()[1] == "• Humidity: low 40 %, Thu 1 Oct at 3pm · high 95 %, Thu 1 Oct at 5am" + assert [line.split(":")[0] for line in report.lookup("extremes", result, ["wind"], [], TODAY).splitlines()[1:]] == ["• Wind", "• Wind gust"] def test_a_days_count_lookup_gives_the_count_the_condition_and_the_period(): result = json.dumps({"period": "Wed 5 Oct 2022 - Sat 3 Oct 2026", "days_checked": 1454, "matching_days": 137, "condition": "UV index ≥ 10"}) - assert report.lookup("days", result, [], []) == "137 of 1,454 days with UV index ≥ 10 (Wed 5 Oct 2022 – Sat 3 Oct 2026)." + assert report.lookup("days", result, [], [], TODAY) == "137 of 1,454 days with UV index ≥ 10 (Wed 5 Oct 2022 – Sat 3 Oct 2026)." averaged = json.dumps({"period": "a - b", "days_checked": 10, "matching_days": 2, "note_averaged": "x"}) - assert "may fall slightly short" in report.lookup("days", averaged, [], []) - assert report.lookup("days", json.dumps({"error": "bad"}), [], []) is None + assert "may fall slightly short" in report.lookup("days", averaged, [], [], TODAY) + assert report.lookup("days", json.dumps({"error": "bad"}), [], [], TODAY) is None def test_a_figure_per_period_lookup_gives_the_figure_over_the_whole_period(): result = json.dumps({"period": "Fri 4 Oct 2024 - Sat 3 Oct 2026", "days_checked": 731, "matching_days": 731, "stat": {"what": "Total rain", "of": "rain", "unit": "mm"}, "value": 1234.5, "by_month": {"2024-10": 0.0}}) - assert report.lookup("days", result, [], []) == "Total rain: 1234.5 mm over 731 days (Fri 4 Oct 2024 – Sat 3 Oct 2026)." + assert report.lookup("days", result, [], [], TODAY) == "Total rain: 1234.5 mm over 731 days (Fri 4 Oct 2024 – Sat 3 Oct 2026)." hot = json.dumps({"period": "a 2025 - b 2026", "days_checked": 3, "matching_days": 3, "stat": {"what": "Highest temperature", "of": "temp_max", "unit": "℃"}, "value": 41.2}) - assert report.lookup("days", hot, [], []).startswith("Highest temperature: 41.2 °C over 3 days") + assert report.lookup("days", hot, [], [], TODAY).startswith("Highest temperature: 41.2 °C over 3 days") def test_a_heading_carries_the_place_and_the_time_of_day_of_its_data(): - assert report._heading("Forecast", {"place": "Melbourne", "time": "Fri 2 Oct 2026 6:05pm"}) == "Forecast (Melbourne, 6:05pm)" - assert report._heading("Weather station", {"time": "Fri 02 Oct 2026 00:07"}) == "Weather station (12:07am)" # the station's 24-hour form - assert report._heading("Weather station", {"time": "Fri 02 Oct 2026 13:30"}) == "Weather station (1:30pm)" - assert report._heading("Air quality", {"time": "Fri 2 Oct 2026 12:00pm"}) == "Air quality (12:00pm)" - assert report._heading("Forecast", {"place": "Melbourne"}) == "Forecast (Melbourne)" # no time: as before - assert report._heading("Air quality", {"time": "yesterday"}) == "Air quality" and report._heading("Air quality", {"time": None}) == "Air quality" + assert report._heading("Forecast", {"place": "Melbourne", "time": "Fri 2 Oct 2026 6:05pm"}, TODAY) == "Forecast (Melbourne, 6:05pm)" + assert report._heading("Weather station", {"time": "Fri 2 Oct 2026 12:07am"}, TODAY) == "Weather station (12:07am)" + assert report._heading("Weather station", {"time": "Fri 2 Oct 2026 1:30pm"}, TODAY) == "Weather station (1:30pm)" + assert report._heading("Air quality", {"time": "Fri 2 Oct 2026 12:00pm"}, TODAY) == "Air quality (12:00pm)" + assert report._heading("Forecast", {"place": "Melbourne"}, TODAY) == "Forecast (Melbourne)" # no time: as before + assert report._heading("Air quality", {"time": "yesterday"}, TODAY) == "Air quality" and report._heading("Air quality", {"time": None}, TODAY) == "Air quality" + + +def test_a_heading_gives_the_day_of_data_that_is_not_todays(): + stale = report.report({"weather_now": NOW, "pollen_asthma": POLLEN}, date(2026, 10, 4)) # the station has been silent since Friday + assert stale.startswith("Weather station (Fri 2 Oct 2:21am)\n") and "\nPollen & asthma (Melbourne, Fri 2 Oct 6:00pm)\n" in stale + assert report.lookup("forecast", FORECAST, [], [], date(2026, 10, 3)).startswith("Forecast (Melbourne, Fri 2 Oct 6:05pm)\n") + assert [e.length for e in report.heading_entities(stale)] == [len("Weather station"), len("Pollen & asthma")] # still bold def test_when_gives_the_form_the_tools_use_in_their_time_field(): @@ -249,7 +265,7 @@ def test_when_gives_the_form_the_tools_use_in_their_time_field(): def test_only_the_name_of_each_heading_is_bold_not_its_brackets_or_time(): - text = report.report({"weather_now": NOW, "air_quality": AIR, "pollen_asthma": POLLEN, "weather_forecast": FORECAST}) + text = report.report({"weather_now": NOW, "air_quality": AIR, "pollen_asthma": POLLEN, "weather_forecast": FORECAST}, TODAY) units = lambda s: len(s.encode("utf-16-le")) // 2 names = [text.encode("utf-16-le")[2 * e.offset:2 * (e.offset + e.length)].decode("utf-16-le") for e in report.heading_entities(text)] assert names == ["Weather station", "Air quality", "Pollen & asthma", "Forecast"] diff --git a/tests/test_scan.py b/tests/test_scan.py index 3eeee34..e5fd833 100644 --- a/tests/test_scan.py +++ b/tests/test_scan.py @@ -1,9 +1,12 @@ import json import time +from datetime import UTC, date, datetime, time as dtime, timedelta +from zoneinfo import ZoneInfo import numpy as np from lib.analysis import scan as correlate +from lib.timeutil import day_bounds from lib.tools import Turn from tests.test_compose import composer @@ -33,7 +36,7 @@ def test_a_planted_relationship_is_found_with_its_direction_and_nothing_else(): rng, n, _, cycle, wind = world() air = {"pm2_5": series(20 - 0.5 * wind + 3 * cycle + rng.normal(0, 2, n)), "co2": series(450 + 30 * cycle + rng.normal(0, 10, n))} weather = {"wind": series(wind), "temperature": series(15 + 8 * cycle + rng.normal(0, 1, n)), "noise": series(rng.normal(0, 1, n))} - result = correlate.scan(air, weather, None, None, ORIGIN, DAYS) + result = correlate.scan(air, weather, None, None, ORIGIN, DAYS, UTC) kept = [(t["air"], t["weather"]) for t in result["tests"] if t["survives"]] assert kept == [("pm2_5", "wind")] # the shared daily cycle and the noise are not links found = next(t for t in result["tests"] if t["weather"] == "wind" and t["air"] == "pm2_5") @@ -47,7 +50,7 @@ def test_pure_noise_finds_nothing_even_across_many_pairs(): rng, n, _, _, _ = world(1) air = {f"a{i}": series(rng.normal(0, 1, n)) for i in range(3)} weather = {f"w{i}": series(rng.normal(0, 1, n)) for i in range(6)} - result = correlate.scan(air, weather, None, None, ORIGIN, DAYS, shuffles=100) + result = correlate.scan(air, weather, None, None, ORIGIN, DAYS, UTC, shuffles=100) assert not any(t["survives"] for t in result["tests"]) and len(result["tests"]) == 18 out = correlate.summarise(result, *names(air, weather)) assert out["verdict"].startswith("Nothing stands out") and out["findings"][0].startswith("Closest, probably chance") @@ -57,7 +60,7 @@ def test_a_lagged_relationship_is_found_at_its_lag(): rng, n, _, cycle, wind = world(2) lagged = np.roll(wind, 6) # the air answers the wind three hours later air = {"pm2_5": series(20 - 0.5 * lagged + rng.normal(0, 1, n))} - result = correlate.scan(air, {"wind": series(wind)}, None, None, ORIGIN, DAYS) + result = correlate.scan(air, {"wind": series(wind)}, None, None, ORIGIN, DAYS, UTC) test = result["tests"][0] assert test["survives"] and test["lag_hours"] == 3.0 and test["r"] < -0.3 @@ -67,7 +70,7 @@ def test_the_air_by_wind_direction_finds_the_sector_it_is_worst_from(): days = np.arange(n) // 48 direction = np.where(days % 3 == 0, 45.0, np.where(days % 3 == 1, 225.0, 135.0)) # north-east, south-west or south-east days air = {"pm2_5": series(10 + 8 * (direction == 45.0) + rng.normal(0, 1, n))} - result = correlate.scan(air, {"wind": series(wind)}, series(direction), series(np.full(n, 20.0)), ORIGIN, DAYS) + result = correlate.scan(air, {"wind": series(wind)}, series(direction), series(np.full(n, 20.0)), ORIGIN, DAYS, UTC) d = result["directions"][0] assert d["survives"] and d["highest"] == "NE" and d["lowest"] in ("SW", "SE") and d["highest_by"] > 3 out = correlate.summarise(result, *names(air, {"wind": 1})) @@ -77,7 +80,7 @@ def test_the_air_by_wind_direction_finds_the_sector_it_is_worst_from(): def test_too_few_days_says_so(): rng, n, _, _, wind = world(4) short = {t: v for t, v in series(rng.normal(0, 1, n)).items() if t < ORIGIN + 10 * 86400} - result = correlate.scan({"pm2_5": short}, {"wind": series(wind)}, None, None, ORIGIN, DAYS) + result = correlate.scan({"pm2_5": short}, {"wind": series(wind)}, None, None, ORIGIN, DAYS, UTC) assert result["tests"] == [] and "at least 21" in result["note"] assert correlate.summarise(result, {"pm2_5": ("PM2.5", "")}, {})["verdict"] == "Not enough data to say" @@ -94,7 +97,7 @@ def test_a_full_size_scan_of_every_pair_completes(): air = {m: series(rng.normal(0, 1, n) + cycle) for m in ("pm2_5", "pm10", "co2", "voc_index", "nox_index")} weather = {w: series(rng.normal(0, 1, n) + cycle) for w in ("temperature", "humidity", "dew_point", "pressure", "wind_speed", "wind_gust", "pressure_change", "rain")} started = time.time() - result = correlate.scan(air, weather, series(rng.uniform(0, 360, n)), series(np.full(n, 20.0)), ORIGIN, DAYS, shuffles=60) + result = correlate.scan(air, weather, series(rng.uniform(0, 360, n)), series(np.full(n, 20.0)), ORIGIN, DAYS, UTC, shuffles=60) assert len(result["tests"]) == 40 and len(result["directions"]) == 5 and time.time() - started < 30 @@ -107,3 +110,27 @@ async def test_air_scan_answers_with_a_verdict_and_can_chart_the_strongest_pair( assert all(len(s.panels) == 2 or s.panels[0].bars for s in specs) # two readings (rain behind the other): only when a relationship stood out assert "error" in json.loads(await comp.air_scan({"metric": "pm1"})) await eco.close() + + +def test_a_time_of_day_keeps_its_slot_across_a_daylight_saving_change(): + tz = ZoneInfo("Australia/Melbourne") + for first in (date(2026, 4, 3), date(2026, 10, 2)): # 5 April gains an hour, 4 October loses one + origin = day_bounds(first, tz)[0] + nine = {int(datetime.combine(first + timedelta(days=d), dtime(9), tz).timestamp()): 1.0 for d in range(4)} + grid = correlate.to_grid(nine, origin, 4, correlate.clock_slots(origin, 4, tz)) + assert list(np.flatnonzero(~np.isnan(grid))) == [18 + 48 * d for d in range(4)] # 9 am every day, not 8 or 10 + + +def test_no_weather_readings_say_there_was_nothing_to_compare_not_that_nothing_stands_out(): + rng, n, _, _, _ = world(6) + result = correlate.scan({"pm2_5": series(rng.normal(0, 1, n))}, {}, None, None, ORIGIN, DAYS, UTC) + assert result["tests"] == [] and "No weather readings" in result["note"] + assert correlate.summarise(result, {"pm2_5": ("PM2.5", "")}, {})["verdict"] == "Not enough data to say" + + +async def test_a_scan_of_years_covers_its_latest_400_days_and_says_so(tmp_path, archived_cache): + comp, eco = await composer(tmp_path, archived_cache) + out = json.loads(await comp.air_scan({"start_date": "2020-01-01"})) + first, last = (date.fromisoformat(d) for d in out["period"].split(" to ")) + assert (last - first).days == 399 and "Only the last 400 days" in out["notes"]["period"] + await eco.close() diff --git a/tests/test_snapshots.py b/tests/test_snapshots.py index d9f7c3b..e82fc39 100644 --- a/tests/test_snapshots.py +++ b/tests/test_snapshots.py @@ -14,27 +14,34 @@ from tests.test_pollen import SEASON_DAY, make as make_pollen, page, transport as pollen_transport +def rows(store, kind: str) -> int: + """How many rows the store holds for this kind.""" + return store.db.execute("SELECT COUNT(*) FROM snapshots WHERE kind=?", (kind,)).fetchone()[0] + + def test_a_repeat_only_moves_last_seen_and_a_change_adds_a_row(tmp_path): - store = Snapshots.open(tmp_path / "s.sqlite") + store = Snapshots(tmp_path / "s.sqlite") assert store.save("pollen", {"level": "Low"}, now=100) is True assert store.save("pollen", {"level": "Low"}, now=200) is False - assert store.latest("pollen") == (200, {"level": "Low"}) and store.count("pollen") == 1 + assert store.latest("pollen") == (200, {"level": "Low"}) and rows(store, "pollen") == 1 assert store.save("pollen", {"level": "High"}, now=300) is True - assert store.count("pollen") == 2 and store.latest("pollen") == (300, {"level": "High"}) and store.latest("forecast") is None + assert rows(store, "pollen") == 2 and store.latest("pollen") == (300, {"level": "High"}) and store.latest("forecast") is None store.put("k", "v") assert store.get("k") == "v" and store.get("missing") is None store.save("pollen", {"level": "Low"}, now=300 + KEEP_SECONDS + 1) # rows not seen for over a year are dropped - assert store.count("pollen") == 1 + assert rows(store, "pollen") == 1 store.close() -def test_the_connection_is_shared_and_closed_by_the_last_user(tmp_path): - a, b = Snapshots.open(tmp_path / "s.sqlite"), Snapshots.open(tmp_path / "s.sqlite") - assert a is b +def test_each_source_has_its_own_connection_to_the_one_file(tmp_path): + a, b = Snapshots(tmp_path / "s.sqlite"), Snapshots(tmp_path / "s.sqlite") + a.save("pollen", {"level": "Low"}, now=100) + b.save("forecast", {"days": []}, now=100) + assert b.latest("pollen") == (100, {"level": "Low"}) and a.latest("forecast") == (100, {"days": []}) # one file a.close() - b.put("k", "v") # still open for the other user + b.put("k", "v") # the other's is still open + assert b.get("k") == "v" b.close() - assert Snapshots.open(tmp_path / "s.sqlite") is not a async def test_a_restart_in_the_evening_starts_from_the_saved_pollen_page_without_fetching(tmp_path): @@ -58,11 +65,11 @@ async def test_the_pollen_history_keeps_one_row_per_change(tmp_path): pollen, _ = make_pollen(tmp_path, lambda: page(grass=level["now"]).replace('', f'')) await pollen.start() await pollen.warm(True) - assert pollen.store.count("pollen") == 1 + assert rows(pollen.store, "pollen") == 1 level["now"] = "High" pollen._validators = {} # the site has a new page await pollen.warm(True) - assert pollen.store.count("pollen") == 2 + assert rows(pollen.store, "pollen") == 2 await pollen.close() @@ -77,7 +84,7 @@ async def test_the_forecast_survives_a_restart_and_a_night_start_needs_no_reques again = Forecast(config(tmp_path, forecast=True, forecast_lat=-37.8, forecast_lon=144.9), None, forecast_transport(calls)) again.now = lambda: datetime.combine(datetime.now(TZ).date(), dtime(22, 0)) await again.start() - assert calls == [] and again.lines(3) == lines and again.source == "Open-Meteo" and again.store.count("forecast") == 1 + assert calls == [] and again.lines(3) == lines and again.source == "Open-Meteo" and rows(again.store, "forecast") == 1 await again.close() @@ -103,7 +110,7 @@ async def test_the_report_always_uses_the_cache_for_pollen_and_forecast(tmp_path pollen = Pollen(config(tmp_path, pollen=True), pollen_transport(lambda: page(), calls)) forecast = Forecast(config(tmp_path, forecast=True, forecast_lat=-37.8, forecast_lon=144.9), None, forecast_transport(calls)) assert not pollen.wants("report") and not forecast.wants("sitrep") and pollen.wants("pollen today") and forecast.wants("forecast") - for tool, args in intent.report_calls(False, False, True, True): + for tool, args in intent.report_calls(["Pollen", "Forecast"]): out = json.loads(await {"pollen_asthma": pollen.handle, "weather_forecast": forecast.handle}[tool](args)) assert "error" in out # nothing cached yet: the report goes without it, no request made assert calls == [] @@ -113,7 +120,7 @@ async def test_the_report_always_uses_the_cache_for_pollen_and_forecast(tmp_path async def test_a_forecast_saved_by_the_older_bom_version_still_loads(tmp_path): from lib.snapshots import Snapshots - store = Snapshots.open(tmp_path / "conditions.sqlite") + store = Snapshots(tmp_path / "conditions.sqlite") today = datetime.now(TZ).date() store.save("forecast", {"source": "Bureau of Meteorology, Testville", "tag": "BOM", "days": [{"date": str(today), "min_c": 11, "max_c": 17, "summary": "Rain.", "rain_chance_pct": 90}]}) diff --git a/tests/test_specs.py b/tests/test_specs.py index 7f3a4b9..495c6d7 100644 --- a/tests/test_specs.py +++ b/tests/test_specs.py @@ -70,14 +70,15 @@ def test_a_period_shows_the_year_on_both_dates_when_it_spans_years(): def test_a_reading_that_cannot_be_negative_is_not_drawn_below_zero_and_temperature_may_be(): - import matplotlib + from matplotlib.figure import Figure from lib import charts - ax = matplotlib.pyplot.figure().add_axes([0, 0, 1, 1]) - charts._pad_limits(ax, 0.0, 8.0, floor=0) + ax = Figure().add_axes([0, 0, 1, 1]) + pad = lambda lo, hi, floor: charts._pad_limits(ax, lo, hi, top=0.12, bottom=0.12, floor=floor) + pad(0.0, 8.0, 0) assert ax.get_ylim()[0] == 0 - charts._pad_limits(ax, 0.0, 8.0) + pad(0.0, 8.0, None) assert ax.get_ylim()[0] < 0 - charts._pad_limits(ax, -4.0, 8.0, floor=0) # real negatives keep their room + pad(-4.0, 8.0, 0) # real negatives keep their room assert ax.get_ylim()[0] < -4 @@ -212,14 +213,30 @@ def test_a_long_list_of_readings_in_a_stack_headline_is_cut_to_fit_the_chart(): def test_a_single_panel_charts_caption_is_one_line_and_a_stacks_is_two(): today = date(2026, 10, 4) - one = lambda sub: Chart("Temperature", sub, [Panel("Temperature", "°C", [line("T")])]) - assert one("Tue 7 Jul – Sun 4 Oct 2026 · daily averages").caption(today) == "Temperature, since Tue 7 Jul 2026" - assert one("Wed 28 Sep – Sun 4 Oct 2026").caption(today) == "Temperature, since Wed 28 Sep 2026" - assert one("Tue 29 Sep 2025 – Sun 4 Oct 2026").caption(today) == "Temperature, since Tue 29 Sep 2025" - assert one("Wed 24 – Tue 30 Sep 2026 · 30-minute readings").caption(today) == "Temperature, Wed 24 – Tue 30 Sep 2026" # not up to today - assert one("Sun 4 Oct 2026").caption(today) == "Temperature, today" and one("Tue 29 Sep 2026").caption(today) == "Temperature, Tue 29 Sep 2026" - assert one("").caption(today) == "Temperature" + one = lambda *period: Chart("Temperature", "whatever the subtitle says", [Panel("Temperature", "°C", [line("T")])], period=period or None) + assert one(date(2026, 7, 7), today).caption(today) == "Temperature, since Tue 7 Jul 2026" + assert one(date(2026, 9, 28), today).caption(today) == "Temperature, since Mon 28 Sep 2026" + assert one(date(2025, 9, 29), today).caption(today) == "Temperature, since Mon 29 Sep 2025" + assert one(date(2026, 9, 24), date(2026, 9, 30)).caption(today) == "Temperature, Thu 24 – Wed 30 Sep 2026" # not up to today + assert one(today, today).caption(today) == "Temperature, today" and one(date(2026, 9, 29), date(2026, 9, 29)).caption(today) == "Temperature, Tue 29 Sep 2026" + assert one().caption(today) == "Temperature" # no period: the title rain_behind = stack([Panel("Humidity", "%", [line("H")]), Panel("Rain", "mm", bars=bars())], date(2026, 9, 28), today) assert rain_behind.caption(today) == "Humidity, Rain, since Mon 28 Sep 2026" # rain behind a line is still one panel stacked = stack([Panel("Humidity", "%", [line("H")]), Panel("Pressure", "hPa", [line("P")])], date(2026, 9, 28), today) assert stacked.caption(today) == "Humidity, Pressure\nMon 28 Sep – Sun 4 Oct 2026" # several panels keep two lines + + +def test_a_panel_of_bars_with_no_bar_is_refused_when_built(): + import pytest + from lib.specs import Bars, Panel + with pytest.raises(ValueError, match="nothing to draw"): + Panel("Rain", "mm", bars=Bars("Rain", "mm", [], [], 86400)) + + +def test_a_bar_below_zero_has_its_bright_end_at_the_bottom(monkeypatch): + from lib import charts + drawn = [] + monkeypatch.setattr(charts, "_columns", lambda ax, x, heights, width, bottom=0.0, **style: drawn.append((list(heights), bottom))) + charts._draw_bars(None, [0.0, 1.0], [-3.0, 2.0], 0.85, 10.0, 3) + heights, bottom = drawn[1] # the caps (the bodies are drawn first) + assert heights == [-0.14, 0.14] and [round(b, 2) for b in bottom] == [-2.86, 1.86] # each cap ends where its bar does diff --git a/tests/test_startup.py b/tests/test_startup.py index b66ad67..6e72f39 100644 --- a/tests/test_startup.py +++ b/tests/test_startup.py @@ -6,6 +6,8 @@ import signal from types import SimpleNamespace as NS +import pytest + import ecobot from lib.ecowitt import archive from datetime import datetime, timedelta, timezone @@ -152,6 +154,54 @@ async def stop_when_archived(): assert "Startup warm-up failed" not in log # not lost to the catch-all +async def test_a_source_that_cannot_even_be_made_is_left_out_and_the_others_start(tmp_path, monkeypatch, caplog): + eco_t, _ = ecowitt_transport(history_days=45) + monkeypatch.setattr(ecobot.Ecowitt, "__init__", lambda self, c, orig=ecobot.Ecowitt.__init__: orig(self, c, eco_t)) + + def broken(self, cfg, transport=None): + raise OSError("the cache folder is read-only") + monkeypatch.setattr(ecobot.AirGradient, "__init__", broken) + sources, _ = await ecobot.start_sources(config(tmp_path)) + for s in sources: + await s.close() + assert [s.name for s in sources] == ["Ecowitt"] and "AirGradient couldn't start - continuing without it" in caplog.text + + +async def test_a_failure_part_way_through_starting_closes_everything_already_open(tmp_path, monkeypatch, caplog): + """Telegram fails once the sources, the model's client and the connection are up: each is closed, the last opened first, + and a close that fails doesn't stop the rest. The bot stops with the error that stopped it.""" + cfg = config(tmp_path) + eco_t, _ = ecowitt_transport(history_days=45) + air_t, _ = air_transport(oldest=datetime.now(timezone.utc) - timedelta(days=5)) + app, closed = FakeApp(), [] + app.stop = lambda: closed.append("telegram") or app._noop() + monkeypatch.setattr(ecobot.Config, "from_env", classmethod(lambda cls: cfg)) + for cls, transport in ((ecobot.Ecowitt, eco_t), (ecobot.AirGradient, air_t)): + monkeypatch.setattr(cls, "__init__", lambda self, c, transport=None, orig=cls.__init__, t=transport: orig(self, c, t)) + monkeypatch.setattr(ecobot.Application, "builder", staticmethod(lambda: FakeBuilder(app))) + + async def stuck_disk(self): + closed.append("Ecowitt") + raise OSError("disk I/O error") + monkeypatch.setattr(ecobot.Ecowitt, "close", stuck_disk) + monkeypatch.setattr(ecobot.AirGradient, "close", lambda self, orig=ecobot.AirGradient.close: closed.append("AirGradient") or orig(self)) + + class Client: + def __init__(self, **settings): + pass + + async def close(self): + closed.append("model") + monkeypatch.setattr(ecobot, "AsyncOpenAI", Client) + + async def down(self, bot): + raise RuntimeError("Telegram is down") + monkeypatch.setattr(ecobot.Bot, "refresh_keyboards", down) + with pytest.raises(RuntimeError, match="Telegram is down"): + await ecobot.main() + assert closed == ["telegram", "model", "AirGradient", "Ecowitt"] and "disk I/O error" in caplog.text + + async def test_pollen_and_forecast_start_only_when_switched_on_and_the_forecast_uses_the_stations_location(tmp_path, monkeypatch): from tests.test_forecast import transport as forecast_transport from tests.test_pollen import page, transport as pollen_transport @@ -162,17 +212,52 @@ async def test_pollen_and_forecast_start_only_when_switched_on_and_the_forecast_ monkeypatch.setattr(ecobot.Pollen, "__init__", lambda self, c: pollen_init(self, c, pollen_transport(page, calls))) monkeypatch.setattr(ecobot.Forecast, "__init__", lambda self, c, place=None: forecast_init(self, c, place, forecast_transport(calls))) monkeypatch.setattr(ecobot.Pollen, "now", lambda self: datetime(2026, 11, 10, 12, 0)) # in the pollen season, whatever day the test runs - off = await ecobot.start_sources(config(tmp_path, airgradient_token="", airgradient_location="")) + off, _ = await ecobot.start_sources(config(tmp_path, airgradient_token="", airgradient_location="")) assert [s.name for s in off] == ["Ecowitt"] for s in off: await s.close() - on = await ecobot.start_sources(config(tmp_path, airgradient_token="", airgradient_location="", pollen=True, forecast=True)) + on, _ = await ecobot.start_sources(config(tmp_path, airgradient_token="", airgradient_location="", pollen=True, forecast=True)) assert [s.name for s in on] == ["Ecowitt", "Pollen", "Forecast"] assert on[2].location == (-37.8, 145.0) and on[2].lines() and on[1].lines() for s in on: await s.close() +async def test_the_other_sources_run_in_the_weather_station_s_time_zone(tmp_path, monkeypatch, caplog): + from zoneinfo import ZoneInfo + eco_t, _ = ecowitt_transport(history_days=45) + air_t, _ = air_transport(oldest=datetime.now(timezone.utc) - timedelta(days=5)) + for cls, transport in ((ecobot.Ecowitt, eco_t), (ecobot.AirGradient, air_t)): + monkeypatch.setattr(cls, "__init__", lambda self, c, transport=None, orig=cls.__init__, t=transport: orig(self, c, t)) + monkeypatch.setenv("TZ", "UTC") + sources, cfg = await ecobot.start_sources(config(tmp_path, tz=ZoneInfo("UTC"))) # TZ says UTC; the station is in Melbourne + for s in sources: + await s.close() + assert [s.name for s in sources] == ["Ecowitt", "AirGradient"] and str(cfg.tz) == "Australia/Melbourne" + assert all(str(s.tz) == "Australia/Melbourne" for s in sources) + assert "TZ=UTC is not the weather station's time zone; using the station's, Australia/Melbourne" in caplog.text + assert "has no name" not in caplog.text + sources, cfg = await ecobot.start_sources(config(tmp_path, ecowitt_api_key="", tz=timezone(timedelta(hours=10)))) + for s in sources: + await s.close() + assert [s.name for s in sources] == ["AirGradient"] and cfg.tz == timezone(timedelta(hours=10)) # no station: the configured zone + assert "The time zone has no name, only an offset (UTC+10:00)" in caplog.text + + +def test_the_time_zone_without_a_station_is_tz_else_the_one_etc_localtime_links_to(monkeypatch, caplog): + from lib import config as config_module + from lib.config import local_tz + monkeypatch.setenv("TZ", "Australia/Perth") + assert str(local_tz()) == "Australia/Perth" + monkeypatch.setattr(config_module.os.path, "realpath", lambda p: "/usr/share/zoneinfo/Australia/Melbourne") + monkeypatch.setenv("TZ", "Mars/Olympus") + assert str(local_tz()) == "Australia/Melbourne" and "TZ: 'Mars/Olympus' is not a time zone name" in caplog.text + monkeypatch.delenv("TZ") + assert str(local_tz()) == "Australia/Melbourne" + monkeypatch.setattr(config_module.os.path, "realpath", lambda p: "/etc/localtime") # a copied file, not a link: no name + assert not hasattr(local_tz(), "key") + + def test_the_alert_cooldown_is_configurable_with_a_default_and_limits(monkeypatch): from lib.config import Config monkeypatch.setenv("TELEGRAM_BOT_TOKEN", "t") @@ -181,16 +266,45 @@ def test_the_alert_cooldown_is_configurable_with_a_default_and_limits(monkeypatc monkeypatch.setenv("ECOWITT_APP_KEY", "b") monkeypatch.delenv("ALERT_COOLDOWN_MINUTES", raising=False) assert Config.from_env().alert_cooldown_minutes == 30 - for value, want in (("45", 45), ("1", 5), ("999", 150), ("junk", 30)): + for value, want in (("45", 45), ("1", 5), ("0", 5), ("-3", 5), ("999", 150), ("junk", 30)): # 0 is under 5 like 1 is monkeypatch.setenv("ALERT_COOLDOWN_MINUTES", value) - assert Config.from_env().alert_cooldown_minutes == want + assert Config.from_env().alert_cooldown_minutes == want, value +def test_the_log_level_is_a_setting_and_a_typo_keeps_info_with_a_warning(monkeypatch, caplog): + from lib.config import Config + for k, v in (("TELEGRAM_BOT_TOKEN", "t"), ("XAI_API_KEY", "k"), ("ECOWITT_API_KEY", "a"), ("ECOWITT_APP_KEY", "b")): + monkeypatch.setenv(k, v) + monkeypatch.delenv("LOG_LEVEL", raising=False) + assert Config.from_env().log_level == "INFO" + for value, want in (("debug", "DEBUG"), ("WARNING", "WARNING"), ("verbose", "INFO")): + monkeypatch.setenv("LOG_LEVEL", value) + assert Config.from_env().log_level == want, value + assert "LOG_LEVEL: 'verbose' is not DEBUG, INFO, WARNING or ERROR; keeping INFO" in caplog.text + + + +def test_a_blank_setting_means_its_default_and_a_missing_key_says_which(monkeypatch, caplog): + from lib.config import Config + for k, v in (("TELEGRAM_BOT_TOKEN", " t "), ("XAI_API_KEY", "k"), ("ECOWITT_API_KEY", "a"), ("ECOWITT_APP_KEY", "b")): + monkeypatch.setenv(k, v) + for k in ("XAI_MODEL", "XAI_BASE_URL", "PLACE", "POLLEN_DISTRICT", "TUYA_BASE_URL"): + monkeypatch.setenv(k, " ") + monkeypatch.setenv("POLLEN", "enabled") + cfg = Config.from_env() + assert (cfg.telegram_token, cfg.xai_model, cfg.xai_base_url, cfg.place, cfg.pollen_district, cfg.tuya_base_url) == ( + "t", "grok-4.3", "https://api.x.ai/v1", "Melbourne", "Central", "https://openapi.tuyaeu.com") + assert cfg.pollen is False and "POLLEN: 'enabled' is not on or off; keeping the default (off)" in caplog.text + monkeypatch.setenv("XAI_API_KEY", " ") + with pytest.raises(SystemExit, match="XAI_API_KEY is not set"): + Config.from_env() + def test_the_rain_quiet_hours_are_configurable(): from lib.config import _hours - assert _hours("", (0, 6)) == (0, 6) and _hours("22-6", (0, 6)) == (22, 6) and _hours("off", (0, 6)) is None - assert _hours("junk", (0, 6)) == (0, 6) and _hours("6-6", (0, 6)) == (0, 6) and _hours("0-25", (0, 6)) == (0, 6) + hours = lambda text: _hours(text, (0, 6), "RAIN_QUIET_HOURS") + assert hours("") == (0, 6) and hours("22-6") == (22, 6) and hours("off") is None + assert hours("junk") == (0, 6) and hours("6-6") == (0, 6) and hours("0-25") == (0, 6) def test_feels_like_in_chart_all_is_off_by_default_and_a_config_option_turns_it_on(monkeypatch): @@ -281,6 +395,21 @@ def line(msg, *args, quotes=True): assert line(timings, quotes=False) == "INFO " + timings +def test_a_traceback_is_logged_whole_after_its_cut_message(): + import sys + from lib.logs import LogFormat + fmt = LogFormat("%(levelname)s %(message)s", width=40) + try: + {}["a key whose name is far longer than the forty characters of the log width"] + except KeyError: + record = logging.LogRecord("lib.bot", logging.ERROR, __file__, 1, "Failed to deliver reply to chat 12345678 in the group", None, + sys.exc_info()) + lines = fmt.format(record).split("\n") + assert lines[0] == "ERROR Failed to deliver reply to chat 1…" and lines[1] == "Traceback (most recent call last):" + assert lines[-1] == "KeyError: 'a key whose name is far longer than the forty characters of the log width'" + assert any(len(line) > 40 for line in lines[2:-1]) # the frames too, whole + + def test_the_log_width_is_71_unless_set_and_0_cuts_none(monkeypatch): from lib.config import Config for k, v in (("TELEGRAM_BOT_TOKEN", "t"), ("XAI_API_KEY", "k"), ("ECOWITT_API_KEY", "a"), ("ECOWITT_APP_KEY", "b")): diff --git a/tests/test_store.py b/tests/test_store.py index ce5053f..d9000a7 100644 --- a/tests/test_store.py +++ b/tests/test_store.py @@ -3,6 +3,11 @@ from lib.ecowitt import store + +def missing(cache, mac: str, cycle: str, grp: str, start: int, end: int): + """One group's gaps in one window (HistoryCache.uncovered, for a single one).""" + return cache.uncovered(mac, [grp], [(cycle, start, end)])[0][grp] + def test_subtract_and_merge(): assert store.subtract((0, 100), [(10, 20), (30, 40)]) == [(0, 9), (21, 29), (41, 100)] assert store.subtract((0, 10), [(0, 10)]) == [] @@ -23,27 +28,29 @@ def test_only_settled_readings_are_kept(tmp_path): cache.store("M", "5min", ["outdoor"], data(old, old + 300, fresh), old - 300, now) loaded = cache.load("M", "5min", ["outdoor"], 0, now) assert sorted(loaded["outdoor"]["temperature"]["list"]) == [str(old), str(old + 300)] - assert cache.missing("M", "5min", "outdoor", old, old + 300) == [] - assert cache.missing("M", "5min", "outdoor", fresh, now) # the unsettled tail is still to fetch + assert missing(cache, "M", "5min", "outdoor", old, old + 300) == [] + assert missing(cache, "M", "5min", "outdoor", fresh, now) # the unsettled tail is still to fetch -def test_the_coverage_of_several_groups_is_each_groups_own_read_at_once(tmp_path): +def test_what_is_not_held_is_read_for_many_windows_resolutions_and_groups_at_once(tmp_path): cache, now = make(tmp_path), int(time.time()) old = now - 9 * 86400 both = {"outdoor": data(old)["outdoor"], "wind": {"wind_speed": {"unit": "km/h", "list": {str(old): "3"}}}} cache.store("M", "30min", ["outdoor", "wind"], both, old, old + 86400) cache.store("M", "30min", ["outdoor"], data(old + 3 * 86400), old + 3 * 86400, old + 4 * 86400) - cache.store("M", "5min", ["wind"], both, old, old + 3600) # another cycle: not counted - held = cache.coverage_of("M", "30min", ["outdoor", "wind", "rainfall"]) - assert held == {g: cache.coverage("M", "30min", g) for g in ("outdoor", "wind", "rainfall")} - assert len(held["outdoor"]) == 2 and len(held["wind"]) == 1 and held["rainfall"] == [] + cache.store("M", "5min", ["wind"], both, old, old + 3600) + groups, windows = ["outdoor", "wind", "rainfall"], [("30min", old, old + 4 * 86400), ("5min", old, old + 3600)] + thirty, five = cache.uncovered("M", groups, windows) + assert thirty == {g: missing(cache, "M", "30min", g, old, old + 4 * 86400) for g in groups} # as each group's own read + assert thirty["outdoor"] == [(old + 86401, old + 3 * 86400 - 1)] and thirty["rainfall"] == [(old, old + 4 * 86400)] + assert five["wind"] == [] and five["outdoor"] == [(old, old + 3600)] # each window its own resolution def test_empty_old_ranges_are_remembered(tmp_path): cache, now = make(tmp_path), int(time.time()) start, end = now - 30 * 86400, now - 29 * 86400 cache.store("M", "30min", ["outdoor"], {}, start, end) - assert cache.missing("M", "30min", "outdoor", start, end) == [] + assert missing(cache, "M", "30min", "outdoor", start, end) == [] def test_changed_units_clear_the_cache(tmp_path): @@ -66,12 +73,12 @@ def test_a_metric_ecowitt_adds_later_makes_the_days_held_be_fetched_again(tmp_pa cache, now = make(tmp_path), int(time.time()) old = now - 10 * 86400 cache.store("M", "30min", ["outdoor"], data(old), old, old + 3600) - assert cache.missing("M", "30min", "outdoor", old, old + 3600) == [] # held + assert missing(cache, "M", "30min", "outdoor", old, old + 3600) == [] # held cache.store("M", "30min", ["outdoor"], data(old + 7200, field="temperature") | {"outdoor": { **data(old + 7200)["outdoor"], "vpd": {"unit": "kPa", "list": {str(old + 7200): "0.8"}}}}, old + 7200, old + 7300) - assert cache.missing("M", "30min", "outdoor", old, old + 3600) # the new metric: the earlier days are asked for again + assert missing(cache, "M", "30min", "outdoor", old, old + 3600) # the new metric: the earlier days are asked for again cache.store("M", "30min", ["outdoor"], data(old), old, old + 3600) - assert cache.missing("M", "30min", "outdoor", old, old + 3600) == [] # and once the vpd-less response repeats, no loop + assert missing(cache, "M", "30min", "outdoor", old, old + 3600) == [] # and once the vpd-less response repeats, no loop def test_vpd_reported_in_inhg_is_stored_and_read_in_kpa_and_old_rows_are_converted(tmp_path): @@ -111,4 +118,28 @@ def test_the_cache_says_where_it_is_and_how_much_it_holds(tmp_path): assert cache.path == tmp_path / "c.sqlite" assert cache.count("M", "5min", "outdoor", old, old + 300) == 2 and cache.count("M", "5min", "outdoor", old + 1000, now) == 0 assert cache.count("M", "30min", "outdoor", 0, now) == 0 - assert cache.newest("M", "5min") == old + 600 and cache.newest("M", "30min") is None + + +def test_a_write_that_fails_part_way_leaves_nothing_behind(tmp_path, monkeypatch): + """A response is saved in one transaction: if the coverage update fails after the readings went in, they are rolled back, + not committed half-written by the next save.""" + import pytest + cache = make(tmp_path) + old = int(time.time()) - 10 * 86400 + real_merge = store.merge + monkeypatch.setattr(store, "merge", lambda spans: (_ for _ in ()).throw(RuntimeError("disk trouble"))) + with pytest.raises(RuntimeError): + cache.store("M", "5min", ["outdoor"], data(old, old + 300), old, old + 300) + monkeypatch.setattr(store, "merge", real_merge) + cache.store("M", "5min", ["outdoor"], data(old + 3600), old + 3600, old + 3600) # a later save commits only its own + assert cache.count("M", "5min", "outdoor", old, old + 300) == 0 and cache.count("M", "5min", "outdoor", old + 3600, old + 3600) == 1 + + +def test_each_ecowitt_resolution_is_kept_asked_for_and_settled_as_its_row_says(): + from datetime import date, timedelta + from lib.ecowitt.api import CYCLES, kept_from + today, now = date(2026, 10, 3), 1_790_000_000 + # Ecowitt keeps 5-minute readings 90 days, 30-minute 365, 4-hour 730, daily 1460; the bot asks two days clear of each edge + assert [(today - kept_from(c, today)).days for c in CYCLES] == [88, 363, 728, 1458] + assert [now - store.horizon(c, now) for c in CYCLES] == [300 + 3600, 1800 + 7200, 14400 + 8 * 3600, 86400 + 86400] # final after + assert [c.span for c in CYCLES.values()] == [timedelta(days=d) for d in (1, 7, 28, 365)] # per request diff --git a/tests/test_templates.py b/tests/test_templates.py index fcc276e..6b46e6f 100644 --- a/tests/test_templates.py +++ b/tests/test_templates.py @@ -3,7 +3,7 @@ from lib import intent, templates from lib.bot import Bot -from tests.fakes import TZ +from tests.fakes import EVERY_SOURCE, TZ NOW = datetime(2026, 10, 1, 12, 0, tzinfo=TZ) @@ -13,15 +13,16 @@ def test_a_button_label_is_its_sentence_without_the_emoji_and_typed_text_is_left assert templates.sentence("\U0001f4c8 Temperature") == "Temperature chart 1d" assert templates.sentence(templates.CAPABILITIES) == "What can you do?" assert templates.sentence("report") is None and templates.sentence("Report") is None - assert len(templates.LABELS) == 9 + assert len(templates.SENTENCES) == 9 def test_every_button_asks_something_the_bot_understands(): def read(label): - return intent.read(templates.sentence(label), NOW, True, True) + return intent.read(templates.sentence(label), NOW) assert read("\U0001f4cb Status").report weather = read("\U0001f326️ Weather") - assert weather.chart_all and weather.chart_in_code and weather.chart_fields[0] == "temperature" and weather.chart_fields[-1] == "wind" + assert intent.wants_chart_all(templates.sentence("\U0001f326️ Weather")) and weather.chart_in_code + assert weather.chart_fields[0] == "temperature" and weather.chart_fields[-1] == "wind" for label, groups in (("\U0001f327️ Rain", "rainfall"), ("\U0001f4c8 Temperature", "outdoor,indoor"), ("\U0001f4a7 Humidity", "outdoor,indoor"), ("\U0001f4a8 Wind", "wind")): name, args, _ = read(label).fast @@ -37,13 +38,13 @@ def read(label): def test_the_keyboard_is_persistent_and_has_every_button(): kb = templates.keyboard() assert kb.is_persistent and kb.resize_keyboard - assert {b.text for row in kb.keyboard for b in row} == templates.LABELS + assert {b.text for row in kb.keyboard for b in row} == set(templates.SENTENCES) assert [len(row) for row in kb.keyboard] == [3, 3, 3] class Recorder(Bot): - def __init__(self): - super().__init__(NS(tz=TZ), None, [], None) + def __init__(self, state): + super().__init__(NS(tz=TZ), None, [], state) self.asked, self.alerts = [], 0 async def respond(self, update, context, text): @@ -63,15 +64,15 @@ async def reply_text(body, **kw): return NS(effective_message=msg, effective_chat=msg.chat, effective_user=NS(id=7, full_name="Rob", username="rob")), sent -async def test_tapping_a_button_asks_its_sentence_and_capabilities_and_alerts_prints_both(): - bot = Recorder() +async def test_tapping_a_button_asks_its_sentence_and_capabilities_and_alerts_prints_both(alert_state): + bot = Recorder(alert_state) for text in ("\U0001f4cb Status", "weather now", templates.CAPABILITIES): await bot.on_message(message(text)[0], NS(bot=NS(username="b", id=99))) assert bot.asked == ["Report", "weather now"] and bot.alerts == 1 # capabilities are not sent to the model -async def test_start_carries_the_keyboard_in_private_chats_only_and_keyboard_off_removes_it(): - bot = Recorder() +async def test_start_carries_the_keyboard_in_private_chats_only_and_keyboard_off_removes_it(alert_state): + bot = Recorder(alert_state) update, sent = message("/start") await bot.on_start(update, NS(args=[])) assert sent[0][1]["reply_markup"].is_persistent @@ -116,17 +117,17 @@ async def test_deliver_puts_the_keyboard_on_the_last_text_or_the_single_photo(tm from lib.bot import deliver markup = templates.keyboard() msg = Replies() - assert await deliver(msg, "hello", [], markup=markup) is True and msg.sent[-1][1]["reply_markup"] is markup + assert (await deliver(msg, "hello", [], markup=markup)).carried is True and msg.sent[-1][1]["reply_markup"] is markup msg = Replies() - assert await deliver(msg, "", [b"p"], markup=markup, titles=["T"]) is True and msg.sent == [("photo", msg.sent[0][1])] + assert (await deliver(msg, "", [b"p"], markup=markup, titles=["T"])).carried is True and msg.sent == [("photo", msg.sent[0][1])] assert msg.sent[0][1]["reply_markup"] is markup and msg.sent[0][1]["caption"] == "T" msg = Replies() - assert await deliver(msg, "answer", [b"p"], markup=markup, titles=["T"]) is True # text first, the photo carries it + assert (await deliver(msg, "answer", [b"p"], markup=markup, titles=["T"])).carried is True # text first, the photo carries it assert [k for k, _ in msg.sent] == ["text", "photo"] and msg.sent[1][1]["reply_markup"] is markup and "reply_markup" not in msg.sent[0][1] msg = Replies() - assert await deliver(msg, "", [b"p", b"q"], markup=markup, titles=["T", "U"]) is False # a group of photos can't carry it + assert (await deliver(msg, "", [b"p", b"q"], markup=markup, titles=["T", "U"])).carried is False # a group of photos can't carry it msg = Replies() - assert await deliver(msg, "hello", []) is False and "reply_markup" not in msg.sent[0][1] + assert (await deliver(msg, "hello", [])).carried is False and "reply_markup" not in msg.sent[0][1] async def test_a_chat_with_an_old_or_no_keyboard_gets_the_current_one_with_its_next_reply_once(tmp_path): @@ -138,12 +139,11 @@ async def test_a_chat_with_an_old_or_no_keyboard_gets_the_current_one_with_its_n assert bot._keyboard_stale(msg) # never had it state.set_keyboard(1, "old1234") assert bot._keyboard_stale(msg) # the buttons changed since - state.set_keyboard(1, templates.VERSION) + state.set_keyboard(1, templates.version()) assert not bot._keyboard_stale(msg) # up to date: nothing is added state.set_keyboard(1, templates.HIDDEN) assert not bot._keyboard_stale(msg) # they hid it: leave it hidden assert not bot._keyboard_stale(Replies("supergroup")) - assert not Bot(NS(tz=TZ), None, [], None)._keyboard_stale(msg) async def test_start_and_keyboard_commands_record_which_keyboard_the_chat_has(tmp_path): @@ -156,11 +156,11 @@ async def test_start_and_keyboard_commands_record_which_keyboard_the_chat_has(tm assert state.keyboard(1) == templates.HIDDEN update, _ = message("/keyboard") await bot.on_keyboard(update, NS(args=[])) - assert state.keyboard(1) == templates.VERSION + assert state.keyboard(1) == templates.version() state.set_keyboard(1, "old1234") update, _ = message("/start") await bot.on_start(update, NS(args=[])) - assert state.keyboard(1) == templates.VERSION + assert state.keyboard(1) == templates.version() async def test_a_reply_in_a_private_chat_carries_the_new_keyboard_once(tmp_path): @@ -181,7 +181,7 @@ async def ask(): return sent first, second = await ask(), await ask() assert first[0][1].get("reply_markup") is not None and second[0][1].get("reply_markup") is None - assert state.keyboard(1) == templates.VERSION + assert state.keyboard(1) == templates.version() async def test_at_startup_private_chats_with_old_buttons_are_told_once_and_others_left_alone(tmp_path): @@ -191,29 +191,30 @@ async def test_at_startup_private_chats_with_old_buttons_are_told_once_and_other for chat_id in (10, 11, 12, 13, -20): # -20 is a group state.add_chat(chat_id, str(chat_id)) state.set_keyboard(10, "old1234") - state.set_keyboard(11, templates.VERSION) + state.set_keyboard(11, templates.version()) state.set_keyboard(12, templates.HIDDEN) # 13 and the group have no record - sent = [] + sent, tried = [], [] class TG: async def send_message(self, chat_id, text, **kw): + tried.append(chat_id) if chat_id == 13: raise Forbidden("bot was blocked by the user") sent.append((chat_id, text, kw["reply_markup"].is_persistent)) bot = Bot(NS(tz=TZ), None, [], state) assert await bot.refresh_keyboards(TG()) == 1 assert sent == [(10, "Buttons updated.", True)] - assert state.keyboard(10) == templates.VERSION and 13 not in state.chats and state.keyboard(-20) is None - assert await bot.refresh_keyboards(TG()) == 0 # nothing more on the next start - assert await Bot(NS(tz=TZ), None, [], None).refresh_keyboards(TG()) == 0 + assert state.keyboard(10) == templates.version() and state.chats[13]["gone"] and state.keyboard(-20) is None + tried.clear() + assert await bot.refresh_keyboards(TG()) == 0 and tried == [] # nothing more on the next start: 13 blocked the bot def test_the_capabilities_list_is_the_users_bullets(): assert templates.capabilities_text() == "\n".join([ "\u2022 Temperature, humidity (indoor, outdoor)", "\u2022 Dew point, vapour pressure deficit, pressure, wind speed", "\u2022 Rain, solar radiation, UV index", "\u2022 PM1, PM2.5, PM10, CO₂, VOC, NOx", "\u2022 History charts"]) - assert len(templates.capabilities_text(air=False).splitlines()) == 4 and len(templates.capabilities_text(weather=False).splitlines()) == 2 - full = templates.capabilities_text(pollen=True, forecast=True).splitlines() + assert len(templates.capabilities_text(["Ecowitt"]).splitlines()) == 4 and len(templates.capabilities_text(["AirGradient"]).splitlines()) == 2 + full = templates.capabilities_text(EVERY_SOURCE).splitlines() assert len(full) == 7 and full[4:] == ["\u2022 Pollen and thunderstorm asthma risk (October to December)", "\u2022 Forecast", "\u2022 History charts"] @@ -223,5 +224,5 @@ def test_the_capabilities_list_is_the_users_bullets(): def test_the_capabilities_list_adds_pollen_and_forecast_only_when_they_are_on(): plain = templates.capabilities_text() assert "Pollen" not in plain and "Forecast" not in plain - full = templates.capabilities_text(pollen=True, forecast=True).splitlines() + full = templates.capabilities_text(EVERY_SOURCE).splitlines() assert full[-3:] == ["• Pollen and thunderstorm asthma risk (October to December)", "• Forecast", "• History charts"] diff --git a/tests/test_timeutil.py b/tests/test_timeutil.py new file mode 100644 index 0000000..9a06fda --- /dev/null +++ b/tests/test_timeutil.py @@ -0,0 +1,62 @@ +"""Local time across daylight saving: Melbourne skips 2am to 3am on 4 Oct 2026 and has 2am to 3am twice on 5 Apr 2026.""" + +from datetime import UTC, date, datetime +from zoneinfo import ZoneInfo + +from lib.timeutil import local_epoch, naive_local, parse_local, seconds_until + +TZ = ZoneInfo("Australia/Melbourne") + + +def test_seconds_until_a_clock_time_are_real_seconds_across_a_change(): + assert seconds_until(18, 0, TZ, datetime(2026, 10, 3, 18, 0)) == 23 * 3600 # the day the clocks go forward + assert seconds_until(1, 30, TZ, datetime(2026, 10, 4, 1, 40)) == (22 * 60 + 50) * 60 # the archive's nightly run + assert seconds_until(18, 0, TZ, datetime(2026, 4, 4, 18, 0)) == 25 * 3600 # the day they go back + assert seconds_until(18, 0, TZ, datetime(2026, 9, 8, 17, 0)) == 3600 # an ordinary day + + +def test_a_naive_local_time_keeps_which_of_the_twice_happening_hours_it_is(): + second = int(datetime(2026, 4, 5, 2, 30, fold=1, tzinfo=TZ).timestamp()) + assert local_epoch(naive_local(second, TZ), TZ) == second + + +def test_a_date_with_an_offset_is_read_as_the_local_time_it_is(): + assert parse_local("2026-10-01T00:00:00Z", TZ) == datetime(2026, 10, 1, 10, 0) # UTC midnight: 10am in Melbourne + assert parse_local("2026-12-01T00:00:00+00:00", TZ) == datetime(2026, 12, 1, 11, 0) # 11am in daylight saving + assert parse_local("2026-10-01 14:30:00", TZ) == datetime(2026, 10, 1, 14, 30) and parse_local(date(2026, 10, 1), TZ) == datetime(2026, 10, 1) + + +def test_ecowitts_daily_records_say_when_they_run_in_local_time(): + from lib.ecowitt.api import daily_run + assert daily_run(TZ, date(2026, 6, 1), date(2026, 8, 31)) == "10am to 10am" + assert daily_run(TZ, date(2026, 12, 1), date(2027, 2, 28)) == "11am to 11am" + assert daily_run(TZ, date(2026, 8, 1), date(2026, 11, 30)) == "10am to 10am (11am to 11am in daylight saving)" + + +async def test_the_last_hours_are_real_hours_the_night_the_clocks_go_forward(tmp_path): + from lib.ecowitt import Ecowitt + from tests.fakes import config + eco = Ecowitt(config(tmp_path)) + asked = [] + + async def readings(cycle, start, end, groups=None): + asked.append((start, end)) + return [] + eco.readings, eco.now = readings, (lambda: datetime(2026, 10, 4, 4, 0)) + await eco.recent(3) + await eco.close() + assert asked == [(datetime(2026, 10, 4, 0, 0), datetime(2026, 10, 4, 4, 0))] # three real hours: 2am to 3am never happened + + +def test_a_day_and_a_time_of_day_are_written_one_way(): + from datetime import date, datetime + from lib.timeutil import clock_text, day_text + assert day_text(date(2026, 10, 3)) == "Sat 3 Oct 2026" and day_text(datetime(2026, 10, 3, 9), year=False) == "Sat 3 Oct" # never "03" + assert [clock_text(datetime(2026, 10, 3, h, m)) for h, m in ((0, 5), (9, 0), (12, 30), (23, 45))] == ["12:05am", "9:00am", "12:30pm", "11:45pm"] + assert clock_text(datetime(2026, 10, 3, 10), whole_hours=True) == "10am" and clock_text(datetime(2026, 10, 3, 10, 5), whole_hours=True) == "10:05am" + + +def test_at_hour_is_that_local_clock_time_on_either_side_of_a_daylight_saving_change(): + from lib.timeutil import at_hour + assert at_hour(date(2026, 4, 5), 12, TZ) == int(datetime(2026, 4, 5, 2, tzinfo=UTC).timestamp()) # standard time, UTC+10 + assert at_hour(date(2026, 4, 4), 12, TZ) == int(datetime(2026, 4, 4, 1, tzinfo=UTC).timestamp()) # daylight time, UTC+11 diff --git a/tests/test_warm.py b/tests/test_warm.py new file mode 100644 index 0000000..d641121 --- /dev/null +++ b/tests/test_warm.py @@ -0,0 +1,68 @@ +import asyncio +import logging + +from lib.warm import Warmer, every, safely + + +async def test_a_warmer_runs_one_refresh_at_a_time(): + """A timed refresh waits for a question's to finish, and a question that arrives during a refresh starts none of its own.""" + release, started, seen = asyncio.Event(), asyncio.Event(), [] + + async def refresh(fresh): + seen.append(("start", fresh)) + started.set() + await release.wait() + seen.append(("end", fresh)) + warmer = Warmer(refresh, interval=0) + warmer.poke() + await started.wait() # a question's refresh is running + timer = asyncio.create_task(warmer.run()) # and the timed one falls due at once + warmer.poke() # another question meanwhile + for _ in range(5): + await asyncio.sleep(0) + assert seen == [("start", False)] + release.set() + for _ in range(10): + await asyncio.sleep(0) + timer.cancel() + assert seen[:3] == [("start", False), ("end", False), ("start", True)] + assert all(a[0] != b[0] for a, b in zip(seen, seen[1:])) # every refresh ends before the next starts + + +async def test_stopping_a_warmer_cancels_the_refresh_a_question_started(): + async def refresh(fresh): + await asyncio.Event().wait() # never ends by itself + warmer = Warmer(refresh) + warmer.poke() + await asyncio.sleep(0) + await warmer.stop() + assert warmer._task.cancelled() + + +async def test_a_background_failure_has_its_traceback_logged_only_when_it_points_at_code(caplog): + async def broken(): + raise TypeError("bad") + + async def down(): + raise ConnectionError("no route to host") + caplog.set_level(logging.WARNING) + assert await safely(broken) is None and await safely(down) is None # neither escapes + bug, outage = [r for r in caplog.records if r.name == "lib.warm"] + assert bug.exc_info and "TypeError: bad" in bug.getMessage() + assert not outage.exc_info and "ConnectionError: no route to host" in outage.getMessage() + + +async def test_every_waits_as_long_as_its_delay_says_each_time(): + runs, asked = [], [] + + async def work(): + runs.append(1) + + def delay(): + asked.append(len(runs)) + return 0 + task = asyncio.create_task(every(delay, work)) + for _ in range(6): + await asyncio.sleep(0) + task.cancel() + assert asked[:3] == [1, 2, 3] # asked again after each run