Cached Sharesight period returns with background backfill, DATA_DIR, logging

CLAUDE.md

@@ -15,7 +15,7 @@ MCP data tools and web search. User-facing docs are in `README.md`; this file is
 - `config.py`: `Settings` from env (`load()`), `ENV_VARS` registry, tuning constants. The only place that
   knows the provider. Every env var must appear in README.md (a test enforces it).
 - `store.py`: SQLite `Store`: history (`messages` shared by groups, `dm_messages` per bot), users, kv,
-  holding-news state, `usage` ledger. Thread-safe; call through `asyncio.to_thread`.
+  holding-news state, `usage` ledger, `period_cache` (finished Sharesight periods, kept until flushed). Thread-safe; call through `asyncio.to_thread`.
 - `msgtext.py` sender_name/describe. `history.py` transcript lines. `textfmt.py` HTML helpers
   (md_to_html, split_html, plain_text, TG_TAG_RE). `tickers.py` Yahoo links (+ `tickers_data.py`).
 - `mcp/`: `server.py` (MCPServer lifecycle, TTL cache, `Registry`), `schema.py` (neutral `ToolDef`,
@@ -24,7 +24,7 @@ MCP data tools and web search. User-facing docs are in `README.md`; this file is
   `openrouter.py` + `zai.py` (the only provider-specific code), `runner.py` (THE tool loop), `policy.py` (THE retry ladder), `gate.py` (which tools a message needs).
 - `prompts.py` all prompts. `ask.py` one entry point: route -> Request -> policy.ask -> usage -> NEEDS_TOOLS rerun.
 - `telegram/`: `handlers.py` (trigger -> request -> reply), `send.py`, `draft.py` (typing + streaming
-  drafts), `commands.py` (/credits), `access.py` (who is an admin). `features/`: `holding_news.py`, `movers.py`, `dm_buttons.py`, `post.py`.
+  drafts), `commands.py` (/credits), `access.py` (who is an admin). `features/`: `holding_news.py`, `movers.py`, `dm_buttons.py`, `post.py`, `periods/` (local `period_returns` tool, cache-first; `windows.py`, `rows.py`, `backfill.py` = PERIOD_BACKFILL loop).
 - `app.py` wiring/startup; `context.py` `Ctx` (settings, store, backend, registry, bot).
 - `lib/` holds only what the bot needs to run. Developer helpers live in `tools/`: `report.py` (`make prompt-report`),
   `capture.py` (real MCP output for fixtures); the bot never imports them.
@@ -57,6 +57,7 @@ MCP data tools and web search. User-facing docs are in `README.md`; this file is
 - z.ai: `tool_choice` is ignored (even "none", so the backend omits tools instead), its built-in `web_search` is ignored whenever function
   tools are present (so search is a function tool run through its search API), thinking can't be disabled (default `reasoning_effort`
   low), an invalid search recency is accepted silently (so the tool schema fixes the list), and usage has no cost (price table).
+- Finished Sharesight periods are cached forever (`period_cache`); after a backdated change run `/flushcache [portfolio] [year|month|day]` or `python bot.py --flush-cache ...`. The unfinished period is never cached. `_keep_sales` in tool args is reserved for the bot's own tools (the runner strips it from the model's).
 - `must_search` requests are never retried without tools (answering from memory would invent news).
 - Telegram HTML: only b/i/u/s/code/pre/a/blockquote/tg-spoiler. `Draft` strips tags (half-written HTML is rejected).
 - No live Telegram/xAI/OpenRouter/MCP access in tests or in this sandbox; say so when behaviour can't be checked.

README.md

@@ -29,6 +29,23 @@ The optional `label` is ignored; it only appears in `ps`/`top` so you can tell i
 A systemd unit just needs `ExecStart=/usr/bin/python3 /path/to/bot.py mimo` and an `Environment=`
 or `EnvironmentFile=` for the variables below.
 
+### Running under systemd
+With `StateDirectory=`, systemd sets `STATE_DIRECTORY` and the bot keeps its database there (`DATA_DIR`
+defaults to it), so the code directory can stay read-only. `mcp_servers.json` is read from the checkout
+unless `MCP_CONFIG` says otherwise. Copy an existing `chat_log.db` (and `-wal`/`-shm`) into the state
+directory before the first start.
+
+```ini
+[Service]
+DynamicUser=yes
+WorkingDirectory=/opt/grokbot
+EnvironmentFile=/opt/grokbot.env
+StateDirectory=grokbot
+Environment=PYTHONUNBUFFERED=1
+ExecStart=/opt/grokbot/.venv/bin/python bot.py
+Restart=on-failure
+```
+
 ## Providers
 - `OPENROUTER_API_KEY` set -> OpenRouter (default model `z-ai/glm-5.3-flash`).
 - `XAI_API_KEY` set -> xAI (default model `grok-4.3`).
@@ -161,6 +178,7 @@ Each is off by default. The holding-news DM and the movers reply are switched on
 | `/start` | DM | Greeting (and the buttons if enabled) |
 | `/holdingnews` | DM, if holding news is on | Run the holding-news check now |
 | `/credits` | an admin (see below), in a group or a DM | Provider balance (OpenRouter; xAI and z.ai have none to show), then requests, tokens (and % cached) and cost per model for the last 24 h and 7 days |
+| `/flushcache [portfolio_id] [year or month or day]` | an admin | Forget cached Sharesight period returns: all, or of one portfolio, or those overlapping a year (`2024`), month (`2024-05`) or day (`2024-05-17`), e.g. `/flushcache 684141 2024`; see [Caching Sharesight history](#caching-sharesight-history) |
 
 **Admins** are the user IDs in `ADMIN_CHAT_IDS` plus anyone who administers a group the bot is in (checked with
 Telegram, cached for five minutes). The bot only knows a group once someone has written in it, or it was added, since
@@ -170,6 +188,23 @@ gets "Portfolio data is for admins only." Holding-news DMs, buttons and down-ser
 unaffected. With nobody in `ADMIN_CHAT_IDS` and no known group, nobody can DM the bot or reach Sharesight: set
 `ADMIN_ONLY=off` to allow everyone.
 
+## Caching Sharesight history
+Questions like "best day / week / month / quarter / year since inception" are answered by the `period_returns` tool
+(admins only), which asks Sharesight for one report per period (about 2.4 s each). Every **finished** period is
+stored in the SQLite database (next to the chat history, so it survives upgrades with `DATA_DIR`) and **never
+expires**: the current, unfinished period is always fetched fresh. Past history rarely changes; after a backdated
+trade or correction in Sharesight, flush the cache with `/flushcache [portfolio_id] [year|month|day]` (an admin, in
+Telegram) or `python bot.py --flush-cache [portfolio_id] [year|month|day]` (needs only `DB_PATH`/`DATA_DIR`, not the
+bot's keys). With a year, month or day, only the periods overlapping it go (every day, week, month, quarter and year
+that contains it), and the background fill re-fetches just those.
+
+Without the cache, a request may fetch at most 64 days, 26 weeks or 120 months/quarters/years that aren't cached; a
+longer range is refused with the earliest start that fits. Set `PERIOD_BACKFILL=<portfolio name>[,...]` (or `all`) to
+fill the cache in the background: months first, then quarters, years, weeks and days (each type for every portfolio before the next), up to two Sharesight calls at a time,
+never while a user's request is using the server (about 1,650 calls per portfolio, an hour or two the first time; progress is logged and it resumes after a restart). Once filled, any range is instant and the
+limits above no longer matter. Every six hours it fetches whatever newly finished. If five fetches in a row fail (Sharesight down or throttling),
+the pass stops and tries again after ten minutes; windows Sharesight simply has no data for are skipped until a restart.
+
 ## Caching and token saving
 **Provider prompt caching** (xAI and OpenRouter keep a conversation on the server holding its cached
 prompt; z.ai caches automatically and gets no key): xAI gets `prompt_cache_key` and the `x-grok-conv-id` header; OpenRouter gets `session_id`,
@@ -246,12 +281,14 @@ Set `TELEGRAM_BOT_TOKEN` and exactly one of the three LLM keys.
 |---|---|
 | `ADMIN_CHAT_IDS` | Telegram IDs of the bot's admins (comma/space separated). Your user ID gets a message when an MCP server goes down and may use `/credits`; a group's chat ID gets the down-server messages only. Admins of the groups the bot is in count as admins too. |
 | `ADMIN_ONLY` | on|off. Only admins may DM the bot or get Sharesight data (default on). |
+| `DATA_DIR` | Directory for the bot's data (the database); a relative `DB_PATH` resolves inside it, an absolute one is used as given. Default: systemd's `STATE_DIRECTORY` (see [Running under systemd](#running-under-systemd)), else the working directory. In Docker, set `DATA_DIR=/data` and bind-mount a host directory there so history survives upgrades. |
 | `DB_PATH` | SQLite file for chat history (default chat_log.db). Instances may share it. |
 | `HISTORY_LIMIT` | Messages of chat history in the prompt; the window is HISTORY_LIMIT to 1.5x (default 20). |
 | `MAX_TOKENS` | Reply cap in tokens, reasoning included (default 3000). |
-| `MCP_CONFIG` | MCP server config file (default mcp_servers.json; missing = no MCP tools). |
+| `MCP_CONFIG` | MCP server config file (default mcp_servers.json next to bot.py, whatever `DATA_DIR` is; missing = no MCP tools). |
 | `MOVERS_BOTS` | Usernames of bots whose end-of-day big-movers lists get explained. Empty (default) = feature off. |
 | `PORTFOLIO_NAMES` | Comma list of Sharesight portfolio names; a message naming one is treated as a portfolio question (default: the recipients' portfolio names). |
+| `PERIOD_BACKFILL` | Comma list of Sharesight portfolio names (or `all`) whose returns per day, week, month, quarter and year are fetched into the local cache in the background (about 1,650 Sharesight calls per portfolio the first time, a few hours), so "best day since inception" answers instantly. Default `all`; list names to limit it, or `off` to disable. See [Caching Sharesight history](#caching-sharesight-history). |
 | `POST_TO_GROUPS_FROM_DM` | on|off. A group admin can have the bot post in that group from a DM (default off). |
 | `SEARCH_MODEL` | OpenRouter only: model that runs the searches (default xiaomi/mimo-v2.6-flash:online). |
 | `SHARESIGHT_HOLDING_NEWS_RECIPIENTS` | Comma list of portfolio:telegram_username pairs to notify. Empty (default) = daily holding-news DM off. |

lib/app.py

@@ -8,7 +8,7 @@
 import json
 import logging
 import os
-import subprocess
+import re
 import sys
 from collections.abc import Sequence
 from pathlib import Path
@@ -29,6 +29,8 @@
 from . import config
 from .context import Ctx
 from .features import dm_buttons, holding_news, post
+from .features.periods import backfill
+from .features.periods.windows import describe_scope, flush_scope
 from .llm.base import Backend
 from .llm.openrouter import OpenRouterBackend
 from .llm.xai import XaiBackend
@@ -52,6 +54,14 @@ def setup_logging() -> None:
         format="%(levelname)s %(message)s" if os.getenv("JOURNAL_STREAM")
         else "%(asctime)s %(levelname)s %(message)s")
     logging.getLogger("httpx").setLevel(logging.WARNING)
+    for handler in logging.getLogger().handlers:
+        handler.addFilter(_drop_protocol_line)
+
+
+def _drop_protocol_line(record: logging.LogRecord) -> bool:
+    """The MCP library's own "Negotiated protocol version" line doesn't say which server it is about; the
+    bot logs "MCP <name>: connected, protocol ..." instead."""
+    return not record.getMessage().startswith("Negotiated protocol version")
 
 
 async def alert_down(ctx: Ctx, server: MCPServer) -> None:
@@ -101,6 +111,8 @@ async def post_init(app: Application) -> None:
             background.append(asyncio.create_task(holding_news.daily_loop(ctx)))
         if st.dm_buttons:
             background.append(asyncio.create_task(dm_buttons.push_on_startup(ctx)))
+        if st.period_backfill:
+            background.append(asyncio.create_task(backfill.loop(ctx)))
 
     async def post_shutdown(app: Application) -> None:
         for task in background:
@@ -124,6 +136,7 @@ async def wrapper(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
 
     # Commands first: the first matching handler wins, and the catch-all would swallow them.
     app.add_handler(CommandHandler("credits", command(commands.credits)))
+    app.add_handler(CommandHandler("flushcache", command(commands.flush_cache)))
     app.add_handler(MessageHandler(
         (filters.ChatType.GROUPS | filters.ChatType.PRIVATE)
         & (filters.UpdateType.MESSAGE | filters.UpdateType.EDITED_MESSAGE)
@@ -141,22 +154,53 @@ def allowed_updates(st: config.Settings) -> list[str]:
             + (["my_chat_member"] if st.post_to_groups or st.admin_only else []))
 
 
-def git_hash() -> str:
-    """Short hash of the commit this checkout is on, or "" if it can't be read (no git, no checkout)."""
+def git_hash(root: Path | None = None) -> str:
+    """Short hash of the commit this checkout is on, or "" if it can't be read. Read from .git directly: the
+    git command refuses a checkout owned by another user (a systemd DynamicUser runs as one)."""
+    root = root or Path(__file__).resolve().parent.parent
     try:
-        done = subprocess.run(["git", "rev-parse", "--short", "HEAD"], cwd=Path(__file__).resolve().parent.parent,
-                              capture_output=True, text=True, timeout=2)
-    except (OSError, subprocess.SubprocessError):
+        git = root / ".git"
+        if git.is_file():  # a linked worktree: ".git" holds "gitdir: <path>"
+            git = (root / git.read_text().split(":", 1)[1].strip()).resolve()
+        common = git / (git / "commondir").read_text().strip() if (git / "commondir").exists() else git
+        head = (git / "HEAD").read_text().strip()
+        if head.startswith("ref:"):
+            ref = head.split(None, 1)[1]
+            try:
+                head = (common / ref).read_text().strip()
+            except FileNotFoundError:  # packed by `git gc`
+                packed = (common / "packed-refs").read_text().splitlines()
+                head = next((line.split()[0] for line in packed if line.endswith(" " + ref)), "")
+        return head[:7] if re.fullmatch(r"[0-9a-f]{40,64}", head) else ""
+    except (OSError, IndexError):
         return ""
-    return done.stdout.strip() if done.returncode == 0 else ""
+
+
+def flush_cache(words: list[str]) -> str:
+    """--flush-cache [portfolio ID] [year|month|day]: needs only the database path, not the bot's keys.
+    Returns the message to print."""
+    scope = flush_scope(words)
+    if scope is None:
+        return "Usage: --flush-cache [portfolio ID] [2024 | 2024-05 | 2024-05-17]"
+    pid, first, last = scope
+    path = config.data_path(os.environ, "DB_PATH", "chat_log.db")
+    n = Store(path).period_flush(pid, first and first.isoformat(), last and last.isoformat())
+    return f"Flushed {n} cached periods ({describe_scope(pid, first, last)})."
 
 
 def main(argv: Sequence[str] | None = None) -> None:
     # The label does nothing: it only shows in ps/top, to tell instances apart when several run
     # side by side with different environments (python bot.py mimo).
     parser = argparse.ArgumentParser(description="Telegram group bot that answers @mentions with an LLM.")
     parser.add_argument("label", nargs="?", help="ignored; identifies this instance in ps/top")
+    parser.add_argument("--flush-cache", nargs="*", metavar="WHAT",
+                        help="delete the cached Sharesight period returns and exit: all, or of a portfolio ID "
+                             "and/or a year, month or day (2024, 2024-05, 2024-05-17)")
     args, _ = parser.parse_known_args(argv)
+    if args.flush_cache is not None:
+        message = flush_cache(args.flush_cache)
+        print(message)
+        sys.exit(0 if message.startswith("Flushed") else 1)
     try:
         st = config.load()
     except config.ConfigError as e:
@@ -168,4 +212,6 @@ def main(argv: Sequence[str] | None = None) -> None:
     log.info("Starting%s%s: %s on %s, reasoning %s, MCP servers: %s",
              f" instance {args.label}" if args.label else "", f" ({commit})" if commit else "", st.model, st.provider,
              st.reasoning or "default", ", ".join(ctx.registry.servers) or "none")
+    if st.period_backfill:
+        log.info("Period backfill: on for %s", ", ".join(sorted(st.period_backfill)))
     app.run_polling(allowed_updates=allowed_updates(st))