Drop idle connections and resend reset requests; selection-colour ticks; ticker picker wording

CLAUDE.md

@@ -117,7 +117,7 @@ An open PR takes in every commit pushed to its branch, so a PR opened for one sm
   is a direct message (`dm`), the chat's key (`chat_key`, `split_key`), and the typing notice (`start_typing`: after
   half a second; Slack's ⏳ on the message), which `text()` and `chart()` stop before posting. Each bot's `post_text`
   and `post_chart` (the PNG's bytes) raise when they fail, `MaybeSent` where a chart's post timed out, and
-  `Transport.sent` logs it and gives the answer, `MAYBE_SENT` or None; `edit_buttons` changes a message's buttons
+  `Transport.sent` logs it and gives the answer, `MAYBE_SENT` or None, after sending once more a request the server reset (`net.was_reset`: an idle connection; `net.fresh` drops a host's idle for 20 s before a request); `edit_buttons` changes a message's buttons
   alone where the network can (Telegram, Discord). `DiscordBot`: a slash command's first message replaces its "thinking…" through
   its `interaction` (application id/token), and a pressed message is edited through the press's. A chart the network
   has already (the same PNG) goes by its id, not uploaded again: Telegram's file id (`telegram_common.with_photo`, a
@@ -136,7 +136,7 @@ An open PR takes in every commit pushed to its branch, so a PR opened for one sm
   (what a press asks for is its act, a letter of `buttons.ACTS`: the screen it shows, what its number is, whether it
   changes the buttons alone, its toast; `m`, and `n` under a note, which the next press takes away), 32 items a page
   (`pick.layout`; fewer where the network holds few rows of buttons, `Dialect.room`: Discord's five) turned by
-  `◀` `▶` under the tickers menu, its list of items checked by checksum (`buttons.checksum`, as the `.jobs` menu's list
+  `◀` `▶` under the ticker picker, its list of items checked by checksum (`buttons.checksum`, as the `.jobs` menu's list
   is), `.price`'s ticks (up to `buttons.MAX_PICKED`, 8) a bit each in the data's mask, their buttons and the `.watchlist` Edit's
   built by `pick.ticker_buttons` and `buttons.page_buttons`, one more refused (act `f`, its `refused`: `buttons.refusal`, in
   `bot.take`); a treemap primes its other charts (`pick.treemaps`), and every chart with period buttons its other
@@ -149,8 +149,8 @@ An open PR takes in every commit pushed to its branch, so a PR opened for one sm
   off (❌), and Edit draws their buttons red. The `.jobs` menu's screens: top, Edit, Run, Upcoming and Add's steps; typed
   `.jobs enable`, `disable` or `delete` alone opens Edit.
 - `lib/buttons.py`: `Button`, drawn by its look (`look(button)`: a toggle's `tone`, `ON` or `OFF`, else `ACTIVE` for the
-  one shown now), which each network maps through its `STYLES` (Telegram's `style` success/danger/primary, Slack's
-  primary/danger, Discord's 3/4/1); `tone(on)` makes a job's or ticker's, red only when off; a picker screen's buttons come
+  one shown now), which each network maps through its `STYLES` (Telegram's `style` primary/danger, Slack's
+  primary/danger, Discord's 1/4: ON and ACTIVE are the network's selection colour); `tone(on)` makes a job's or ticker's, red only when off; a picker screen's buttons come
   from `pick.press_button`; rows of buttons (`chunks`) and every menu's ◀ ▶ (`page_buttons`); a chart's period buttons
   (`keyboard`), the `.jobs` menu's (`menu_button`, a `MenuPress`), what a pressed button's data asks for (`parse`), and the
   toast it gets at once (`toast`); each transport draws them, and Slack and Discord take a button's data as its id, so

doc/commands.md

@@ -66,7 +66,7 @@ The bot answers commands in every chat it is in. By default, it fetches its upda
 * A watch list per chat, which anyone in it can change, picked up by the reports there
 * Listing of portfolios and portfolio performance
 * A treemap of a portfolio (`/treemap portfolio [sector|industry|country|currency|market]`): the treemap tiles each holding by value, grouped and coloured by industry (the default), sector, country (its market's, as two letters, with a flag in the caption), currency or market
-* Buttons to pick from where a command takes a portfolio or tickers and is typed without them: `/treemap`, `/holdings` (the first portfolio alphabetically; a treemap by industry, or by the grouping typed) and `/price` (the first ticker, over 7D unless a period or `max` is typed) answer at once, with the buttons under the chart: portfolios four to a row and `/treemap`'s Grouped by button after them, which opens a menu of Country, Currency, Industry, Market and Sector with Back (the one shown marked; a choice redoes the chart in place and keeps the menu open, Back returns to the portfolios); for tickers the period row, ending in `+` to the tickers (those held, else the chat's watchlist) with `◀` `▶` pages: up to 8 ticked (drawn green), which OK draws (a ninth is refused with a message to the presser alone, the chart left as it is). With one ticker ticked, `+` clears it, so that the first tick picks afresh. Another portfolio or ticker, a grouping or a period redoes the chart in place (a tick or a page of tickers changes only the buttons, on Telegram and Discord), and a `/treemap` draws its other charts in the background so the next press is quick. `/price T1 T2 [period]` typed with tickers gets the same period row and `+` when the tickers are among those held (else watched) and the period is a button one. Every chart with a period row, from any command, draws its other periods in the background once it is sent, so a period press is quick. Thirty-two items make a page (fewer on Discord, which holds five rows of five buttons). A job of one posts the same, buttons and all (but draws no charts ahead)
+* Buttons to pick from where a command takes a portfolio or tickers and is typed without them: `/treemap`, `/holdings` (the first portfolio alphabetically; a treemap by industry, or by the grouping typed) and `/price` (the first ticker, over 7D unless a period or `max` is typed) answer at once, with the buttons under the chart: portfolios four to a row and `/treemap`'s Grouped by button after them, which opens a menu of Country, Currency, Industry, Market and Sector with Back (the one shown marked; a choice redoes the chart in place and keeps the menu open, Back returns to the portfolios); for tickers the period row, ending in `+` to the tickers (those held, else the chat's watchlist) with `◀` `▶` pages: up to 8 ticked (drawn in the network's selection colour), which OK draws (a ninth is refused with a message to the presser alone, the chart left as it is). With one ticker ticked, `+` clears it, so that the first tick picks afresh. Another portfolio or ticker, a grouping or a period redoes the chart in place (a tick or a page of tickers changes only the buttons, on Telegram and Discord), and a `/treemap` draws its other charts in the background so the next press is quick. `/price T1 T2 [period]` typed with tickers gets the same period row and `+` when the tickers are among those held (else watched) and the period is a button one. Every chart with a period row, from any command, draws its other periods in the background once it is sent, so a period press is quick. Thirty-two items make a page (fewer on Discord, which holds five rows of five buttons). A job of one posts the same, buttons and all (but draws no charts ahead)
 * Listing of holdings sorted by various attributes 
 * Running the other stock reports on demand
 

doc/telegram.md

@@ -19,7 +19,7 @@ under charts and the buttons of the `/jobs` menu, "typing…" while a command wo
 common commands in a direct chat with the bot: it comes with the first command, and again when it changes; `/start` or
 `/help` brings it back where it went (with the messages, when the chat is cleared).
 
-Buttons are coloured: the period shown blue, a ticked ticker green, a job that is off (or a ticker marked for deletion)
+Buttons are coloured: the period shown blue, a ticked ticker blue too, a job that is off (or a ticker marked for deletion)
 red. Telegram apps from February 2026 draw the colours; older ones draw every button plain.
 
 A chart is uploaded once: Telegram gives it a file id, which finbot keeps (in `DATA_DIR/cache`, 30 days) and sends the

lib/buttons.py

@@ -10,7 +10,7 @@
 from lib import answer
 from lib import state
 
-ON, OFF = 'on', 'off' # a toggle's state, as a button's tone: drawn green and red where the network colours buttons
+ON, OFF = 'on', 'off' # a toggle's state, as a button's tone: the selection colour, or red, where buttons have one
 ACTIVE = 'active' # the look of the button shown now, which a tone overrides
 
 

lib/discord_common.py

@@ -308,22 +308,22 @@ def edit_buttons(self, message, buttons):
 
 
 DISCORD_ROW = 5 # buttons in a row of components, and rows in a message: Discord's limits
-STYLES = {buttons.ON: 3, buttons.OFF: 4, buttons.ACTIVE: 1} # a button's style by its look; grey (2) for the rest
+STYLES = {buttons.ON: 1, buttons.OFF: 4, buttons.ACTIVE: 1} # a button's style by its look; grey (2) for the rest
 
 
 def components(buttons):
 	"""Rows of buttons.Button as Discord's action rows: a row of at most DISCORD_ROW buttons (a longer one in rows of
-	that), and as many rows at most (the buttons past that are left out), a toggle that is on green and one that is off
-	red, the one shown now blurple and the rest grey. A row of buttons that fits stays a row of its own, so a picker's
-	period buttons keep theirs."""
+	that), and as many rows at most (the buttons past that are left out), a toggle that is on, and the one shown now,
+	blurple (Discord's selection colour), one that is off red, and the rest grey. A row of buttons that fits stays a
+	row of its own, so a picker's period buttons keep theirs."""
 	rows = [row[i:i + DISCORD_ROW] for row in buttons for i in range(0, len(row), DISCORD_ROW)][:DISCORD_ROW]
 	return [{'type': 1, 'components': [{'type': 2, 'style': button_style(button), 'label': button.label[:80],
 		'custom_id': button.data} for button in row]} for row in rows]
 
 
 def button_style(button):
-	"""The style number of a Discord button: 3 (green) for a toggle that is on, 4 (red) for one that is off, else 1
-	(blurple) for the one shown now and 2 (grey)."""
+	"""The style number of a Discord button: 1 (blurple, its selection colour) for a toggle that is on and the one
+	shown now, 4 (red) for a toggle that is off, else 2 (grey)."""
 	return STYLES.get(buttons.look(button), 2)
 
 

lib/net.py

@@ -6,7 +6,9 @@
 would not set a new one."""
 import http.cookiejar
 import re
+import threading
 import time
+import urllib.parse
 
 import requests
 
@@ -17,6 +19,10 @@
 # no domain may set or receive a cookie
 session.cookies.set_policy(http.cookiejar.DefaultCookiePolicy(allowed_domains=[]))
 CONNECT_TIMEOUT = 5 # seconds to wait for a host to answer the connection: one that is down costs this, not http_timeout
+IDLE_SECONDS = 20 # a host's connections idle this long are dropped, not reused: servers close them, and a request sent
+# on one fails with a reset (a chart upload, a reply)
+_used = {} # host -> when a request to it last began (time.monotonic)
+_used_lock = threading.Lock()
 RATE_LIMIT_WAIT = 30 # the longest a rate limit is waited out, in seconds: one that asks for longer is tried then
 
 
@@ -42,11 +48,22 @@ def request(method, url, **kwargs):
 	logged."""
 	return session.request(method, url, **timeouts(logged(method, url, kwargs)))
 
+def fresh(url):
+	"""Drop the shared session's connections if the host of this url has had none used for IDLE_SECONDS: a server
+	closes an idle connection, and a request sent on it fails with a reset. Returns None."""
+	host, now = urllib.parse.urlsplit(url).netloc, time.monotonic()
+	with _used_lock:
+		idle = now - _used.get(host, now) >= IDLE_SECONDS
+		_used[host] = now
+	if idle:
+		session.get_adapter(url).poolmanager.clear()
+
 def logged(method, url, kwargs):
 	"""The request's arguments without `logged`; the request noted first, its method and URL without secrets or the
 	scheme, cut to the log_width setting (log.cut): at INFO; at DEBUG with logged='debug' (the routine of a press:
 	its acknowledgement, the message edited in place); not at all with logged=False (Telegram's long polling, asked
-	again and again)."""
+	again and again). A host's idle connections are dropped first (fresh)."""
+	fresh(url)
 	shown = kwargs.pop('logged', True)
 	if shown:
 		(log.debug if shown == 'debug' else log.info)(log.cut(f"{method} {redact(url).split('://', 1)[-1]}"))
@@ -77,6 +94,12 @@ def twice(send):
 	except requests.RequestException:
 		return send()
 
+def was_reset(error):
+	"""Whether a request failed because the server reset its connection (errno 104), as one left idle is: requests
+	wraps urllib3's ProtocolError, which wraps the ConnectionResetError, in its ConnectionError."""
+	return isinstance(error, ConnectionResetError) or any(was_reset(arg) for arg in getattr(error, 'args', ())
+		if isinstance(arg, BaseException))
+
 def patiently(send):
 	"""send(), tried once more after the wait that a rate limit asks for (RateLimited; RATE_LIMIT_WAIT at most)."""
 	try:

lib/slack_common.py

@@ -356,14 +356,14 @@ def section_blocks(text):
 
 
 def button_style(button):
-	"""The Block Kit style of a button (STYLES): danger (red) for a toggle that is off, primary (green) for one that is
-	on or the one shown now; none for the rest."""
+	"""The Block Kit style of a button (STYLES): danger (red) for a toggle that is off, primary (Slack's one selection
+	colour, green) for one that is on or the one shown now; none for the rest."""
 	style = STYLES.get(buttons.look(button))
 	return {'style': style} if style else {}
 
 
 def actions_block(row):
-	"""A Block Kit row of buttons, the one shown now marked, a toggle green (on) or red (off)."""
+	"""A Block Kit row of buttons, the one shown now marked, a toggle in the selection colour (on) or red (off)."""
 	return {'type': 'actions', 'elements': [{'type': 'button', 'text': {'type': 'plain_text', 'text': button.label},
 		'value': button.data, 'action_id': button.data, **button_style(button)} for button in row]}
 

lib/telegram_common.py

@@ -34,7 +34,7 @@ def try_call(method, **kwargs):
 		log.warn(f"Telegram {method} failed:", net.redact(e))
 
 
-STYLES = {buttons.ON: 'success', buttons.OFF: 'danger', buttons.ACTIVE: 'primary'} # a button's style by its look
+STYLES = {buttons.ON: 'primary', buttons.OFF: 'danger', buttons.ACTIVE: 'primary'} # a button's style by its look
 ALLOWED_UPDATES = ['message', 'callback_query'] # what finbot takes, whichever way it arrives: messages, presses
 KEYBOARD_FILE = 'finbot_keyboards.json' # chat id -> the version of the command keyboard that DM has (keep_keyboard)
 PHOTO_SECONDS = 30 * 86400 # how long a chart's file id is reused: Telegram keeps a bot's files, cleanup the cache file
@@ -161,8 +161,9 @@ def answerCallbackQuery(callback_query_id, text=None, alert=False):
 
 
 def button_style(button):
-	"""Telegram's style of a button (STYLES): success (green) for a toggle that is on, danger (red) for one that is
-	off, primary (blue) for the one shown now; none for the rest. A client before February 2026 draws none."""
+	"""Telegram's style of a button (STYLES): primary (blue, its selection colour) for a toggle that is on and for the
+	one shown now, danger (red) for a toggle that is off; none for the rest. A client before February 2026 draws
+	none."""
 	style = STYLES.get(buttons.look(button))
 	return {'style': style} if style else {}
 

lib/transports.py

@@ -163,10 +163,17 @@ def sent(self, what, post):
 		"""The answer of post(), which posts one message (what: 'message' or 'chart', for the log): the network's
 		message; MAYBE_SENT when the request that posts a chart timed out (MaybeSent: it may have arrived); None when it
 		failed, which is logged: an error for a message, which is lost, a warning for a chart, whose report goes as text
-		instead (delivery.deliver)."""
+		instead (delivery.deliver). A request the server reset (a stale connection, which net.fresh avoids) is sent
+		again once, whole: at worst a message or chart that did arrive comes twice."""
 		network = self.name.capitalize()
 		try:
-			answer = post()
+			try:
+				answer = post()
+			except Exception as e:
+				if not net.was_reset(e):
+					raise
+				log.warn(f"{network} {what} to {self.where} was reset, sending it again:", net.redact(e))
+				answer = post()
 		except MaybeSent as e:
 			log.warn(f"{network} {what} to {self.where} timed out, it may still arrive:", net.redact(e))
 			return MAYBE_SENT

tests/test_edges.py

@@ -518,6 +518,55 @@ def test_the_session_keeps_no_cookies(self):
 		net.session.cookies.extract_cookies(response, urllib.request.Request('https://fc.yahoo.com/'))
 		self.assertEqual(len(net.session.cookies), 0)
 
+	def test_a_hosts_connections_idle_too_long_are_dropped_before_the_next_request(self):
+		"""A server closes a connection left idle, and a request sent on it fails with a reset: the session drops a
+		host's connections once none has been used for IDLE_SECONDS, and not before, nor those of a host just used."""
+		from lib import net
+		clock, cleared = [1000.0], []
+		adapter = type('Adapter', (), {'poolmanager': type('Pools', (), {'clear': lambda self: cleared.append(1)})()})()
+		with mock.patch.object(net.time, 'monotonic', lambda: clock[0]), mock.patch.object(net.session, 'get_adapter',
+				lambda url: adapter), mock.patch.dict(net._used, clear=True):
+			net.fresh('https://api.telegram.org/bot1/sendPhoto') # the first use of a host
+			clock[0] += net.IDLE_SECONDS - 1
+			net.fresh('https://api.telegram.org/bot1/sendMessage') # used a moment ago
+			self.assertEqual(cleared, [])
+			clock[0] += net.IDLE_SECONDS
+			net.fresh('https://api.telegram.org/bot1/sendPhoto') # idle for IDLE_SECONDS
+			self.assertEqual(cleared, [1])
+			net.fresh('https://api.telegram.org/bot1/sendPhoto')
+			self.assertEqual(cleared, [1])
+
+
+class ResetConnections(FinbotCase):
+	def test_a_chart_or_message_the_server_reset_is_sent_again_whole(self):
+		"""A request that fails with a connection reset (errno 104: the server closed an idle connection) is sent again
+		once as it was, a chart too, not as text with its heading alone; another failure, or a second reset, is not
+		sent again, and the chart's report goes as text as before."""
+		import requests, urllib3
+		reset = requests.ConnectionError(urllib3.exceptions.ProtocolError('Connection aborted.',
+			ConnectionResetError(104, 'Connection reset by peer'))) # as requests raises one
+		bot, tries = telegram_common.TelegramBot('55'), []
+		def post(outcome):
+			"""A post that raises what each try is given."""
+			def one():
+				"""One try: its outcome."""
+				tries.append(1)
+				error = outcome[len(tries) - 1]
+				if error:
+					raise error
+				return {'message_id': 1}
+			return one
+		with contextlib.redirect_stderr(io.StringIO()):
+			self.assertEqual(bot.sent('chart', post([reset, None])), {'message_id': 1})
+			self.assertEqual(len(tries), 2)
+			tries.clear()
+			self.assertIsNone(bot.sent('chart', post([reset, reset])))
+			self.assertEqual(len(tries), 2)
+			tries.clear()
+			self.assertIsNone(bot.sent('message', post([RuntimeError('Forbidden'), None])))
+			self.assertEqual(len(tries), 1)
+		self.assertTrue(net.was_reset(reset) and not net.was_reset(requests.ConnectionError('refused')))
+
 
 class SignalHandOff(unittest.TestCase):
 	def test_a_signal_handler_hands_its_work_to_a_thread_one_at_a_time(self):