Push mode refuses a network without its secret; telegramSecretToken; .peg button; nginx docs

CLAUDE.md

@@ -57,7 +57,9 @@ The work branch named in the session instructions (`claude/...`) is the persiste
   straight back; `acknowledge(event, text, alert)` where the network waits for that: Telegram's toast, Discord's
   interaction answer; `bot.take` works out a press's text) and its bot (`TelegramBot`, `SlackBot`, `DiscordBot`).
   `<network>_webhook.py`, the HTTP mode: the finbot.env key of its URL (`URL`, whose path `path()` gives), its secret
-  headers and fields (redacted in the log) and how a request proves itself (`WEBHOOK`). The pull mode's persistent
+  headers and fields (redacted in the log) and how a request proves itself (`WEBHOOK`; `secured()` says finbot.env has the
+  token, signing secret or public key for that: without it the network is turned off, `bot.unsecured()`, its URL served
+  by nobody and its network not started). The pull mode's persistent
   client, handing its updates to `bot.take`, wanted where finbot.env has its keys (`TOKENS`) and no webhook URL
   (`WEBHOOK_URL`): `telegram_longpolling.py` (`TelegramPoll`: getUpdates), `slack_websocket.py` (`SlackSocket`:
   Socket Mode), `discord_websocket.py` (`DiscordGateway`: Hello, IDENTIFY or RESUME, the heartbeat and its ACK; a close

bot.py

@@ -55,6 +55,11 @@ def listeners():
 	app-level token): finbot fetches the updates of those networks itself."""
 	return [kind() for kind in LISTENERS if kind.wanted()]
 
+def unsecured():
+	"""The webhooks whose URL finbot.env sets but not what checks a request (Webhook.secured): their networks are
+	turned away, not served and not started."""
+	return [hook for hook in WEBHOOKS if hook.asked() and not hook.secured()]
+
 def networks_line():
 	"""'networks: discord (push), slack (pull)': the networks finbot.env enables, by name, each with how its updates
 	reach finbot: push (it posts them to a webhook URL) or pull (finbot fetches them); 'networks: none' for none."""
@@ -279,6 +284,9 @@ def http_server():
 	except config.ConfigError as e:
 		config.print_errors(e)
 		sys.exit(1)
+	for hook in unsecured():
+		log.error(f"{hook.service}: {hook.URL} is set but {hook.MISSING} is not: finbot cannot check where its requests"
+			" come from, so the network is turned off")
 	log.info(networks_line())
 	for line in config.startup_report():
 		log.info('config:', line)
@@ -299,8 +307,9 @@ def http_server():
 		log.info(f'Opening socket on http://{config.env.ip}:{server.effective_port}')
 	else:
 		log.info('No webhook is set: finbot takes no HTTP requests')
+	turned_off = {hook.service for hook in unsecured()}
 	for network, key in STARTS: # here only, never on import
-		if getattr(config.env, key):
+		if getattr(config.env, key) and network.READER.service not in turned_off:
 			try:
 				network.start(worker.slash_commands())
 			except Exception as e:

defaults/finbot.env

@@ -38,7 +38,7 @@ discordBotToken=
 # Chat networks: inbound over HTTPS (push mode). Telegram: set both to have it post its updates to your server
 # instead of finbot fetching them
 telegramOutgoingWebhook=
-telegramOutgoingToken=
+telegramSecretToken=
 # Discord: the Interactions Endpoint URL and the app's public key, for slash commands and buttons over HTTPS instead of
 # the gateway API
 discordOutgoingWebhook=

doc/commands.md

@@ -259,7 +259,7 @@ Command:
 ### ratio (pe, peg)
 ![Forward PE report in Slack](../img/finbot_fpe.png?raw=true "Forward PE report in Slack")
 
-List the stocks followed by a ratio: `/ratio` is the forward P/E, lowest first; `/ratio current` the trailing P/E (`current` or `trailing`), of the earnings reported; `/ratio peg` the PEG. Then, optionally, the end: `top` (the default), `bottom` or `reverse` for the highest first, or `negative`, in either order: `/ratio peg reverse`, `/ratio current negative`. A ticker (`/ratio T1`) gets its own figures. `/pe` and `/peg` still work when typed, as `/ratio` and `/ratio peg` (only `/ratio` is in the slash menu); Telegram's keyboard has `.pe`. A forward P/E and a PEG are analysts' estimates (of earnings, of their growth), so their lists take only stocks covered by at least `min_analysts` analysts (7 by default; see `settings.conf`), as `/buy` and `/sell` do; a trailing P/E is no estimate, so every stock counts.
+List the stocks followed by a ratio: `/ratio` is the forward P/E, lowest first; `/ratio current` the trailing P/E (`current` or `trailing`), of the earnings reported; `/ratio peg` the PEG. Then, optionally, the end: `top` (the default), `bottom` or `reverse` for the highest first, or `negative`, in either order: `/ratio peg reverse`, `/ratio current negative`. A ticker (`/ratio T1`) gets its own figures. `/pe` and `/peg` still work when typed, as `/ratio` and `/ratio peg` (only `/ratio` is in the slash menu); Telegram's keyboard has `.pe` and `.peg`. A forward P/E and a PEG are analysts' estimates (of earnings, of their growth), so their lists take only stocks covered by at least `min_analysts` analysts (7 by default; see `settings.conf`), as `/buy` and `/sell` do; a trailing P/E is no estimate, so every stock counts.
 
 Command:
 ```

doc/configuration.md

@@ -29,7 +29,7 @@ of the chat network keys it needs.
 | `slackBotToken` | | Slack bot token, for replies and chart uploads in Slack |
 | `slackAppToken` | | Slack app-level token (`xapp-`): commands over a socket finbot opens to Slack (Socket Mode), when `slackOutgoingWebhook` is empty |
 | `telegramOutgoingWebhook` | | The URL Telegram posts its updates to (push mode); empty, finbot fetches them (pull mode) |
-| `telegramOutgoingToken` | | Secret token Telegram must present (push mode) |
+| `telegramSecretToken` | | Secret token Telegram must present (push mode); was `telegramOutgoingToken`, which still works |
 | `slackOutgoingWebhook` | | The URL Slack posts its events and button presses to (push mode); empty, Socket Mode with `slackAppToken` |
 | `slackOutgoingToken` | | Token Slack must present (push mode): the older way, which Slack has deprecated; not needed with `slackSigningSecret` |
 | `slackSigningSecret` | | Secret Slack signs each request with (push mode) |

doc/discord.md

@@ -14,6 +14,7 @@ The bot does the same either way. What differs is how it hears from Discord:
 | Listens on the internet | No: finbot opens the connection to Discord | Yes, behind your HTTPS server: a domain name with a valid certificate |
 | Commands | Slash commands (`/price`), and messages (`.price`, `@finbot price`) | Slash commands (`/price`): Discord posts no messages |
 | Python modules | `websocket-client` (Debian: `python3-websocket`) | `PyNaCl` and `waitress` (Debian: `python3-nacl`, `python3-waitress`) |
+| How finbot knows it is Discord | Nothing arrives unasked: the gateway connection is one finbot opened with the bot token (`discordBotToken`) | Discord's signature on every request, checked with the app's Public Key (`discordPublicKey`). Without it, finbot turns Discord off |
 | `finbot.env` keys | `discordBotToken` | `discordBotToken`, `discordOutgoingWebhook` and `discordPublicKey` |
 
 In both: the scheduled reports with their charts; period buttons under charts and the buttons of the `/jobs` menu,
@@ -78,7 +79,8 @@ API.
    Discord publishes no list of its addresses, so the signature on each request is what keeps strangers out: finbot
    refuses a request without a good one.
 4. In `finbot.env`, set `discordOutgoingWebhook` to the URL Discord should post to, and `discordPublicKey` to the app's
-   _General Information > Public Key_. Restart finbot, so that it can answer Discord's check in the next step.
+   _General Information > Public Key_ (64 hex digits; without it finbot turns Discord off, and says so in the log).
+   Restart finbot, so that it can answer Discord's check in the next step.
 5. In the Developer Portal, set _General Information > Interactions Endpoint URL_ to the same URL, and _Save Changes_.
    Discord checks it at once, sending a signed request and one with a bad signature, which finbot refuses as Discord
    expects.

doc/slack.md

@@ -16,7 +16,8 @@ The bot does the same either way. What differs is how it hears from Slack:
 | How messages, slash commands and button presses reach finbot | Over a socket that finbot opens to Slack | Slack posts them to your HTTPS server |
 | Listens on the internet | No: finbot opens the socket to Slack | Yes, behind your HTTPS server: a domain name with a valid certificate |
 | Python module | `websocket-client` (Debian: `python3-websocket`) | `waitress` (Debian: `python3-waitress`) |
-| `finbot.env` keys | `slackBotToken`, `slackAppToken` | `slackBotToken`, `slackOutgoingWebhook`, `slackSigningSecret` |
+| How finbot knows it is Slack | Nothing arrives unasked: the socket is one finbot opened with the app-level token (`slackAppToken`) | Slack's signature on every request, made with the app's Signing Secret (`slackSigningSecret`); or the older Verification Token (`slackOutgoingToken`) in the request. With neither, finbot turns Slack off |
+| `finbot.env` keys | `slackBotToken`, `slackAppToken` | `slackBotToken`, `slackOutgoingWebhook`, and `slackSigningSecret` (or `slackOutgoingToken`) |
 
 finbot chooses its mode from `finbot.env`: with `slackOutgoingWebhook` set, finbot uses push mode, with it empty, finbot uses pull
 mode. On the Slack side, enabling _Socket Mode_ expects finbot to pull, and disabling it pushes updates to the _Event Subscriptions Request URL_.
@@ -51,12 +52,10 @@ Slack delivers the messages, the button presses and the slash commands to finbot
 `http://127.0.0.1:5000` (`ip` and `port` in `finbot.env`), so it needs a web server in front of it, on a domain name
 with a valid certificate.
 
-1. Put finbot behind an HTTPS server. An nginx example, which lets in only Amazon's addresses, where Slack's requests
-   come from:
+1. Put finbot behind an HTTPS server. An nginx example:
    ```
    server {
    	listen 8443 ssl;
-   	deny all;
 
    	server_name         www.example.com;
    	ssl_certificate     /etc/letsencrypt/live/www.example.com/fullchain.pem;
@@ -65,14 +64,9 @@ with a valid certificate.
 
    	location /slack {
    		proxy_pass http://127.0.0.1:5000/slack;
-   		include /etc/nginx/aws_subnets;
-   		deny all;
    	}
    }
    ```
-   `util/nginx_aws_cron` writes `/etc/nginx/aws_subnets`, an `allow` line for each of Amazon's networks, and reloads
-   nginx. Run it once as root before nginx loads this configuration, or put it in `/etc/cron.daily/` to keep the list
-   current.
 2. In `finbot.env`, set `slackOutgoingWebhook` to the URL Slack should post to, and restart finbot, so that it can
    answer Slack's check of the URL in step 4.
 3. Print the app's manifest with that URL, which Slack then posts everything to:
@@ -85,6 +79,7 @@ with a valid certificate.
    than five minutes earlier: a replay, or a server clock that is wrong.
    * The older _Verification Token_, set as `slackOutgoingToken`, still works in place of the signing secret, but
      Slack has deprecated it: the token travels in every request, where the signing secret never does.
+   * With `slackOutgoingWebhook` set and neither of them, finbot turns Slack off (it says so in the log at start).
 6. Go on with [installing the app](#installing-the-app), below.
 
 ```

doc/telegram.md

@@ -11,7 +11,8 @@ Telegram:
 |---|---|---|
 | How messages and button presses reach finbot | finbot asks Telegram for them (long polling) | Telegram posts them to your HTTPS server |
 | Listens on the internet | No: finbot only makes outgoing requests | Yes, behind your HTTPS server: a domain name with a valid certificate, on port 443, 80, 88 or 8443 |
-| `finbot.env` keys | `telegramBotToken` | `telegramBotToken`, `telegramOutgoingWebhook` and `telegramOutgoingToken` |
+| How finbot knows it is Telegram | Nothing arrives unasked: finbot's own requests carry the bot token (`telegramBotToken`) | The secret token (`telegramSecretToken`) that finbot gave Telegram when setting the webhook, which Telegram sends back in a header with every update. Without it, finbot turns Telegram off |
+| `finbot.env` keys | `telegramBotToken` | `telegramBotToken`, `telegramOutgoingWebhook` and `telegramSecretToken` |
 
 Everything else is the same in both: the scheduled reports with their charts, the chat commands (`/price`, `/movers`,
 `/performance` and the rest, [the list](commands.md#supported-commands), or as messages, `.price`), period buttons
@@ -62,11 +63,11 @@ Telegram delivers the messages in your chats, and each button press, to finbot o
 with a valid certificate. Telegram posts only to ports 443, 80, 88 and 8443.
 
 1. Set up the bot as for [pull mode](#pull-mode), above.
-2. Put finbot behind an HTTPS server. An nginx example, which lets in only Telegram's addresses:
+2. Put finbot behind an HTTPS server. An nginx example, which lets in only Telegram's addresses (the networks it posts from, as
+   [Telegram lists them](https://core.telegram.org/bots/webhooks#the-short-version)):
    ```
    server {
    	listen 8443 ssl;
-   	deny all;
 
    	server_name         www.example.com;
    	ssl_certificate     /etc/letsencrypt/live/www.example.com/fullchain.pem;
@@ -75,26 +76,25 @@ with a valid certificate. Telegram posts only to ports 443, 80, 88 and 8443.
 
    	location /telegram {
    		proxy_pass http://127.0.0.1:5000/telegram;
-   		include /etc/nginx/telegram_subnets;
+   		allow 149.154.160.0/20;
+   		allow 91.108.4.0/22;
    		deny all;
    	}
    }
    ```
-   `util/nginx_telegram_cron` writes `/etc/nginx/telegram_subnets`, an `allow` line for each of Telegram's networks,
-   and reloads nginx. Run it once as root before nginx loads this configuration, or put it in `/etc/cron.daily/` to
-   keep the list current. `util/iptables_telegram_cron` goes further, in the firewall: of the traffic to ports 443 and
-   8443 it lets in only Telegram's networks on 8443 and drops the rest, so nothing else can be served on those ports.
-3. In `finbot.env`, set `telegramOutgoingWebhook` to the URL Telegram should post to, and `telegramOutgoingToken` to a
+3. In `finbot.env`, set `telegramOutgoingWebhook` to the URL Telegram should post to, and `telegramSecretToken` to a
    password of your choosing: 1 to 256 characters, of letters, digits, `_` and `-`. Telegram sends the password with
-   every update, and finbot turns away a request without it.
+   every update, and finbot turns away a request without it. With `telegramOutgoingWebhook` set and no
+   `telegramSecretToken`, finbot turns Telegram off (it says so in the log at start). The key was called
+   `telegramOutgoingToken`, which still works, with a note in the log to rename it.
 4. Restart finbot. As it starts, it sets the webhook again, deleting it first, which also wakes up a stale connection.
    If Telegram refuses, finbot does not start, and the log says why.
 5. Send `/help` in the chat. In a direct chat with the bot, a keyboard of the common commands follows the reply.
 
 ```
 telegramBotToken=0000000000:AAAAAAAAAAAAAAAAAAAAAAAAAA
 telegramOutgoingWebhook=https://www.example.com:8443/telegram
-telegramOutgoingToken=a-long-random-password
+telegramSecretToken=a-long-random-password
 ```
 
 ## Preloading charts