[shigahi]

Summary

Closes #185.

notion.site answers bot-like user agents with a server-rendered page that contains the page content as plain HTML. The worker always fetched pages as Chrome, so crawlers that don't run JavaScript (GPTBot, ClaudeBot, PerplexityBot and others) only saw "JavaScript must be enabled in order to use Notion".

• Crawler detection (isCrawler): the user agent matches bot\b|crawler|spider|chatgpt-user|claude-user|perplexity-user|meta-externalagent, or the request asks for text/markdown. The server-rendered page has no Notion app (no scripts, no #notion-app), so people must never get it. CUBOT phones are excluded explicitly, because their model name matches bot.
• Upstream user agent: crawler page requests are fetched with Mozilla/5.0 (compatible; WorknotBot/1.0; +https://worknot.classmethod.live/). The visitor's user agent is not forwarded, because notion.site returns 403 for Googlebot or Bingbot user agents from IPs that aren't Google's or Bing's. A fixed UA with a bot token gets the server-rendered page.
• Canonical: the server-rendered page has <link rel="canonical"> pointing to notion.site. LinkRewriter removes it, and HeadRewriter adds the site's own canonical as before.
• Links: AnchorRewriter points a[href] links on the notion.site domain to the custom domain, using the slug for mapped pages.
• Render check: a new crawler content check fetches the root page as GPTBot. It verifies the page text, a single canonical URL on the site, and that no links go to notion.site. Like every check, it also runs against notion.site, so if Notion stops server-rendering for bots it shows up as NOTION, not FAIL.

Decision for review: Googlebot

Googlebot and Bingbot match the pattern too, so they now get the server-rendered page instead of the JavaScript app. Google renders JavaScript, so this isn't strictly needed, but it is what notion.site serves to the real Googlebot. I couldn't test it as the real Googlebot. If you'd rather keep search engines on the JavaScript app for now, excluding googlebot|bingbot is a one-line change.

Not included

• attachment: URLs in the server-rendered body (images and files) are left as is. They aren't loadable, but crawlers mostly need the text.
• Pages without a slug get the home page as their canonical URL. This bug existed before this PR, and this PR makes it more visible because crawlers now follow links to those pages. Filed as Unmapped pages declare the home page as canonical URL #190.

Verification

• npm test: 18 tests pass. Six new cases check the user agent sent to Notion: GPTBot, ChatGPT-User, Googlebot and an agent asking for Markdown get WorknotBot; a desktop browser and a CUBOT phone get Chrome.

• npm run check:render against live Notion: crawler content passes, with no FAIL. Four checks are NOTION, meaning they fail on notion.site too: database views, database rows, database item, and navigation.

• In wrangler dev, with a slug items mapped to the test page's database, comparing the old and new worker:

  Request	Old worker	New worker
  / as GPTBot	116 chars, app shell	615 chars with page text
  / as Chrome	app shell	app shell (unchanged)

  For GPTBot, the new worker returns one canonical URL, https://render-check.example.com. The links become https://render-check.example.com/items and https://render-check.example.com/<page-id>. /items as GPTBot returns the database page, with canonical and og:url set to /items.

• Merges cleanly with Add Content Signals to the generated robots.txt #189, which also touches the test file.