[shigahi]

Summary

Closes #187.

• Adds three selects under SEO & AI Attribution: search, ai-input and ai-train, each with the options not set, yes and no. A caption links to contentsignals.org. It also says that compliance is voluntary and that blocking crawlers needs AI Crawl Control.

• The generated worker gets a CONTENT_SIGNAL config constant, for example "search=yes, ai-train=no". The value is built at generation time in a fixed order, and values other than yes/no are dropped.

• robots.txt adds Content-Signal: ... to the group for all crawlers:

  • With no custom rules: User-agent: * / Content-Signal: ... / Allow: /.
  • If the custom rules already have a User-agent: * group, the line goes into that group, after any user-agent lines that follow it.
  • Otherwise, a new User-agent: * group with the signal comes first.

  The signal is added to the existing group rather than as a separate one, because a separate * group could hide the user's rules from parsers that only read the first matching group.

• When no signal is set, robots.txt is unchanged.

Not included: sending the same value as a Content-Signal response header, mentioned as optional in #187. That header only matters for Markdown for Agents conversions, which need #185 first, so it can follow later.

Verification

• npm test: 16 tests pass, including four new ones for robots.txt:
  • unchanged without signals
  • fixed order, with unset and invalid values skipped
  • the signal joins the existing * group, which has a second user-agent line
  • a new group is added when the rules have no * group
• npm run build succeeds.
• Drove the generator in Chromium: after setting search=yes and ai-train=no, the copied script contains const CONTENT_SIGNAL = "search=yes, ai-train=no";.