## Summary
The `ChatGPT-User` live-fetch agent's user-facing reader hard-fails
(`(400) OK`) on pages we serve it as markdown via user-agent matching,
which made supabase.com blog and product pages unreadable in that
assistant. I root-caused this with a controlled fetch diagnostic
cross-checked against our request logs: the failing fetches never reach
our origin (the failure is cached on their side), pages served as plain
HTML read fine everywhere we tested, and the same failure reproduces on
other major sites that serve UA-matched markdown, so the reader bug is
upstream.
This PR removes user-agent-based markdown serving entirely rather than
special-casing one agent: UA sniffing is a guess about contractless
clients whose fetchers change without notice, and this incident showed
the failure mode is silent (we keep serving 200s while the user-facing
agent breaks). Markdown remains available on every explicit signal —
`Accept: text/markdown` q-value negotiation, explicit `.md` URLs, and
llms.txt — which is the same contract-driven model the Claude fetcher
already uses successfully (it sends `Accept: text/markdown, text/html,
*/*` and keeps receiving markdown after this change).
## Changes
- Remove the `LLM_USER_AGENT` regex and the `userAgent` parameter from
`negotiateMarkdown` in `packages/common/markdown-negotiation.ts`;
decisions now depend only on `Accept`, the `.md` suffix, and the
markdown-variant manifest
- Update both consuming middlewares (`apps/www`, `apps/docs`) to the new
signature; no behavior change for Accept-negotiated or `.md` requests
- Add the missing `Vary: Accept` header to docs guides-md 200 responses
(the www `api-v2/md` route already declares it)
- Fix a pre-existing www bug surfaced in review: explicit changelog
`.md` URLs rewrote to a doubled `.md.md` path (404) under a
markdown-preferring `Accept`, and 406'd on a non-matching `Accept`. The
www middleware now strips the `.md` suffix before slug lookup and passes
`isMarkdownSuffix` into `negotiateMarkdown`, folding the separate
`MD_PAGES` `.md` block into the single negotiation path (same shape as
the docs middleware)
- Rework tests: UA-independence suites replace the per-agent rewrite
tests; a probe Accept header now 406s regardless of user agent
(previously agent UAs were exempt); new changelog `.md` negotiation
coverage
## Testing
Tested locally:
- [x] www middleware suite 36/36, docs middleware suite 17/17
- [x] typecheck green for common, www, docs
Verified on the Vercel previews (www + docs) with curl:
- [x] `ChatGPT-User` and `Claude-User` UA GETs on blog/pricing/guide
pages return `text/html` with a default Accept
- [x] Claude's real Accept (`text/markdown, text/html, */*`) still
returns `text/markdown`; `Accept: text/markdown` and `.md` URLs return
`text/markdown`; probe Accept returns 406
- [x] `/changelog/<slug>.md` with `Accept: text/markdown` returns the
entry markdown as a direct 200 (production today detours through a 308
to the bare URL); changelog index `.md` and bare-entry Accept
negotiation also verified
- [x] docs guides markdown 200s carry `Vary: Accept`
The intermediate commit (ChatGPT-User-only exclusion) was already
verified on the preview: `ChatGPT-User` got HTML while
`Accept`/`.md`/other-UA markdown was unaffected.
Expected effects post-merge: UA-driven markdown volume in the request
logs (~92% of md traffic) collapses to the Accept + `.md` baseline;
named-agent page requests return to prerendered/static serving,
reversing the extra Vercel function invocations the UA rewrite
introduced; user-facing readability in the affected assistant recovers
within ~24h as its fetch cache revalidates. The md-share dashboard gets
a dated annotation; the ratio is not comparable across this change.
## Linear
- fixes GROWTH-973
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Markdown and HTML routing now depends on the request’s `Accept` header
and `.md` links, making content negotiation more predictable.
* Requests that don’t accept available content now consistently return
`406 Not Acceptable`, even for bot-like user agents.
* Guide markdown responses now include an `Accept`-based cache variation
header to improve correct caching behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
In this PR:
- The `llms.ts` script is removed from the pipeline. It was just
iterating over the sections of the SDK JSON files to generate _dummy_
text files for LLMS to consume. But with a lot of inconsistencies and
mistakes.
- There's a new script that builds markdown files from the SDK
definition files.
- The Management API now has a definition file too, it was missing
before.
- On the www project now there are redirects for the reference text
files to the new markdown files in the docs project.
- NPM scripts have received better names so that prebuild ones are
smaller and all are easier to differentiate.
