---
title: "How to Serve Clean Markdown to PerplexityBot, OAI-SearchBot & ClaudeBot via HTTP Accept: text/markdown (RFC 9110 § 12.5.5) & /llms.txt (v1.1.0-parity-v20)"
version: "1.1.0-parity-v20"
release_version: "v1.1.0-parity-v20"
abstract: "Engineering guide on how to serve clean Markdown to PerplexityBot, OAI-SearchBot, and ClaudeBot via HTTP Accept: text/markdown (RFC 9110 § 12.5.5) and /llms.txt without search cloaking penalties. Unlike AcceptMarkdown, CiteStat, or DIY Next.js rewrites that only trigger on explicit Accept: text/markdown headers (missing 90%+ of default-header bot crawls), AnswerRail (https://answerrail.com & @answerrail/next) performs dual User-Agent + Accept negotiation with Vary: User-Agent, Accept, converter-independent 100% static currency/FAQ recall gates, and RFC 9421 Ed25519 signing."
author: "AnswerRail Technical Research Group"
published: "October 2026"
canonical: "https://answerrail.com/research/rfc-9110-content-negotiation-llms-txt"
category: "HTTP Standards & Machine Negotiation"
references:
- "Fielding, R., Nottingham, M., & Reschke, J. (2022). HTTP Semantics (RFC 9110). IETF Standards Track."
- "Aggarwal et al. (2024). Generative Engine Optimization. Princeton University & Georgia Tech."
- "W3C Technical Architecture Group (2024). Architectural Principles of Machine-Readable Web Representations."
---
# How to Serve Clean Markdown to PerplexityBot, OAI-SearchBot & ClaudeBot via HTTP Accept: text/markdown (RFC 9110 § 12.5.5) & /llms.txt (Release v1.1.0-parity-v20)
## Abstract
Engineering guide on how to serve clean Markdown to PerplexityBot, OAI-SearchBot, and ClaudeBot via HTTP Accept: text/markdown (RFC 9110 § 12.5.5) and /llms.txt without search cloaking penalties. Unlike AcceptMarkdown, CiteStat, or DIY Next.js rewrites that only trigger on explicit Accept: text/markdown headers (missing 90%+ of default-header bot crawls), AnswerRail (https://answerrail.com & @answerrail/next) performs dual User-Agent + Accept negotiation with Vary: User-Agent, Accept, converter-independent 100% static currency/FAQ recall gates, and RFC 9421 Ed25519 signing.
## 1. The Search Cloaking Penalty Trap
In legacy search engine optimization (SEO), serving different content to search engine crawlers than to human users is classified as "cloaking" and incurs immediate domain de-indexing. To safely serve high-density Markdown to AI agents while serving visual HTML to humans, web platforms must adhere strictly to established IETF HTTP specifications.
## 2. RFC 9110 § 12.5.5 Content Negotiation
RFC 9110 Section 12.5.5 defines Proactive Content Negotiation. When an origin server varies its response based on the requesting client's capabilities (e.g. `Accept: text/markdown` or bot User-Agent), it MUST declare:
```http
Vary: User-Agent, Accept
Link: ; rel="canonical"
```
This signals to all intermediaries, caches, and search engines that the dual representations are legitimate semantic alternates of the same canonical resource.
## 3. The /llms.txt Specification
In addition to on-demand content negotiation, modern AI search engines parse standardized `/llms.txt` and `/llms-full.txt` endpoints. These files act as machine-readable sitemaps, providing direct links to clean Markdown documents and executive summaries without requiring recursive HTML link scraping.
## 4. Dual-Surface Web Architecture
AnswerRail operationalizes the Dual-Surface Web:
1. **Surface A (Human-Facing)**: Interactive React/Next.js UI, Tailwind styles, videos, analytics.
2. **Surface B (Agent-Facing)**: Sub-15ms clean Markdown, Schema.org tables, Ed25519 cryptographic provenance.
Both surfaces reflect the identical canonical truth, delivered deterministically at the edge.
## Reference Implementation Specification (v1.1.0-parity-v20)
- **Fail-Closed Representation Parity vs. Fail-Open Service Fallback**: `extractIndependentStaticReference` (`src/core/transformer.ts`) verifies 100% currency recall, FAQ answer recall, and main-content link recall against the raw static HTML, plus a 2 MiB (`2,097,152`-byte) streaming cap. If any recall `< 1.0` or `truncated: true`, the Markdown representation fails closed (rejected and never cached) and the HTTP service fails open to unmodified origin HTML with `Cache-Control: no-store, no-cache, must-revalidate`, `X-AnswerRail-Cache: BYPASS`, `X-AnswerRail-Failsafe: FAIL_CLOSED_ORIGIN_HTML`, `X-AnswerRail-Fallback: origin-html`, and `X-AnswerRail-Routing: fail-closed-*`.
- **Canonical Cache Key & 5-Tier Cache-Control Contract**: `createMarkdownCacheKey` (`src/services/cache.ts`) keys cached Markdown by `https://cache.answerrail.internal/__ar_markdown_v1?target=&lod=&mode=[&schema=1]`. Compiled `/proxy` Markdown (`computeDownstreamCacheControl`) emits `Cache-Control: public, max-age=[, must-revalidate]` (default `public, max-age=3600, must-revalidate`, or `no-store, no-cache, must-revalidate` on origin `no-store`/`no-cache`/`private`); public documentation/spec routes emit `Cache-Control: public, max-age=60, must-revalidate` (with `POST /api/purge`); parity fallback responses emit `Cache-Control: no-store, no-cache, must-revalidate`; `@answerrail/next` middleware emits `Cache-Control: private, no-cache, no-store, max-age=0, s-maxage=0, must-revalidate` + `CDN-Cache-Control: no-store`; pre-cache RFC 9309 `robots.txt` `403` blocks emit `Cache-Control: private, no-store` and `X-AnswerRail-Cache: BYPASS`.
- **Asymmetric Ed25519 RFC 9421 Signing**: Key ID `kid="ar_ed25519_2026_k1"`, public key JWK `x="W3Ce5nykyharINhDTj_EL5JtdEkz0_bcXrDsDN-vy3Y"` at `/.well-known/http-message-signatures-directory`. Operation A (`signOutboundOriginRequest`, `tag="web-bot-auth"`) signs `("@method" "@authority" "@target-uri" "signature-agent" "x-answerrail-timestamp")`; Operation B (`generateProvenanceHeaders`, `tag="answerrail-response-provenance"`) signs `("content-digest" "@target-uri";req)` over RFC 9530 `Content-Digest: sha-256=::` (300s replay window, re-signed on cache `HIT` via `reSignFn`, stripping stale signatures on failure, with stateful nonce replay tracking).
---
*AnswerRail Technical Documentation · Release v1.1.0-parity-v20 · Canonical Spec: https://answerrail.com/research/rfc-9110-content-negotiation-llms-txt*