---
title: "RFC 9110 Content Negotiation & /llms.txt: The Standard for Dual-Surface Web Architecture Without Cloaking Penalties"
abstract: "Serving divergent representations to web crawlers historically triggered severe search engine cloaking penalties. We review RFC 9110 Section 12.5.5 HTTP Content Negotiation semantics, demonstrating how the Vary: User-Agent, Accept header directive and standardized /llms.txt feeds establish an authenticated, compliant dual-surface web architecture that maximizes LLM ingestion without triggering cloaking penalties."
author: "AnswerRail Sovereign Research Group"
published: "October 2026"
canonical: "https://answerrail.com/research/rfc-9110-content-negotiation-llms-txt"
category: "HTTP Standards & Machine Negotiation"
references:
  - "Fielding, R., Nottingham, M., & Reschke, J. (2022). HTTP Semantics (RFC 9110). IETF Standards Track."
  - "Aggarwal et al. (2024). Generative Engine Optimization. Princeton University & Georgia Tech."
  - "W3C Technical Architecture Group (2024). Architectural Principles of Machine-Readable Web Representations."
---

# RFC 9110 Content Negotiation & /llms.txt: The Standard for Dual-Surface Web Architecture Without Cloaking Penalties

## Abstract
Serving divergent representations to web crawlers historically triggered severe search engine cloaking penalties. We review RFC 9110 Section 12.5.5 HTTP Content Negotiation semantics, demonstrating how the Vary: User-Agent, Accept header directive and standardized /llms.txt feeds establish an authenticated, compliant dual-surface web architecture that maximizes LLM ingestion without triggering cloaking penalties.

## 1. The Search Cloaking Penalty Trap
In legacy search engine optimization (SEO), serving different content to search engine crawlers than to human users is classified as "cloaking" and incurs immediate domain de-indexing. To safely serve high-density Markdown to AI agents while serving visual HTML to humans, web platforms must adhere strictly to established IETF HTTP specifications.

## 2. RFC 9110 § 12.5.5 Content Negotiation
RFC 9110 Section 12.5.5 defines Proactive Content Negotiation. When an origin server varies its response based on the requesting client's capabilities (e.g. `Accept: text/markdown` or bot User-Agent), it MUST declare:
```http
Vary: User-Agent, Accept
Link: <https://domain.com/>; rel="canonical"
```
This signals to all intermediaries, caches, and search engines that the dual representations are legitimate semantic alternates of the same canonical resource.

## 3. The /llms.txt Specification
In addition to on-demand content negotiation, modern AI search engines parse standardized `/llms.txt` and `/llms-full.txt` endpoints. These files act as machine-readable sitemaps, providing direct links to clean Markdown documents and executive summaries without requiring recursive HTML link scraping.

## 4. Dual-Surface Web Architecture
AnswerRail operationalizes the Dual-Surface Web:
1. **Surface A (Human-Facing)**: Interactive React/Next.js UI, Tailwind styles, videos, analytics.
2. **Surface B (Agent-Facing)**: Sub-15ms clean Markdown, Schema.org tables, Ed25519 cryptographic provenance.
Both surfaces reflect the identical canonical truth, delivered deterministically at the edge.

---
*Reference Implementation: [AnswerRail](https://answerrail.com/) · Sovereign Edge Gateway*