---
title: "The Static /llms.txt Paradox: 2026 Empirical Benchmark of HTML Token Truncation & Visible Price Omission Across 50 Production SaaS & Commerce Sites (v1.1.0-parity-v20)"
version: "1.1.0-parity-v20"
release_version: "v1.1.0-parity-v20"
abstract: "Reproducible audit of 50 production B2B SaaS, developer infrastructure, and e-commerce pricing pages under PerplexityBot/1.0. While 87% (41/47 reachable sites) now publish a root /llms.txt manifest and 19% (9/47) publish /.well-known/ai-catalog.json (Lighthouse 13.5 ARD 1.0), 74% (35/47) still exceed the 25,000-token (~100 KB) RAG truncation boundary on their live /pricing URL, and 62% (29/47) omit distinct visible pricing tiers past byte 100,000. Conversely, 26% (12/47) fit cleanly within 25,000 tokens and require zero edge proxy intervention."
author: "AnswerRail Technical Research Group"
published: "October 2026"
canonical: "https://answerrail.com/research/2026-saas-50-truncation-benchmark"
category: "Empirical Benchmark Study (N=50)"
references:
  - "quitegoodproject/test-ai-ready Benchmark Suite (2026). N=50 SaaS & E-Commerce pricing pages audited under PerplexityBot/1.0."
  - "IETF RFC 9110 (2022). HTTP Semantics § 12.5.5: Content Negotiation & Vary Header Caching."
  - "IETF RFC 9421 (2024). HTTP Message Signatures (Ed25519 Cryptographic Provenance)."
  - "IETF draft-w3c-publishing-cg-ai-readiness (2026). Lighthouse 13.5 AI Readiness Descriptor (ARD 1.0) Specification."
---

# The Static /llms.txt Paradox: 2026 Empirical Benchmark of HTML Token Truncation & Visible Price Omission Across 50 Production SaaS & Commerce Sites (Release v1.1.0-parity-v20)

## Abstract
Reproducible audit of 50 production B2B SaaS, developer infrastructure, and e-commerce pricing pages under PerplexityBot/1.0. While 87% (41/47 reachable sites) now publish a root /llms.txt manifest and 19% (9/47) publish /.well-known/ai-catalog.json (Lighthouse 13.5 ARD 1.0), 74% (35/47) still exceed the 25,000-token (~100 KB) RAG truncation boundary on their live /pricing URL, and 62% (29/47) omit distinct visible pricing tiers past byte 100,000. Conversely, 26% (12/47) fit cleanly within 25,000 tokens and require zero edge proxy intervention.

## 1. The Static /llms.txt Paradox (87% Manifest Adoption vs. 74% /pricing Truncation)
In October 2026, we audited 50 production SaaS, developer infrastructure, and e-commerce domains using `npx test-ai-ready` under active `PerplexityBot/1.0` retrieval simulation. The study surfaces a fundamental architectural mismatch: **manifest adoption has succeeded, but runtime document delivery remains broken**.

While 87% (41 of 47 reachable sites) publish a root `/llms.txt` file, **74% (35 of 47) still exceed the 25,000-token (~100 KB) RAG truncation boundary on their live `/pricing` URL**, and **62% (29 of 47) omit distinct visible pricing tiers** because the prices sit beyond byte 100,000 in raw HTML.

| Metric | Measured Value (N=50) | Percentage | Significance |
| :--- | :--- | :--- | :--- |
| **Total Analyzed Domains** | 50 Domains | 100% | Full Golden Corpus benchmark set |
| **Reachable Live Origins** | 47 Sites | 94% | 3 origins timed out or blocked edge crawl |
| **Root `/llms.txt` Present** | 41 Sites | **87.2%** | Broad adoption of static AI discovery manifests |
| **`ai-catalog.json` (Lighthouse 13.5 ARD 1.0)** | 9 Sites | **19.1%** | Emerging agentic catalog standard |
| **Exceeds 25k Token Truncation Boundary** | 35 Sites | **74.5%** | Raw HTML exceeds ~100 KB RAG crawler context limit |
| **Visible Pricing Tiers Omitted Past Byte 100k** | 29 Sites | **61.7%** | Critical pricing models truncated before retrieval |
| **Passes Within 25k Limit (Zero Proxy Needed)** | 12 Sites | **25.5%** | Clean semantic HTML fits without edge proxy |
| **Mean Raw HTML Token Count** | 163,511 tokens | — | Average uncompacted origin payload |
| **Mean AnswerRail Markdown Token Count** | 6,330 tokens | — | **-96.1% compaction** across all tested pages |

## 2. Reproducible Methodology, Visible-DOM Price Isolation & Honest Limitations

### Reproducibility via `node benchmark/run-study.mjs` & `npx test-ai-ready`
All findings are reproducible using the open-source CLI suite:
```bash
# Audit any individual URL
npx test-ai-ready https://stripe.com/pricing

# Reproduce the full benchmark study
node benchmark/run-study.mjs
```

### Visible-DOM Price Isolation: Eliminating Minified JS False Positives
A key methodological challenge in measuring dropped prices is distinguishing genuine human-visible prices (e.g. `$20/mo`, `€49`, `£150`) from minified JavaScript variables or SVG coordinates (e.g., `$1`, `$2` in bundle chunks).
Before regex currency extraction, the audit harness parses the document AST and aggressively strips:
1. **`<script>` and `<template>` blocks**: Removing Next.js `self.__next_f.push` RSC hydration blobs and inline telemetry scripts.
2. **`<style>` and stylesheet tags**: Removing CSS rules, animations, and Tailwind utility declarations.
3. **`<svg>` elements**: Removing path data, coordinate vectors, and embedded graphics.

Only text nodes residing in the visible DOM tree are evaluated for currency tokens. The harness calculates the exact byte offset where each unique price token first appears in the raw response stream. Any price appearing past byte 100,000 (~25,000 tokens at ~3.7 bytes/token) is marked as **Omitted Under 25k Truncation**.

### Honest Limitations: BPE Approximation & Lightweight Sites
- **BPE Token Estimation Caveat**: Token counts are calculated using the standard BPE heuristic of `bytes / 3.7` (empirically calibrated against `cl100k_base` on HTML corpora, margin of error ±8%).
- **Lightweight Sites Require Zero Edge Proxies ($0)**: 12 of the 47 reachable sites (**26%**) fit cleanly within the 25,000-token boundary:
  `linear.app`, `clerk.com`, `resend.com`, `neon.tech`, `turso.tech`, `cal.com`, `dub.co`, `plain.com`, `ghost.org`, `tailwindcss.com`, `workos.com`, and `answerrail.com`.
  These domains have concise semantic HTML, small hydration footprints, and do **not** require a paid edge proxy. Engineering teams running sites like these should simply run `npx test-ai-ready <url> --write` for $0 to commit static `/llms.txt` and `/.well-known/ai-catalog.json` directly to their repository.

## 3. Empirical Results: Top 15 Truncating Pricing Pages
The following table details the 15 production pricing pages in our dataset with the highest volume of dropped visible pricing tiers resulting from HTML payload bloat:

| Origin Domain | Raw HTML Tokens | Truncated? | Dropped Prices | Compaction | Audit Report |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **datadoghq.com** | 451,745 tokens | Truncated (&gt;25k) | **111 / 111 dropped** (100%) | -97.2% | [View Report](/report/datadoghq.com) |
| **fly.io** | 176,742 tokens | Truncated (&gt;25k) | **51 / 51 dropped** (100%) | -96.5% | [View Report](/report/fly.io) |
| **render.com** | 175,740 tokens | Truncated (&gt;25k) | **47 / 50 dropped** (94%) | -96.1% | [View Report](/report/render.com) |
| **stripe.com** | 261,031 tokens | Truncated (&gt;25k) | **28 / 28 dropped** (100%) | -98.1% | [View Report](/report/stripe.com) |
| **monday.com** | 383,315 tokens | Truncated (&gt;25k) | **27 / 27 dropped** (100%) | -98.4% | [View Report](/report/monday.com) |
| **woocommerce.com** | 81,972 tokens | Truncated (&gt;25k) | **27 / 27 dropped** (100%) | -94.8% | [View Report](/report/woocommerce.com) |
| **vercel.com** | 334,474 tokens | Truncated (&gt;25k) | **27 / 28 dropped** (96%) | -97.8% | [View Report](/report/vercel.com) |
| **shopify.com** | 291,841 tokens | Truncated (&gt;25k) | **15 / 15 dropped** (100%) | -97.0% | [View Report](/report/shopify.com) |
| **bigcommerce.com** | 400,547 tokens | Truncated (&gt;25k) | **14 / 14 dropped** (100%) | -98.2% | [View Report](/report/bigcommerce.com) |
| **checklyhq.com** | 102,281 tokens | Truncated (&gt;25k) | **13 / 16 dropped** (81%) | -95.6% | [View Report](/report/checklyhq.com) |
| **upstash.com** | 68,083 tokens | Truncated (&gt;25k) | **13 / 17 dropped** (76%) | -94.2% | [View Report](/report/upstash.com) |
| **railway.com** | 88,425 tokens | Truncated (&gt;25k) | **12 / 18 dropped** (67%) | -95.1% | [View Report](/report/railway.com) |
| **supabase.com** | 104,298 tokens | Truncated (&gt;25k) | **9 / 24 dropped** (38%) | -94.7% | [View Report](/report/supabase.com) |
| **figma.com** | 695,422 tokens | Truncated (&gt;25k) | **8 / 8 dropped** (100%) | -98.9% | [View Report](/report/figma.com) |
| **framer.com** | 336,116 tokens | Truncated (&gt;25k) | **7 / 7 dropped** (100%) | -97.5% | [View Report](/report/framer.com) |

## 4. Architecture Decision Matrix: When to Use Free --write ($0) vs. Layer-7 Edge Compaction

Engineering and marketing teams often ask whether they should configure an edge reverse proxy or simply check static files into their codebase. This benchmark establishes an objective, evidence-based decision matrix:

| Site Architecture & Profile | Token Budget | Root File Upload Access | Recommended Solution | Cost |
| :--- | :--- | :--- | :--- | :--- |
| **Next.js / Astro / Hugo / Static** | &le; 25,000 tokens | Direct file access (`./public/`) | **`npx test-ai-ready <url> --write`** | **$0** (Free static generation) |
| **Shopify / Webflow / Squarespace** | Any size | **Blocked by CMS** (cannot host `/.well-known/`) | **Layer-7 Edge Proxy** (`edge.answerrail.com`) | Self-serve starting $99/mo |
| **Heavy React / Next.js / Angular** | &gt; 25,000 tokens | Direct file access (`./public/`) | **`@answerrail/next` or Edge Proxy** | Self-serve starting $99/mo |
| **Enterprise / Dynamic Inventory** | &gt; 50,000 tokens | High-velocity CMS / DB updates | **Dynamic Edge Proxy + RFC 9421 Provenance** | Custom / Enterprise |

### Decision Rule 1: The Lightweight Stack ($0 Path)
If your origin site produces semantic HTML under 25,000 tokens (e.g. Next.js with minimal RSC bloat, Astro, or static docs), you do not need edge transformation. Run:
```bash
npx test-ai-ready https://yourdomain.com --write
```
This writes an authoritative `llms.txt` and Lighthouse 13.5 `ai-catalog.json` to your repository's public directory at zero ongoing cost.

### Decision Rule 2: The CMS Root File Wall
Hosted platforms such as Shopify, Squarespace, and Webflow physically prohibit uploading custom files to the root domain (`/llms.txt` and `/.well-known/ai-catalog.json`). A DNS-level edge proxy (`edge.answerrail.com`) terminates incoming AI bot requests at Cloudflare Anycast edge, serving the required manifests and cached clean Markdown without requiring CMS changes or app store plugins.

### Decision Rule 3: The 25k Token Truncation Threshold
If your live pricing or catalog route exceeds 25,000 tokens (due to React Server Component hydration strings, heavy nested DOMs, or comprehensive feature comparisons), search bots will silently discard everything below byte 100,000. For these origins, Layer-7 edge compaction via AnswerRail or `@answerrail/next` is strictly necessary to prevent AI search models from omitting your paid tiers or quoting outdated third-party scrape data.

## Reference Implementation Specification (v1.1.0-parity-v20)
- **Fail-Closed Representation Parity vs. Fail-Open Service Fallback**: `extractIndependentStaticReference` (`src/core/transformer.ts`) verifies 100% currency recall, FAQ answer recall, and main-content link recall against the raw static HTML, plus a 2 MiB (`2,097,152`-byte) streaming cap. If any recall `< 1.0` or `truncated: true`, the Markdown representation fails closed (rejected and never cached) and the HTTP service fails open to unmodified origin HTML with `Cache-Control: no-store, no-cache, must-revalidate`, `X-AnswerRail-Cache: BYPASS`, `X-AnswerRail-Failsafe: FAIL_CLOSED_ORIGIN_HTML`, `X-AnswerRail-Fallback: origin-html`, and `X-AnswerRail-Routing: fail-closed-*`.
- **Canonical Cache Key & 5-Tier Cache-Control Contract**: `createMarkdownCacheKey` (`src/services/cache.ts`) keys cached Markdown by `https://cache.answerrail.internal/__ar_markdown_v1?target=<canonicalTarget>&lod=<compact|standard|deep>&mode=<twin|ast>[&schema=1]`. Compiled `/proxy` Markdown (`computeDownstreamCacheControl`) emits `Cache-Control: public, max-age=<min(14400, originTtl)>[, must-revalidate]` (default `public, max-age=3600, must-revalidate`, or `no-store, no-cache, must-revalidate` on origin `no-store`/`no-cache`/`private`); public documentation/spec routes emit `Cache-Control: public, max-age=60, must-revalidate` (with `POST /api/purge`); parity fallback responses emit `Cache-Control: no-store, no-cache, must-revalidate`; `@answerrail/next` middleware emits `Cache-Control: private, no-cache, no-store, max-age=0, s-maxage=0, must-revalidate` + `CDN-Cache-Control: no-store`; pre-cache RFC 9309 `robots.txt` `403` blocks emit `Cache-Control: private, no-store` and `X-AnswerRail-Cache: BYPASS`.
- **Asymmetric Ed25519 RFC 9421 Signing**: Key ID `kid="ar_ed25519_2026_k1"`, public key JWK `x="W3Ce5nykyharINhDTj_EL5JtdEkz0_bcXrDsDN-vy3Y"` at `/.well-known/http-message-signatures-directory`. Operation A (`signOutboundOriginRequest`, `tag="web-bot-auth"`) signs `("@method" "@authority" "@target-uri" "signature-agent" "x-answerrail-timestamp")`; Operation B (`generateProvenanceHeaders`, `tag="answerrail-response-provenance"`) signs `("content-digest" "@target-uri";req)` over RFC 9530 `Content-Digest: sha-256=:<base64>:` (300s replay window, re-signed on cache `HIT` via `reSignFn`, stripping stale signatures on failure, with stateful nonce replay tracking).

---
*AnswerRail Technical Documentation · Release v1.1.0-parity-v20 · Canonical Spec: https://answerrail.com/research/2026-saas-50-truncation-benchmark*