8 findings on AI-readable documentation (w/DocsAlot)

Documentation used to help developers. Today, it also needs to help AI agents.

When developers use Cursor, Claude Code, or ChatGPT to integrate a product, those tools rely on documentation to answer users' queries. Documentation that works well for developers doesn't always work well for AI agents.

The challenge is that most teams don't know how AI-readable their documentation actually is.

To answer that question, I spoke with Faizan Khan, founder of DocsAlot. He shared an analysis of 91 documentation sites exclusively for this article, alongside insights from DocsAlot's public benchmark.

Before diving into the findings, it's worth understanding how DocsAlot evaluates AI-readable documentation and what the benchmark measures.

TL;DR

• DocsAlot evaluates AI-readable documentation across four categories: availability, structure, content quality, and accessibility. Together, these measure whether AI agents can discover documentation, navigate it, retrieve it reliably, and complete developer tasks.

• Using this framework, DocsAlot analyzed 91 documentation sites and found an average DocsAgent Score of 54.4. Only 11 sites (12.1%) scored 80 or higher, while missing code examples, implementation constraints, and incomplete task instructions were the most common issues across the benchmark.

• AI agents perform best when task pages are complete. The highest-scoring documentation sites consistently included prerequisites, parameters, working examples, implementation constraints, and a clear path back to the documentation index.

• Purpose-built documentation platforms outperformed custom implementations. Fern recorded the highest average DocsAgent Score, followed by Mintlify, ReadMe, Docusaurus, and GitBook.

• DocsAlot's traversal experiments showed that navigation directly affects AI performance. AI agents were more likely to recover from incomplete pages when every task page linked back to a documentation index such as llms.txt or index.md.

• Documentation updates typically appeared in AI products within 24 to 72 hours, although some retrieval systems took up to two weeks to reflect changes.

• Looking ahead, DocsAlot believes first-pass answerability should become the standard for measuring AI-readable documentation. Instead of measuring whether AI agents can find a page, teams should measure whether they can answer customer questions correctly using the documentation alone.

How DocsAlot evaluates AI-readable documentation

This article draws on two datasets.

The first is DocsAlot's public benchmark of 196 documentation sites, built using an evaluation framework informed by MongoDB's Agent-Friendly Documentation Spec, created by Dachary Carey (https://github.com/agent-ecosystem/agent-docs-spec).

The second is an analysis of 91 documentation sites that Faizan Khan shared exclusively for this article. The findings throughout this article come from that dataset.

Each documentation site is evaluated across four categories:

• Availability: Can an AI agent discover and access the documentation?
• Structure: Can it navigate the documentation to find the right information?
• Content quality: Does the page contain the information needed to complete the task?
• Accessibility: Can the documentation be fetched and parsed reliably by AI agents?

Every finding in this article comes back to one or more of these four categories.

What does the DocsAgent Score distribution actually look like?

Across the 91 documentation sites analyzed for this article, the average DocsAgent Score was 54.4, while the median was 60.

Most documentation sites clustered around the middle of the distribution, with relatively few reaching the highest scores. After normalizing the raw scores to a strict 100-point scale, only 11 of 91 documentation sites (12.1%) scored 80 or higher.

Lower-scoring documentation sites repeatedly missed the information AI agents rely on to complete a task, including prerequisites, implementation constraints, parameters, and working examples. When those details are missing, agents generate responses using the information available on the page, increasing the likelihood of incorrect code or incomplete answers.

Key takeaway: Audit implementation guides for missing prerequisites, parameters, implementation constraints, and working examples. These gaps consistently appeared across lower-scoring documentation sites.

Where does documentation struggle most for AI agents?

Availability had the widest spread of scores across the benchmark, showing large differences in whether AI agents could consistently discover and retrieve documentation.

Content quality recorded the lowest scores. Every documentation site in the benchmark failed at least one content-quality check.

The most common gaps were missing code examples, undocumented implementation constraints, incomplete task instructions, and pages that explained features without showing developers how to implement them.

Key takeaway: Pick your top 10 implementation guides and review each one for missing code examples, implementation constraints, and task instructions. Those were the most common content gaps across the benchmark. This is important because every page should confidently match the right cluster of user queries, and clearly signal when it is not a fit, so users and agents can look elsewhere for the answer.

What was the most common issue across the documentation benchmark?

Missing code examples was the most common recommendation, affecting 81 of the 91 documentation sites (89.0%).

The next most common recommendations were documenting implementation constraints and requirements (77 sites, 84.6%), adding llms.txt (72 sites, 79.1%), and fixing malformed llms.txt files (68 sites, 74.7%).

Working code examples, implementation constraints, and a documentation index were missing across a large share of the benchmark.

Key takeaway: Review your highest-traffic task pages first. Add a working code example, document implementation constraints and requirements, and make sure every page links back to a valid documentation index. This is important because in the absence of code examples, on the best-match page, the agents tend to hallucinate the answers as reported by DocsAlot

What do AI agents consistently get wrong when traversing documentation?

AI agents typically start at the documentation root or llms.txt, move to an overview page, and then open the first task page that matches the request. When that page omits a working example, implementation constraints, or the next step, agents often complete the task using the information already on the page. That leads to undocumented parameters, incorrect request flows, and invented implementation details.

DocsAlot tested whether a clearer path back to the documentation index improved recovery. Adding a link to llms.txt or index.md at the top of every page consistently helped agents reorient after landing on an incomplete page. The same link at the bottom of the page had a smaller impact, while pages without an index link performed the worst.

Key takeaway: The first task page should stand on its own. Include a working example, implementation constraints, the next implementation step, common failure cases, and a link back to the documentation index.

Does llms.txt actually improve AI readability?

An llms.txt file is only one part of the picture. AI agents need a documentation index they can use to understand the documentation structure and navigate to the right page. That index can be llms.txt, index.md, or another equivalent entry point.

DocsAlot's traversal experiments found that agents recovered more often when every task page linked back to the documentation index. Without that path, agents were more likely to continue using the information already on the page, increasing incorrect responses.

Key takeaway: Focus on making your documentation index easy to reach from every task page. The name and structure of the file  matters less than giving AI agents a reliable way to navigate the rest of the documentation.

Which documentation platform scored highest for AI readability?

Across the 241 documentation sites in the public benchmark, Fern recorded the highest average DocsAgent Score at 92.3, followed by Mintlify (86.3), ReadMe (82.1), Docusaurus (78.7), and GitBook (73.9). Custom-built documentation averaged 50.6, the lowest of any platform included in the benchmark. Documentation hosted on DocsAlot was excluded from the analysis.

The benchmark shows a clear difference between purpose-built documentation platforms and custom implementations. At the same time, scores varied widely within each platform. Mintlify documentation ranged from 35 to 100, while custom-built documentation ranged from 5 to 97.

The highest-scoring documentation sites consistently included complete task pages, working examples, documented implementation constraints, markdown parity, and a clear documentation index.

Key takeaway: There is a high correlation between purpose-built documentation platforms, and AI readability. However, it's not a strict requirement as some documentations even scored higher on custom-built documentation sites. The quality of the documentation still determines the final outcome.

How long does it take AI tools to reflect documentation updates?

DocsAlot doesn't measure how quickly documentation changes appear across AI products as part of its benchmark. For documentation hosted on DocsAlot, however, Faizan Khan tracks this using synthetic prompts, checking how long different AI products take to return updated information after a documentation change.

The results vary by retrieval surface. Some products reflect documentation updates within 24 to 72 hours, while others take one to two weeks.

According to Faizan, the difference is driven more by how often a product refreshes and retrieves documentation than by the underlying model. Perplexity is typically one of the fastest to surface recent changes. Deep research products also tend to update quickly because they retrieve information directly from the web instead of relying primarily on cached indexes.

Key takeaway: Documentation updates can appear in AI responses within a few days, but depending on the product, they may take up to two weeks to become consistently available.

What metric could replace citation tracking for AI-readable documentation?

Faizan Khan believes the next benchmark should measure first-pass answerability.

Citation tracking measures whether an AI tool can locate and reference a documentation page. It doesn't measure whether that page contains enough information to answer the question or complete the task.

The next benchmark should measure first-pass answerability instead.

The idea is simple: take the 20 questions customers ask most often and measure how many an AI agent can answer correctly after reading one or two documentation pages, starting at the root

Building that benchmark requires real customer questions from sources such as support tickets, AI chat transcripts, or community forums. Teams that already have this data can run the exercise today by asking an AI agent those questions using only their documentation and measuring how many are answered correctly without additional searches or human intervention.

Key takeaway: Measure how often your documentation answers customer questions on the first attempt. That tells you whether an AI agent can complete a task using your documentation and not just whether it can find it.

What this means for teams shipping documentation right now

The benchmark gives teams a clear framework for improving AI-readable documentation. The difficult part is execution.

Many of the companies that come to Hackmamba have already experimented with AI tools to generate or expand their documentation. They quickly discover that AI can only document information that already exists. Implementation constraints, edge cases, migration decisions, production examples, and tribal knowledge usually live in engineering discussions, pull requests, support tickets, and the people who built the product. Anthropic's own guidance recommends grounding Claude in source material because unsupported gaps can lead to incorrect outputs.

That's where Hackmamba helps.

Our technical writers work directly with engineering and product teams to interview subject matter experts, review API specifications, pull requests, RFCs, support tickets, and existing documentation, then turn that knowledge into implementation guides, API references, SDK documentation, migration guides, and onboarding documentation. We've supported documentation initiatives for Databricks, Replit, CodeRabbit, Mintlify, Cloudinary, Auth0, OpenRouter, Celo, GBG, and most recently Midnight.

If you've already tried using AI for documentation and now need documentation that's production-ready for both developers and AI agents, Hackmamba can help.