Generative Engine Optimization (GEO): Why Your Site Needs llms.txt

Search returned ten links and you picked one. An assistant returns one answer and cites three sources. The unit of competition has changed from a ranking position to a quotation, and most sites are still optimising for a list that no longer exists.

Generative engine optimisation — GEO, sometimes called answer engine optimisation — is what happens when you take that shift seriously: you write and structure a site so that a retrieval system can find a passage, understand it without surrounding context, and quote it accurately. Much of it is just clear writing. The rest is a handful of structural decisions, one of which is a small text file called llms.txt.

What an answer engine actually does

It is tempting to imagine a model reading your site the way a person does. In practice the pipeline is closer to this:

1. Retrieve. A search index — built by a crawler, sometimes augmented by live fetching — returns candidate pages for a query.
2. Chunk. Pages are split into passages, typically a heading plus one to three paragraphs, a few hundred tokens each.
3. Rerank. Chunks are scored for relevance to the specific question, often without their page context.
4. Generate. The model reads the top chunks and writes an answer, citing the sources it used.

Step two is where ordinary SEO advice breaks. A page that ranks well because of a strong overall argument can be invisible at the chunk level, because no single passage answers anything on its own. Conversely, a plainly written paragraph that begins with the answer and names its subject is easy to retrieve, easy to verify, and easy to cite.

What the AI crawlers are, and are not

Different bots do different jobs, and blocking one is not the same as blocking another. Names and policies change, so check each provider's current documentation before making a decision; this is the state of play at the time of writing.

Two practical points. First, blocking GPTBot does not remove you from ChatGPT search, and blocking Google-Extended does not remove you from Google Search snippets — those are different controls, and conflating them is the most common policy mistake here. Second, any crawler that arrives because a user asked about your page is arguably a visitor, not a scraper: ChatGPT-User and Claude-User behave like a browser fetching on someone's behalf.

The llms.txt proposal

llms.txt was proposed in September 2024 by Jeremy Howard as a Markdown file at the root of a site that gives a language model a compact, curated map of the site's useful content. The idea is simple: instead of asking a crawler to reverse-engineer a site from navigation, JavaScript and cookie banners, hand it a plain-text index with a summary of each important page.

The format

The conventional shape: an H1 with the project name, a blockquote with a one-sentence summary, optional prose with key facts, then H2 sections containing link lists, each link with an optional note after a colon. An optional llms-full.txt can hold the full text of the documents for tools that want everything in one fetch.

# Pikkola Journal

> A technical blog about small, fast, browser-based tools: remote work, diffing, metadata and generative search.

Pikkola is an independent software studio that builds free browser tools.
Every article here is written and edited in-house; no sponsored posts are published.

## Articles
- [Managing timezones across remote teams](https://blog.pikkola.com/posts/managing-timezones-across-remote-teams): overlap windows, rotation and the DST gap.
- [Web-based text diffing and JSON Patch](https://blog.pikkola.com/posts/web-text-diffing-and-json-patch): LCS limits, word diffs, RFC 6902.
- [Open Graph and X cards](https://blog.pikkola.com/posts/open-graph-twitter-cards-guide): tags, image crops, edge rendering.

## Tools
- [Pikkola Time](https://time.pikkola.com): timezone meeting planner, free, no account.
- [Pikkola Diff](https://diff.pikkola.com): in-browser text and JSON diff.
- [Pikkola Meta](https://meta.pikkola.com): social card preview generator.

## Optional
- [About](https://blog.pikkola.com/about), [Privacy](https://blog.pikkola.com/privacy)

Note what the file does not do: it does not repeat the articles, it does not contain keywords, and it is not a sitemap. It is a short, honest description of what the site is and which pages are worth reading. If you would not say a sentence to a colleague, do not put it in the file.

Write passages that can be quoted

This is the part that actually moves the needle, and it is mostly craft. The pattern that retrieves and cites well looks like this:

• Answer first, then explain. A section that opens with “A 1200×630 image is the safest size for a social card because every major platform crops toward 1.91:1” is quotable from its first sentence. The same information, three paragraphs in, usually is not.
• Name the subject in every paragraph. “It also supports JSON” becomes “Pikkola Diff also supports JSON”. Pronouns are cheap for humans and expensive for retrieval.
• Use a definition sentence. “Merge patch is a JSON format in which null deletes a key.” Definitions are the highest-value chunk type on the internet.
• Put numbers in tables. Tables survive chunking intact and are easy to verify, which makes them attractive to a system that would rather cite than compute.
• Date your claims. “As of November 2026” lets a model decide whether to trust a fact, and gives you a place to update it.
• Cut the marketing adjectives. “Powerful, seamless, blazing-fast” cannot be verified and does not survive a relevance score. “Runs entirely in the browser; no request leaves the tab” can.

Structure beats length

Long documents are not penalised, but unstructured ones are. One idea per section, a heading that says what the section answers, and a short summary at the top of anything longer than a few screens. If a section has two ideas, a chunker will pick one of them and drop the other.

robots.txt as a publishing decision

The three plausible policies, and what each one costs:

Whichever you choose, write the reason in a comment at the top of the file. robots.txt is advisory — a determined crawler can ignore it — but a stated, dated policy is worth more than an undocumented block, and it prevents the file from silently contradicting your sitemap.

Markup that helps, and markup that does not

Structured data does not make a model quote you, but BlogPosting with a real author, datePublished and dateModified makes the page machine-legible in a way that costs ten minutes. FAQPage helps when the page genuinely answers those questions in visible text. BreadcrumbList and Organization establish who is speaking.

What does not help: marking up content that is not on the page, stuffing keywords, or publishing five hundred schema types in the hope that one lands. Structured data that contradicts the visible page is worse than none.

Measuring AI traffic is harder than it looks

Referrer headers from AI surfaces are inconsistent and sometimes stripped, so analytics undercount. What does work, in order of reliability:

1. Server logs. Group by user agent for crawler activity and by referrer for visits. A daily count of OAI-SearchBot, ClaudeBot and PerplexityBot requests is the clearest early signal that you are being indexed at all.
2. Referrer domains in your analytics. Look for chatgpt.com, perplexity.ai, claude.ai, gemini.google.com and copilot.microsoft.com. Treat the numbers as a floor.
3. Direct traffic with a landing page that nobody shares. Unexplained direct visits to a deep article are frequently an answer engine with a stripped referrer.

If you publish a page specifically to be cited, give it a unique URL and watch that one. Aggregate site-wide numbers will not tell you whether the experiment worked.

A thirty-minute audit

• Pick your three most useful pages and read the first sentence of every section. Does each one answer something on its own?
• Replace pronouns with the subject in the opening paragraph of each section.
• Add a date to every claim that can expire.
• Check that headings are questions or statements, not labels like “Overview”.
• Confirm the important content is in the server-rendered HTML, not behind client-side rendering.
• Write llms.txt with a summary, ten links and one sentence each.
• Write a dated robots.txt policy that matches what you actually want.
• Add BlogPosting with author, dates and a real image.
• List the AI referrers in your analytics and check them weekly for a month.

One tool worth adding to the loop: Pikkola Rank scores a draft for length, repetition, risky claims and structural SEO problems before you publish, which is a useful proxy for the same qualities that make a passage citable — specific, dated, and free of padding.

The uncomfortable conclusion

GEO is not a new set of tricks layered over SEO. It is a return to the older discipline of writing a page that answers a question clearly enough that someone else can repeat the answer. Retrieval systems reward exactly that, and they punish vague, self-referential, adjective-heavy prose more harshly than a search ranking ever did, because a model has to actually use the sentence it selects.

Write the version you would want quoted, put it near the top, and make it easy for a machine to find. Then add llms.txt, because it takes ten minutes and because the sites that will benefit if the proposal ever does get adopted are the ones that already have one.