# llms.txt

Each documented package serves its whole API surface as plain Markdown at `<base>/llms.txt`. On
this site, that is
[/starlight-pydocs/api/demopkg/llms.txt](/starlight-pydocs/api/demopkg/llms.txt). The endpoint is
on by default. Set `llmsTxt: false` to remove it from every package.

A single Markdown file suits a language model better than a crawl of rendered HTML. It also makes
a good diff target: the package's own unit tests snapshot this same format, so a change in the
rendered surface shows up as a change here too.

## What it contains

The file starts with a short header that names the package and links to the rendered pages. After
that comes every page of the package, in navigation order. The example below is abridged, from
`demopkg`:

````md {12-18} {20-22} {24-34} {36}
# demopkg

> API reference for the Python package `demopkg`, generated by starlight-pydocs.
> Rendered pages: https://ewels.github.io/starlight-pydocs/api/demopkg/

# demopkg

*module*

Demo package used by the starlight-pydocs test suite.

## demopkg.Report

*class*

```python
class Report(BaseReport)
```

Bases: `demopkg.report.BaseReport`

Re-exported from: `demopkg.report`

A named collection of scored sections.

**Parameters**

- `name` (`str`) — Human readable report name, also used as the file stem.
- `scores` (`dict[str, float] | None`) (default: `None`) — Mapping of metric name to score. Defaults to an empty mapping.

**Attributes**

- `name` — The report name.
- `scores` (`dict[str, float]`) — The scores passed to the constructor.

[View source](https://github.com/ewels/starlight-pydocs/blob/main/fixtures/demopkg/src/demopkg/report.py#L57-L160)
````

Every object follows the same layout, shown as four highlighted runs above. The heading is the
dotted path.
Italics give the kind and any labels. A `python` fence holds the signature. Next comes provenance:
bases, re-exports, alias targets and deprecation notices. After that come the docstring sections.
The source link comes last, after everything the docstring produced and before the object's own
members.

Heading depth follows the rendered page, so a method sits one level below its class. Doctest
blocks are fenced as `pycon`. Docstring prose passes through as written, so
[cross-references](/starlight-pydocs/guides/cross-references/) keep their `[title][dotted.path]`
form. A model can use that form better than a resolved URL.

The file is prerendered, so a static host serves it as `text/plain` regardless of the
`text/markdown` the route sets. Browsers show it, `curl` pipes it, and models read it.

## One page at a time

The same Markdown is served per page, at the page's own path plus `.md`. The page at
`/api/demopkg/report/` is also
[/starlight-pydocs/api/demopkg/report.md](/starlight-pydocs/api/demopkg/report.md), and a
package's root page answers at its base:
[/starlight-pydocs/api/demopkg.md](/starlight-pydocs/api/demopkg.md).

Each one is served again at `.md.txt`, for example
[/starlight-pydocs/api/demopkg/report.md.txt](/starlight-pydocs/api/demopkg/report.md.txt). The
bytes are identical. Only the extension differs, and it is the extension a host reads: `.md.txt`
gets `text/plain`, which browsers and viewers show inline, where a `.md` is often downloaded
instead. Link the `.md` for machines and the `.md.txt` for people.

Use the whole-package file to give a model an entire API in one request. Use the per-page file
when something already holds a single page's URL: an agent handed a link, or a reader who wants
this page and not the other twelve.

Each generated page advertises its own `.md` in the document head, so nothing has to know the
convention to find it:

```html
<link rel="alternate" type="text/markdown" href="/api/demopkg/report.md" />
```

Set `pageMarkdown: false` to remove all of these routes, and that tag with them.

`<path>.md` is also the convention the Starlight page-action plugins fetch, so their buttons work
on the generated pages as well as on the ones you wrote. This site runs
[starlight-page-actions](https://starlight-page-actions.dlcastillop.com/), which is where the
**Copy Markdown** and **Open** buttons under every page title come from:

```ts title="astro.config.ts"
starlight({
  plugins: [
    starlightPydocs({ packages: [{ name: 'demopkg', search: ['../fixtures/demopkg/src'] }] }),
    // Copies `src/content/docs/**` to `<path>.md` and adds the buttons. The
    // generated pages are not files there, but they answer at `<path>.md`
    // themselves, so the same buttons work on both.
    starlightPageActions({ position: 'page-title' }),
  ],
});
```

:::caution
Leave that plugin's `baseUrl` unset if you also run starlight-llms-txt. With it, both plugins
write a site-wide `llms.txt` and the last one to run wins.
:::

## Wiring it into starlight-llms-txt

[starlight-llms-txt](https://delucis.github.io/starlight-llms-txt/) builds a site-wide `llms.txt`
from the `docs` content collection, the Markdown files under `src/content/docs/`. The generated
API pages are not files there. This plugin injects them as routes when Astro builds. Nothing lists
those routes for starlight-llms-txt, so it cannot see any of them. That is why you must tell the
two plugins about each other, using the `optionalLinks` option: it takes a list of URLs directly,
rather than reading pages to find them.

This site runs both, so you can click through the result:

- [/starlight-pydocs/llms.txt](/starlight-pydocs/llms.txt) — the entry point: what this project
  is, links to the two documentation sets below, then the three per-package endpoints at the
  bottom under `## Optional`.
- [/starlight-pydocs/llms-full.txt](/starlight-pydocs/llms-full.txt) — every guide on this site
  concatenated into one Markdown file.
- [/starlight-pydocs/llms-small.txt](/starlight-pydocs/llms-small.txt) — the same with
  non-essential content stripped, for smaller context windows.
- [/starlight-pydocs/api/demopkg/llms.txt](/starlight-pydocs/api/demopkg/llms.txt) — the endpoint
  this plugin serves, which the entry point links to but cannot contain.

The configuration behind that:

```ts title="astro.config.ts" {19-26}
import starlight from '@astrojs/starlight';
import { defineConfig } from 'astro/config';
import starlightLlmsTxt from 'starlight-llms-txt';
import starlightPydocs, { pydocsSidebarGroup } from 'starlight-pydocs';

const SITE = 'https://ewels.github.io/starlight-pydocs';

export default defineConfig({
  site: 'https://ewels.github.io',
  base: '/starlight-pydocs',
  integrations: [
    starlight({
      title: 'Starlight Pydocs',
      plugins: [
        starlightPydocs({
          packages: [{ name: 'demopkg', search: ['../fixtures/demopkg/src'] }],
        }),
        starlightLlmsTxt({
          optionalLinks: [
            {
              label: 'demopkg API reference (Markdown)',
              url: `${SITE}/api/demopkg/llms.txt`,
              description:
                'The example package, rendered by this site: google-style docstrings, pydantic models, deprecations, inheritance and `__all__` filtering.',
            },
          ],
        }),
      ],
      sidebar: [{ label: 'API reference', items: [pydocsSidebarGroup] }],
    }),
  ],
});
```

Each entry takes `label`, `url` and an optional `description`. It lands under the `## Optional`
heading of the generated `llms.txt`, where a model can follow it when it wants more than the
prose. Build the URLs from the site origin, as `SITE` does above, because clients that read this
file do not know your site's base path.

Add one entry per package you want a model to find, and no more. This site documents `demopkg`
twice: once extracted from the working tree, and once from a pre-generated dump at
`1x/api/demopkg` that stands in for an older release. It advertises only the current one, so a
model never lands on an old release when it answers a question.

## Consuming it yourself

The endpoint is a normal prerendered file, so anything can read it:

```sh frame="none"
curl -s https://ewels.github.io/starlight-pydocs/api/demopkg/llms.txt > demopkg-api.md
```

If you want the same information as structured data instead of Markdown, use the
[Content Layer loader](/starlight-pydocs/guides/vanilla-astro/#the-content-layer-loader). It
exposes one entry per documented object, with its path, kind, brief, signature and page. Use it to
build a table, a cheat sheet, or your own index.

## Listing the generated pages

Site-wide features that iterate over pages — share card images, a custom index, an extra sitemap —
build their list from your content collection. The generated pages are injected routes, so they are
not in it. Ask the plugin for them instead:

```ts title="src/pages/og/[...route].ts" {2,5}
import { getCollection } from 'astro:content';
import { listPydocsPages } from 'starlight-pydocs/pages';
import context from 'virtual:starlight-pydocs/context';

const generated = await listPydocsPages(context);
// [{ base: 'api/demopkg', slug: 'api/demopkg/report',
//    title: 'demopkg.report', description: 'Report classes and the …' }, …]

const pages = {
  ...Object.fromEntries((await getCollection('docs')).map((entry) => [entry.id, entry.data])),
  ...Object.fromEntries(generated.map((page) => [page.slug, page])),
};
```

The `slug` is the page's path within your site, with the package base already on the front, so it
works as a route key and as a URL. The `description` is the module's first docstring line, which is
also what the page itself uses for its `<meta name="description">` and OpenGraph description.

This site generates its own share cards this way. The
[`astro-og-canvas`](https://github.com/delucis/astro-og-canvas) route reads both lists, and a
`Head` component override points each page at its image.