Lessons From Building Two Directories: What 59,000 Schools Taught Me About Agent-Ready Websites
I run a curated SaaS directory and a map of every school in Poland. Here's what the second one taught me about data pipelines, robots.txt as policy, and building for AI agents as a third reader.
I run two directories. This one, SaaS Cubes, lists a few dozen software products — around forty as I write this — that founders submit and I review by hand, and a good share of the submissions never complete the backlink step that makes a listing free. The other one, Mapa Oświatowa, an interactive map of Poland's 59,000 educational institutions, lists every nursery, preschool, school and university in the country — and nobody submits anything. The records come from public registries, and my job is to keep them honest.
On paper these are the same product category. In practice they share almost nothing: not the data model, not the business model, not even the hard part. One is starved for supply; the other is drowning in it. One sells to founders who understand backlinks; the other serves parents who have never heard the word.
What the second project taught me, though, applies directly to the first — and to any SaaS with a public website. It forced me to design for a reader I had been quietly ignoring: software. Search engines were always there, but in 2026 the third reader is an AI agent that fetches your page, decides whether it can trust it, and either cites you or moves on. This post is the case study — the data supply chain, the free tools that went wrong, and the agent layer piece by piece — and what I'm carrying back to SaaS Cubes.
Two Directories, Two Opposite Problems
When I started SaaS Cubes, I assumed the hard part of a directory was the software. It isn't. The hard part is the supply. A SaaS directory competes with dozens of others for the attention of founders who have ten minutes to spare, and every listing is a small negotiation: is this product real, is the description honest, does the screenshot match what ships? Curation is the product. The cube grid is just the packaging.
Mapa Oświatowa inverted every one of those assumptions. Supply is public, enormous and legally mandated: the Polish Ministry of Education maintains a registry of every school in the country, and it is open data. Nobody needs to submit anything. The problem is the opposite — 59,000 records that go stale on their own schedule, eight registries that disagree with each other, and readers who will judge the whole site by the one entry they check, which is usually their own child's school.
| Question | SaaS Cubes | Mapa Oświatowa |
|---|---|---|
| Where records come from | Founders submit, I review | Eight public registries, diffed on every import |
| How many | Around 40, and every one earned | 59,437, changing constantly |
| What is hard | Getting supply and keeping quality | Freshness, provenance and findability |
| Who pays | Founders, for a permanent listing | Institutions, for promotion; parents pay nothing |
| Who reads | Founders, buyers, search engines, AI | Parents, search engines, AI |
The business model followed the same inversion. SaaS Cubes charges the people being listed, because a listing is something a founder wants. A school does not want to be listed — it already is, by law. So the map is free for parents and free for institutions, and the paid product is promotion: a highlighted placement for 199 PLN per 30 days (about $54 at the time of writing) or 1,499 PLN per year (about $400), sold through a contact form rather than a checkout, because a Polish school buying anything involves an invoice, a signature and someone in accounting. Different buyer, different funnel, same underlying rule: charge the party that gets the value, not the party that provides the content.
Data Is a Supply Chain, Not a Download
The naive plan was: download the registry, seed the database, done. The registry — RSPO, maintained by the Ministry of Education — does cover schools and preschools. It does not cover nurseries, which live in a separate registry run by a different ministry and have no shared identifier. It does not cover universities, which sit in a third system called RAD-on, run by a research institute, again with no common key. Coordinates are often missing or wrong. Photos do not exist. Opening hours do not exist. And a school that closed in June is still in the file in September, because the registry records the closure on its own timeline.
Every one of those gaps became a source, and every source became a pipeline with its own quirks:
The map is built on OpenStreetMap (ODbL 1.0), on GUS and SIO data (CC BY 4.0) and on Commons photos (CC BY or CC BY-SA, photo by photo). All of those require attribution, and some of them require share-alike. That is not a footer nuisance; it decides what you can and cannot let other machines do with your pages. It shaped the robots.txt policy below more than any SEO consideration did.
The SaaS Cubes lesson is smaller in scale but identical in shape. When a founder submits a product, the AI-assisted flow fetches their landing page and drafts a listing. That draft is derived data with a provenance question attached: what page, fetched when, summarised by which model? I used to treat it as copy. I now treat it as a record with a source, and the editorial description that the directory writes about a product is kept separate from what the founder submitted, for the same reason a school's registry data is kept separate from the text an LLM wrote about it.
The Third Reader
For most of the web's history a page had two readers: a person and a search engine. You wrote for the person and added a little structure for the crawler. In my last post I argued that AI assistants are now a distribution channel for SaaS, and that an llms.txt file is cheap insurance. I still think so. But on the map I went further, because the third reader is no longer hypothetical there — agents fetch institution pages every day, and a parent asking an assistant "which preschools near me have a speech therapist" is exactly the query the site exists to answer.
So the principle became: one source of truth, several shapes, and the client decides which shape it gets. A parent gets HTML. A crawler gets HTML plus JSON-LD. An agent that asks nicely gets Markdown with the structured data attached. A developer gets JSON from a documented API. None of these is a separate copy of the content; they are renderings of the same records.
| Layer | Question it answers | Where it lives |
|---|---|---|
| llms.txt | What is this site, and where do I start? | /llms.txt |
| OpenAPI 3.1 | What can I ask the API, exactly? | /openapi.json |
| API catalog (RFC 9727) | Where are the spec, the docs, the status page? | /.well-known/api-catalog |
| Markdown negotiation | Can I have this page without the chrome? | Accept: text/markdown on four routes |
| Content-Signal in robots.txt | What am I allowed to do with it? | /robots.txt, every group |
| JSON-LD Dataset blocks | Is this data citable, and under what licence? | Four statistics pages |
The human-readable end of that table is the public API docs at mapaoswiatowa.pl/dla-deweloperow: no key, no registration, GET only, 120 requests per minute per IP. The page renders its endpoint list and every closed vocabulary from the same constants file the application uses, so that page cannot drift from the client code. The other copies can, and have: the same vocabularies are also written down in a server constants file, the OpenAPI document and llms.txt, nothing enforces agreement, and the spec is the copy that goes stale first.
The third reader also raises the stakes on being wrong. The map ships a set of free calculators for parents — a linkable asset and good distribution — and one of them carried a table of legal fee limits with an expiry date. When the date passed, the lookup returned null, the calculator read null as "no cap applies", and for five months it told a parent paying 2,400 PLN that they would receive the full 1,500 PLN subsidy. Nothing crashed, nothing logged, and the wrong number sat on a page that ranks for the question — exactly the kind of page an assistant quotes. It now returns an explicit "undetermined" state and renders no amount at all. The rule that came out of it applies to a pricing page as much as to a calculator: every constant is data with a source and an expiry, and a missing value is its own state, never a zero. It is also why our statistics page carries a last-updated date instead of pretending to be timeless.
The Agent Layer, Piece by Piece
None of this is exotic technology. It is a handful of conventions, most of them younger than the site, and the interesting part is the decisions inside each one.
llms.txt without llms-full.txt
The file follows the grammar from llmstxt.org: one H1, a blockquote summary, a short unheaded block, then H2 sections of links. It opens with a "start here" list of five URLs and then describes the institution types, the geography, the data sources and every API filter, including the exact enum strings that return a 400 if you misspell them. It carries a hand-maintained "state of description" date, and it is cached for three days rather than forever, because a stale contract is worse than none. The live file is in Polish, since that is the language of the readers and of the data; here is the opening, translated, with the field names and enum values exactly as they appear:
# Mapa Oświatowa
> The largest interactive database of Polish educational
> institutions — over 59,000 schools, preschools, nurseries
> and universities on one map. Data from the public registries
> RSPO, the MRPiPS nursery registry, RAD-on and OpenStreetMap.
Site: https://mapaoswiatowa.pl
Public API (no key, GET only): https://api.mapaoswiatowa.pl
[... H2 sections: About the Site, Institution Types,
Geography, Data Sources, API ...]
Values of the `type` field in the API
(exactly these strings, upper case):
| `type` | Meaning | URL slug |
|-------------|----------------------------------|---------------|
| `NURSERY` | Nurseries, children's clubs (<3) | `zlobki` |
| `PRESCHOOL` | Preschools, preschool units (3–6)| `przedszkola` |
What the site deliberately does not have is an llms-full.txt. SaaS Cubes has one, and for a few dozen listings it makes sense: the whole directory fits in a context window. For 59,000 pages a full dump would be a second copy of the site that goes stale the moment it is written, and the evidence that retrieval bots read these files at all is thin — llms.txt is fetched mostly by IDE agents, MCP servers and SEO tools. That argues for keeping the file small and current, and pointing to the API as the "full version". The full version is the thing that is always up to date.
OpenAPI plus an API catalog (RFC 9727)
The OpenAPI 3.1 document describes 23 paths. Its version number is a date, because nobody was going to maintain semver on a read-only public API, and a date tells an agent exactly what it needs to know: how old is this description? It is served with a permissive CORS header so that Swagger UI or Redoc can load it from anywhere, and it deliberately omits two things: the binary points feed that draws the map (it carries coordinates and no names, so it is useless to anyone but the map) and every admin, auth and submission route.
The part I had not seen on many sites is the catalog. RFC 9727, published in 2025, defines a well-known location, /.well-known/api-catalog, that returns a linkset (RFC 9264) pointing at the spec, the human docs, a status endpoint and — via a relation from RFC 8631 — a description "written for machine consumption", which is where llms.txt fits. The site advertises the catalog three ways: a link element in the head of every HTML page, a Link header on every API response, and a Link header on the homepage that names all four resources at once:
$ curl -sI https://mapaoswiatowa.pl/ | grep -i ^link
link: </.well-known/api-catalog>; rel="api-catalog"; type="application/linkset+json",
</openapi.json>; rel="service-desc"; type="application/openapi+json",
</dla-deweloperow>; rel="service-doc"; type="text/html",
</llms.txt>; rel="describedby"; type="text/plain"
That header is on the homepage only. The first version put it on every route, until I did the arithmetic: 250 bytes on 59,000 institution pages, on every request, for an agent that only needs to find the catalog once. The head element covers every page; the header covers the one place a client is likely to start.
Markdown on request
This is the piece I would build first if I were starting over. Cloudflare published a convention in 2026 under the name "Markdown for Agents": if a client sends Accept: text/markdown, a site may answer with a Markdown rendering of the page instead of HTML, plus a header estimating how many tokens the body costs. The map implements it on four routes — the homepage, institution pages, city pages and blog posts:
$ curl -sI -H 'Accept: text/markdown' https://mapaoswiatowa.pl/
HTTP/2 200
content-type: text/markdown; charset=utf-8
cache-control: no-store
link: <https://mapaoswiatowa.pl/>; rel="canonical"
vary: Accept
x-markdown-tokens: 1092
x-robots-tag: noindex
The document an agent gets is YAML front matter, the page body, and the same JSON-LD the HTML page carries, in a fenced block at the end. The mechanics are less obvious than the spec makes them sound. Next.js caches rendered routes by URL, not by the Accept header, so serving Markdown from the page's own address risks handing a cached Markdown body to the next browser. The site therefore rewrites the request in middleware to an internal route that returns 404 unless an internal header is present — there is one public URL per page, and the Markdown variant is not a second address anyone can link to. That is also why the response says noindex and Vary: Accept.
The genuinely hard part was the blog. The posts are MDX with about two dozen custom components, and most of those components carry their content in attributes, not in children — a comparison table row is a pipe-separated string, a checklist item has a checked prop. A regex over the source keeps the five components that look like HTML and silently drops the nineteen that don't. The converter therefore walks the MDX syntax tree and has a handler per component: checklists become - [x] lists, comparison tables become GFM tables, FAQ items become heading-plus-paragraph. Tedious, and the only version that doesn't lose content.
All of the above passes an external checker (isitagentready.com scans a domain for these conventions). Passing it proves the site is legible to agents. It does not prove agents read any of it — that evidence arrives slowly, in server logs, and you should hold every "agent-ready" claim, including this one, to that standard.
robots.txt Is a Policy Document Now
For twenty years robots.txt was two lines you copied from somewhere. On the map it is a hand-written file, because the framework's built-in generator only knows the classic directives and cannot emit the one that matters most now: Content-Signal, a convention proposed by Cloudflare in 2025 that lets a site say, separately, whether its content may be used for search, for AI answers, and for AI training.
User-Agent: GPTBot
User-Agent: ChatGPT-User
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Disallow: /admin
Disallow: /zgloszenie/
Disallow: /udostepnij/
Disallow: /mapa?
Three decisions are hiding in those nine lines.
The signal is repeated in every group. RFC 9309, the robots standard, says a crawler obeys exactly one group — the most specific one that names it. A directive written only under User-Agent: * is invisible to GPTBot, ClaudeBot, PerplexityBot and Google-Extended — the crawlers you most want to let in — because each of them has its own group and stops reading there. So the same line appears six times.
Training is a no, answers are a yes. This is the licence question from earlier coming back. Data under ODbL and CC BY carries attribution duties that can survive a search result or a cited AI answer, and cannot survive absorption into a training corpus. ai-input=yes, ai-train=no is the only setting consistent with the upstream licences. It is a legal position, not an SEO one.
Agents get a tighter map than search engines. The AI groups are kept off the comparison tool, the share page and the map's query URLs. That came from reading logs, not from theory: on one day roughly 39% of AI-crawler requests landed on utility pages that already carried noindex and had nothing an assistant could cite. Every one of those fetches was crawl budget not spent on an institution page.
Content-Signal is a Cloudflare-initiated convention, not an IETF standard, and honouring it is up to each crawler. Writing the line is cheap and correct; it is not enforcement. If a bot ignores it, the remedy is at the network edge, not in the text file.
What Carries Over to SaaS Cubes, and What Doesn't
It is tempting to conclude that everything above should be ported wholesale to a SaaS directory. Some of it should. Some of it is solving a scale problem that a few dozen listings do not have.
Build one source of truth and let humans, search engines and agents negotiate the shape they want. Write down your licence obligations before you write your robots.txt, because they decide what you can allow. And treat every constant — a fee cap, a plan price, a category slug — as data with a source and an expiry, because the number that fails silently is the one that ends up in an AI answer with your name on it.
The map is the messier, larger, less profitable of my two directories, and it is the one that taught me the most. Nothing about "agent-ready" required a new framework or a budget. It required deciding, once, that software is a reader with rights and limits — and then writing those down where software looks.
If your SaaS should be one of the things agents find, a SaaS Cubes listing is hand-curated, permanent, carries a do-follow backlink, and ships in our llms.txt and llms-full.txt feeds — readable by any machine that looks. See pricing and get listed →
Sources
- [1]The /llms.txt file — a proposal to standardise using an /llms.txt file to provide information to help LLMs use a website — Answer.AI / Jeremy Howard
- [2]RFC 9727: api-catalog: A Well-Known URI and Link Relation to Help Discovery of APIs — IETF, 2025
- [3]RFC 9264: Linkset: Media Types and a Link Relation Type for Link Sets — IETF, 2022
- [4]RFC 9309: Robots Exclusion Protocol — IETF, 2022
- [5]Markdown for Agents — Cloudflare Docs
- [6]Markdown for Agents: serving Markdown to AI agents via content negotiation — Cloudflare Blog
- [7]Content Signals Policy — Cloudflare
- [8]Is It Agent Ready? — scan a domain for agent-readiness conventions — isitagentready.com
- [9]OpenAPI Specification v3.1.1 — OpenAPI Initiative
- [10]Dataset structured data — Google Search Central
- [11]Mapa Oświatowa — llms.txt — Mapa Oświatowa
- [12]Publiczne API Mapy Oświatowej (developer documentation) — Mapa Oświatowa
- [13]RSPO — Rejestr Szkół i Placówek Oświatowych — Ministry of National Education (Poland)
- [14]RAD-on — Reports, Analyses, Data on Polish higher education and science — OPI PIB
- [15]Open Database License (ODbL) v1.0 — Open Data Commons
Mateusz Pawlica
With over 12 years of experience building digital products — from mobile apps to AI-powered web platforms — Mateusz specializes in creating modern web applications and implementing AI automation for businesses. He has shipped 20+ projects across SaaS, e-commerce, and education, including Mapa Oświatowa, a map of every school in Poland.
