Search · Fetch · Render · Map · Crawl
The web, read back as text your agent can use.
One HTTP service in front of search, page extraction and a real browser. Ask for a query and get ranked results with the pages already read into them — not a list of links you then have to go fetch one by one.
The contract
Six routes. Every one of them bounded — a deadline you can set, a ceiling you cannot exceed, and a partial answer instead of a hung request.
Ranked results across categories, fused by reciprocal rank — not by raw score, which buries whichever engine scores lowest. Set content: 3 and the top pages come back already read.
POST/fetchA page's main text, via readability. If the page renders client-side and comes back near-empty, it falls back to a real browser in-process — no second round trip. paginate walks a long article instead of truncating it.
POST/renderRendered HTML after the JavaScript has actually run. Settles on network idle, not on load — the event that fires before the content arrives.
POST/mapEvery URL a site declares or links to, within a scope. Synchronous and bounded. Sitemaps are read first — the cheapest, most complete source a site offers.
POST/crawlAnswers 202 and a job id. A hundred pages is minutes of work, so it runs behind the request rather than under it.
GET/crawl/{id}Progress and the pages read so far, twenty at a time. next is the offset of the following page, or absent when there is none.
Why not self-host
You can run SearXNG yourself in an afternoon. These are the four things that break in month two.
The page is empty
Half the web renders client-side. A static fetch returns the shell, and your extractor happily hands your agent nothing at all.
A browser is a function call awayThe datacenter IP is refused
Bot management refuses a cloud address whatever headers the request carries. The exit IP decides whether the page can be read — and a residential IP alone isn't enough, because Go's TLS handshake matches no browser.
Residential exit and a real fingerprint, on one pathAn engine goes down
An upstream gets rate-limited and your general category quietly returns less, with nothing in the response to say so.
Brave backs the general category, running its own indexOne slow category holds the answer
Academic engines take seconds. Concatenating their scores with the web's buries them; waiting on them hangs the call.
Reciprocal rank, a deadline, and partial: trueThe numbers you'll hit
All of them are ceilings, published up front — so nothing surprises you in production.
- Search deadline
- 4000ms
- Render timeout
- 20s hard
- Pages per crawl
- 100
- URLs per map
- 1000
- Inline page reads
- 10
- Page cache
- 15min
Default. Ask for less, never more than the deployment's own.
8s by default. Clamped regardless of what is asked.
Read behind the request, twenty per page of results.
Default 100. Synchronous, deadline capped at 50s.
Per search. 4000 runes each unless you say otherwise.
Holds full text, so a later, larger read still hits.
Who may call
A service credential, never a browser one.
Present your application key as a bearer token, or a token from your identity provider carrying the tornad:search scope.
Create one key per application from your account, so it gets revoked without touching another's — and every log line can name who called.
/render executes its JavaScript. A credential a browser can hold is one an attacker can spend scanning whatever your deployment can reach.# the whole integration
curl https://tornad.dev/search \
-H "Authorization: Bearer $TORNAD_KEY" \
-H "Content-Type: application/json" \
-d '{"q":"gramsci","content":3}'
# or, inside the cluster, none at all
TORNAD_UNGUARDED=trueThe contract is served, not copied.
GET /openapi.json returns this API's OpenAPI 3 description, generated from the handlers themselves. CI regenerates it and fails on a difference — so a handler cannot change without the contract following it.