# Getting Started ::warning **Pre-1.0.** The public API, the provider list and the CLI flags can still change. Pin exact versions if you build on it now. :: ::caution **Search results and page content are data, never instructions.** Titles, snippets, highlights and Markdown come from sites anyone can publish. Show them, index them, summarize them. Do not let an agent read them as a message to itself, seriously. :: ## Why this exists Every search API has its own idea of a request. Exa wants a POST with `x-api-key`, Brave a GET with `X-Subscription-Token`, Jina a Bearer token, Tavily puts the key in the request body. They all give back roughly the same thing, a list of URLs with titles and snippets, just wrapped twelve different ways. Pull one client per engine into an agent and you have twelve response shapes and twelve ways to misread a date. So `@agntn/web` puts twelve providers behind one class shape and three capabilities: **search** (query → results), **search image** (public image URL → matching pages) and **read** (URL → normalized page content). Same calls, same `SearchResult`, `ImageSearchResult` and `ReadResult` everywhere. The adapter decides how to ask, you decide what to ask for. ## Install ```bash pnpm add @agntn/web ``` The AI SDK tools on the `/ai` subpath need `ai` and `zod` next to them: ```bash pnpm add ai zod ``` ## First search ```ts [search.ts] import { create } from "@agntn/web"; // Reads EXA_API_KEY from process.env and imports the Exa adapter const exa = await create("exa"); const results = await exa.search("typescript runtime benchmarks", { maxResults: 5 }); for (const result of results) { console.log(result.title, result.url); } ``` Importing `@agntn/web` loads none of the twelve built-in providers. `create("brave")`, `create("tavily")` or `create("searxng")` imports that one adapter on the first call and resolves the class, no switch statement anywhere, which is why `create()` is async. The key comes from the provider's env var unless you pass `apiKey` yourself: ```ts const exa = await create("exa", { apiKey: "your-key-here" }); const searx = await create("searxng", { baseURL: "https://searx.example.com" }); ``` ## Same call, any provider ```ts const brave = await create("brave"); // BRAVE_API_KEY const context = await create("context"); // CONTEXT_DEV_API_KEY const jina = await create("jina"); // JINA_API_KEY const mojeek = await create("mojeek"); // MOJEEK_API_KEY const tavily = await create("tavily"); // TAVILY_API_KEY const tinyfish = await create("tinyfish"); // TINYFISH_API_KEY await brave.search("query"); // the same call on every one of them ``` The [Explorer](https://web.agntn.dev/explorer) runs exactly these calls against the docs worker, so you can see what comes back before you write a line. Codex search reuses existing Pi, OMP, Codex and OpenCode OAuth logins rather than an API key. See [OpenAI Codex](https://web.agntn.dev/providers/openai-codex) for login discovery, refresh ownership and explicit overrides. ## What ships | Provider | Env var | Search | Image | Read | | ------------ | ---------------------------------------------------- | ------ | ----- | ----------------- | | Brave | `BRAVE_API_KEY` | yes | | | | Context.dev | `CONTEXT_DEV_API_KEY` | yes | | yes | | Exa | `EXA_API_KEY` | yes | | | | Firecrawl | `FIRECRAWL_API_KEY` | yes | | yes | | Jina | `JINA_API_KEY` | yes | | yes, key optional | | Mojeek | `MOJEEK_API_KEY` | yes | | | | OpenAI Codex | Existing login; optional `OPENAI_CODEX_ACCESS_TOKEN` | yes | | | | SearXNG | none, self-hosted | yes | | | | SerpAPI | `SERPAPI_API_KEY` | yes | yes | | | SerpBase | `SERPBASE_API_KEY` | yes | | | | Tavily | `TAVILY_API_KEY` | yes | | yes | | TinyFish | `TINYFISH_API_KEY` | yes | | yes | Each one has a page under [Providers](https://web.agntn.dev/providers) with the endpoints, the filters it honours, the fields it fills and the traps. The traps are the useful part. ## The three capabilities ```ts import { readUrl, searchAll, searchByImage } from "@agntn/web"; const results = await searchAll("latest node.js release"); // every configured provider, deduplicated const matches = await searchByImage("https://example.com/image.jpg", { provider: "serpapi" }); const page = await readUrl("https://example.com/article", { format: "markdown", maxChars: 20_000 }); ``` [Searching](https://web.agntn.dev/guide/search) covers the call to one provider and its options. [Fan-out](https://web.agntn.dev/guide/fanout) is the parallel query, fallback and pagination. [Reading](https://web.agntn.dev/guide/read) is URL to content. [Reverse image search](https://web.agntn.dev/guide/image) is the image lookup. ## One result shape ```ts interface SearchResult { url: string; title: string; snippet: string; score?: number; publishedDate?: string; author?: string; image?: string; favicon?: string; text?: string; // full page text when requested or provided highlights?: string[]; // passages relevant to the query summary?: string; // generated summary when requested metadata?: Record; // whatever else the provider knew } ``` `url`, `title` and `snippet` are always there. The optional fields depend on what the engine exposes, each provider page says which ones it fills. `metadata` is the escape hatch for the rest, and yes, it is a bag. ## Errors ```ts import { AuthError, HTTPError, RateLimitError, UnknownProviderError } from "@agntn/web"; try { await provider.search("query"); } catch (error) { if (error instanceof AuthError) { // missing or invalid API key, error.provider says which } if (error instanceof RateLimitError) { console.log(`Retry after ${error.retryAfter}s`); } if (error instanceof UnknownProviderError) { // provider name not registered } } ``` A 401 normally becomes `AuthError`. Spent credits are the exception, and every provider spells them differently: Context.dev's `USAGE_EXCEEDED` on a 401, Tavily's 432 and 433 usage limits, SerpBase's `status: 1020` inside a 200. All of them become `PaymentError`, an `HTTPError` subclass that preserves the original status, redacted URL and body and allows automatic fallback. A 429 becomes `RateLimitError` with `retryAfter`; other failures use `HTTPError` or `WebError`. No response at all, DNS failure, refused connection or a timeout, is `HTTPError` with `statusCode` 0. `body` names the transport cause and `cause` keeps the original error. All of them extend `WebError`, one `instanceof` catches the lot. ## Next - [Searching](https://web.agntn.dev/guide/search): options, filters, content controls and pagination. - [Fan-out](https://web.agntn.dev/guide/fanout): every provider at once, and automatic fallback. - [Reading](https://web.agntn.dev/guide/read): pages into Markdown with an exact output bound. - [CLI](https://web.agntn.dev/guide/cli): the same calls from a shell. - [Agents](https://web.agntn.dev/guide/agents): AI SDK tools, the MCP server, Pi and OMP. # Searching ## The contract ```ts interface SearchProvider { search(query: string, options?: SearchRequestOptions): Promise; } ``` Every search adapter has this one method. `create(name)` imports that adapter on its first call and gives you the provider, `isSearchProvider(provider)` is the type guard when you hold something you did not create yourself. ```ts import { create, isSearchProvider } from "@agntn/web"; const provider = await create("brave"); if (isSearchProvider(provider)) { const results = await provider.search("typescript runtimes", { maxResults: 5 }); } ``` ## Options ```ts interface SearchRequestOptions extends ExecutionOptions { maxResults?: number; // default 10 highlights?: boolean; // default true summary?: boolean; // default false fullText?: boolean; // default false includeDomains?: readonly string[]; excludeDomains?: readonly string[]; sources?: readonly string[]; categories?: readonly string[]; startPublishedDate?: string; endPublishedDate?: string; category?: string; } ``` `maxResults` goes to the provider as the requested count and is applied again to what comes back, so an engine that pages by ten still gives you five. `highlights` asks for passages relevant to the query where the engine has them (Exa, Firecrawl). `summary` asks Exa for a summary of each result or Tavily and OpenAI Codex for an answer to the query. `fullText` asks for the whole page text, which is a lot of tokens, so think before you turn it on. The filters are per provider. A provider that cannot do one does not fail, it just tells you through the detailed helpers: | Provider | Domain filters | Source values | Category values | Date bounds | | ------------ | ---------------- | ----------------------- | -------------------------------------------- | ----------- | | Brave | none | none | none | start, end | | Context.dev | include, exclude | none | none | none | | Exa | include, exclude | none | forwarded as given | start, end | | Firecrawl | include, exclude | `web`, `news`, `images` | `research`, `pdf`, `developer` | none | | Jina | include | none | `web`, `images`, `news` | none | | Mojeek | include, exclude | none | none | start, end | | OpenAI Codex | none | none | none | none | | SearXNG | none | none | forwarded as given | none | | SerpAPI | none | none | none | none | | SerpBase | none | none | `image`, `images`, `news`, `video`, `videos` | none | | Tavily | include, exclude | none | `general`, `news`, `finance` | start, end | | TinyFish | include, exclude | none | `news`, `research_paper` | start, end | `getSearchFilterCapabilities(name)` gives you the same table for one provider at runtime. ## Diagnostics `search()` is a list API. When you want to know what the provider actually did with your options, ask for the detailed answer: ```ts import { searchProviderDetailed } from "@agntn/web"; const answer = await searchProviderDetailed("brave", "typescript runtimes", { maxResults: 10, includeDomains: ["github.com"], }); answer.provider; // "brave" answer.results; // SearchResult[] answer.ignoredFilters; // ["includeDomains"], Brave has no domain filter answer.undeclaredFilters; // [] on built-ins, custom providers without capability metadata land here answer.pagination; // { status: "next", continuation: "…" } answer.metadata; // metadata for the whole response when the provider has it ``` Silently dropping a filter is the worst thing a wrapper can do, you end up trusting results that were never filtered. So the ignored ones are always listed. Firecrawl puts its request id, warnings and credits used in `metadata`, Tavily puts the generated answer in `metadata.answer`. `isDetailedSearchProvider(provider)` tells you whether a provider exposes `searchDetailed()` directly. The detailed helpers also take `favicon: false`, which drops the favicon URL from every result and every fan-out evidence record before it reaches you. Brave's run about 280 bytes each and SerpAPI's about 400, so the agent surfaces pass it unless the model asks. `search()` on a provider has no such switch, it hands over what the engine mapped. ## Pagination Brave, Mojeek, SearXNG, SerpAPI, SerpBase and TinyFish have deeper pages. The detailed helpers normalize the state: ```ts type SearchPagination = | { status: "next"; continuation: string } // the provider said there is more | { status: "unknown"; continuation: string } // another page must be probed | { status: "end" } // no further page | { status: "unsupported" }; // the provider does not page ``` ```ts const first = await searchProviderDetailed("brave", "typescript runtimes", { maxResults: 10 }); if (first.pagination.status === "next") { const second = await searchProviderDetailed("brave", "typescript runtimes", { maxResults: 10, continuation: first.pagination.continuation, }); } ``` The token is opaque and bound to the provider, the query and every option that changes the page. Change any of them and you get `InvalidSearchContinuationError` before another request leaves. But opaque does not mean signed. A token you did not produce yourself is untrusted input, treat it like one. Tokens are at most `MAX_SEARCH_CONTINUATION_LENGTH` (4 096) characters, the provider's own state inside at most `MAX_PROVIDER_SEARCH_CONTINUATION_LENGTH` (2 048). ## Execution controls ```ts interface ExecutionOptions { signal?: AbortSignal; deadline?: number; // absolute Unix timestamp in milliseconds concurrency?: number; // batch and fan-out only; default 3, maximum 10 } ``` Every network call takes them. `signal` cancels the request in flight. `deadline` is one budget for the whole operation, fallback and fan-out included, so waiting on a slow first provider does not reset the clock for the second. `concurrency` bounds how many requests the batch and fan-out helpers start at once. None of them change a continuation token. When the deadline cuts a fan-out, `searchAllDetailed` returns the providers that finished and lists the rest in `errors`, a batch does the same per item, and a cancelled `signal` still rejects. `deadlineAfterSeconds(30)` turns a budget into that timestamp, the agent tools use it for `timeoutSeconds`. ## Errors | Error | When | | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `EmptyQueryError` | the query is blank | | `UnknownProviderError` | the name is not registered | | `SearchNotSupportedError` | the provider has no `search()` | | `InvalidDateFilterError` | a date bound is not an ISO date, or the end is before the start | | `InvalidSearchContinuationError` | the token does not belong to this provider, query and options | | `AuthError` | the provider rejected the key with 401, or the key is missing | | `PaymentError` | spent credits, however the provider spells them: Context.dev 401 with `USAGE_EXCEEDED`, Tavily 432 or 433, SerpBase `status: 1020`. Automatic fallback keeps going | | `RateLimitError` | the provider answered 429, `retryAfter` in seconds | | `HTTPError` | any other non-2xx, with `statusCode`, redacted `url` and `body`; no response at all is `statusCode` 0, `body` names the transport cause and `cause` keeps the original error | `normalizeError(error, provider)` is what the adapters use to map a raw HTTP failure onto this list. It is exported, custom providers should use it too. # Fan-out ## Ask them all ```ts import { searchAll } from "@agntn/web"; // Detects providers from env vars, queries them in parallel const results = await searchAll("latest node.js release"); for (const result of results) { console.log(`[${result.providers.join(", ")}]`, result.title, result.url); } ``` `searchAll` runs every configured search provider through `Promise.allSettled`, so one engine failing does not empty the list. Results are deduplicated by normalized URL (fragment removed, trailing slashes trimmed, `utm_*` junk dropped, the rest sorted) and capped by `maxResults`, ten by default. ```ts interface SearchAllResult extends SearchResult { provider: string; // the representative record's provider providers: string[]; // every provider that returned this URL, in requested order evidence: SearchAllEvidence[]; // each provider's complete record, scores untouched } ``` The first provider in requested order gives the representative `title` and `snippet`. `evidence` keeps every record, so you can compare what Brave and Exa said about the same page. What the library does not do is merge their scores. A Brave score and an Exa score are not on the same scale and pretending otherwise would be dumb. Pick the providers yourself when env has more keys than you want to spend: ```ts const results = await searchAll("query", { providers: ["exa", "brave"], maxResults: 5 }); ``` ## The detailed answer ```ts import { searchAllDetailed } from "@agntn/web"; const answer = await searchAllDetailed("query", { concurrency: 3 }); answer.results; // SearchAllResult[] answer.successfulProviders; // every provider that answered, even with nothing left after deduplication answer.errors; // [{ provider, error }] for the ones that failed or ran past the deadline answer.filterReports; // ignored and undeclared filters, per provider answer.providerPagination; // [{ provider, pagination }], one independent state per provider answer.providerMetadata; // [{ provider, metadata }] when a provider returned response metadata ``` Fan-out has no single cursor, and it cannot have one. Take a `next` or `unknown` token from `providerPagination` and continue it with `searchProviderDetailed` and that provider. `searchAllDetailed` rejects a `continuation` of its own. ## One query, automatic fallback When you want one answer and do not care which engine gives it: ```ts import { searchWithFallback } from "@agntn/web"; const answer = await searchWithFallback("typescript runtimes", { maxResults: 10 }); answer.provider; // the provider that answered answer.attempts; // ["exa", "brave"], everything tried, in order answer.failures; // [{ provider: "exa", error: "HTTP 402: …" }] ``` The library starts with the first configured provider in detection order (Exa, Brave, Context.dev, Firecrawl, Jina, Tavily, TinyFish, SerpAPI, SerpBase, Mojeek, OpenAI Codex, then custom ones) and moves on after a payment, rate limit, timeout, connection or server failure. Auth failures and invalid requests stop the chain right there. A bad key is your configuration problem, not a reason to burn the next provider's quota. When every eligible provider fails you get `ProviderFallbackError` with the full `attempts` and `failures` and the last error as `cause`. A continuation from a fallback answer pins later calls to the provider inside the token. Two pages never come from two engines. ## Batches ```ts import { readBatch, searchBatch } from "@agntn/web"; const searches = await searchBatch(["TypeScript 7", "Node.js releases"], { provider: "exa" }); const pages = await readBatch(["https://example.com/one", "https://example.com/two"]); ``` Up to `MAX_BATCH_ITEMS` (10) inputs, `DEFAULT_CONCURRENCY` (3) at a time, `MAX_CONCURRENCY` (10) at most. Order is preserved and one failure does not throw away the others. Each search item is the detailed answer for its query, or `{ query, error, attempts, failures }` when automatic selection ran out of providers. Without an explicit provider each query falls back on its own. With `provider: "all"` every query fans out and one scheduler bounds the outer queries and the inner providers together, otherwise ten queries times nine providers would be ninety requests in flight. Each of those items also carries `errors` for the providers that failed or ran past the deadline. A single `continuation` is rejected for a batch, it cannot belong to several queries. ## Configured, reachable, default ```ts import { detectAvailableProviders, detectAvailableProvidersAsync, listProviders, listProvidersAsync, resolveDefaultProvider, resolveDefaultProviderAsync, } from "@agntn/web"; detectAvailableProviders(); // ["exa", "brave", …] from env vars, no network await detectAvailableProvidersAsync(); // the same, minus providers whose isAvailable() probe fails resolveDefaultProvider(); // the first of them, or NoProviderConfiguredError await resolveDefaultProviderAsync(); // the first reachable one, or NoProviderAvailableError ``` `listProviders()` returns every registered provider with `configured`, `envVar` and the complete `capabilities` matrix. The async variant adds `reachable` for providers with a probe. SearXNG is the only built-in with one, because a self-hosted instance on `localhost:8080` is configured by definition and down half the time. # Reading ## Read a URL ```ts import { readUrl } from "@agntn/web"; const page = await readUrl("https://example.com/article", { provider: "jina", format: "markdown", maxChars: 20_000, }); page.title; page.content; // Markdown, at most 20 000 code points page.truncated; // true when more remained page.continuation; // opaque token for the next slice ``` ```ts interface ReadResult { url: string; title?: string; description?: string; content: string; text?: string; html?: string; publishedDate?: string; image?: string; links?: string[]; images?: string[]; metadata?: Record; truncated?: boolean; continuation?: string; } ``` ## Readers | Provider | Formats | Native options | Key | | ----------- | -------------------- | ------------------------------------------------------------------------------- | --------------------------------------------------- | | Jina | markdown, text, html | `format`, `maxTokens`, `targetSelector`, `removeSelector`, `timeout`, `noCache` | optional, sent as Bearer when `JINA_API_KEY` is set | | Context.dev | markdown, html | `format`, `targetSelector`, `removeSelector`, `timeout`, `noCache` | `CONTEXT_DEV_API_KEY` | | Firecrawl | markdown, html | `format`, `targetSelector`, `removeSelector`, `timeout`, `noCache` | `FIRECRAWL_API_KEY` | | TinyFish | markdown, html | `format`, `targetSelector`, `removeSelector`, `timeout`, `noCache` | `TINYFISH_API_KEY` | | Tavily | markdown, text | `format`, `timeout` | `TAVILY_API_KEY` | `readProviders()` lists them at runtime, custom readers included, `isReadProvider(provider)` is the type guard. Firecrawl rejects `maxTokens` instead of ignoring it. You asked for a token bound and got an unbounded page, that is a lie, not a feature. ## Automatic selection Without a `provider`, `readUrl` starts with Jina Reader, which needs no key for a basic read, and tries the other configured readers after a payment, rate limit, timeout, connection or server failure. Jina's HTTP 409 counts too, it just means the page could not be fetched right now. Auth failures and invalid requests stop the chain. When the reader matters, ask for the detailed answer: ```ts import { readUrlDetailed } from "@agntn/web"; const { result, requestedProvider, provider, attempts, failures } = await readUrlDetailed("https://example.com/article"); requestedProvider; // "auto" provider; // "firecrawl", the reader that answered attempts; // ["jina", "firecrawl"] failures; // [{ provider: "jina", error: "HTTP 402: …" }] ``` `ProviderFallbackError` is thrown when every eligible reader fails, with the same `attempts` and `failures` on it. ## The output bound `maxChars` is counted in Unicode code points after the provider answers, so it means the same thing on every reader and for every script. Token counts do not, that is why this is not `maxTokens`. When more remains, `truncated` is `true` and `continuation` carries an opaque token. Pass it back with the same URL and the same native options: ```ts const first = await readUrl(url, { maxChars: 20_000 }); if (first.truncated) { const second = await readUrl(url, { maxChars: 20_000, continuation: first.continuation }); } ``` The token is pinned to the reader that answered and to a fingerprint of the content. Page changed between slices, you get `StaleReadContinuationError`. URL or options differ, `InvalidReadContinuationError`. A paginated slice also drops the provider's `text` and `html` duplicates, otherwise they would smuggle the whole page past the bound. `links` and `images` stay out of a bounded read too unless you pass `links: true` or `images: true`: a Wikipedia article through TinyFish is 20 000 characters of content next to 138 kB of links, and a bound that lets that through isn't one. The library and the CLI are unbounded unless `maxChars` is set. The agent surfaces default to `DEFAULT_AGENT_READ_MAX_CHARS` (20 000) and accept at most `MAX_AGENT_READ_CHARS` (200 000). `maxTokens` is the provider's own request option, not an approximation of this bound. ## Batches ```ts import { readBatch, readBatchDetailed } from "@agntn/web"; const pages = await readBatch(["https://example.com/one", "https://example.com/two"], { maxChars: 8000 }); // [{ url, result }, { url, error }] const detailed = await readBatchDetailed(["https://example.com/one"]); // [{ url, result, requestedProvider, provider, attempts, failures }] ``` Up to ten URLs, three at a time by default, order preserved, one failure does not throw away the others. A `continuation` is rejected for a batch. ::caution Page content is the page's author talking, not the library. Render it as text, keep it out of `v-html` , and do not let an agent take a sentence in it as an instruction. :: # Reverse Image Search ## Search by image ```ts import { searchByImage } from "@agntn/web"; const matches = await searchByImage("https://example.com/image.jpg", { provider: "serpapi", maxResults: 5, }); for (const match of matches) { console.log(match.pageUrl, match.imageUrl, match.imageWidth, match.imageHeight); } ``` This is its own capability, not a text search with a URL stuffed into the query. A provider without image lookup gets `ImageSearchNotSupportedError`, not a fake query it would answer with garbage. `searchImageProviders()` lists the ones that have it. Right now that is SerpAPI, which goes through Google Lens. ```ts interface ImageSearchResult { pageUrl: string; // the page that shows the image imageUrl: string; // the matched image title: string; provider: string; source?: string; thumbnailUrl?: string; imageWidth?: number; imageHeight?: number; thumbnailWidth?: number; thumbnailHeight?: number; position?: number; exactMatch?: boolean; } ``` Page and image stay separate on purpose. A match says where the image appears and which file matched, and never pretends those are the same thing. ## What the URL must be The image URL goes to the provider as is. So it has to be public, absolute, `http` or `https`, and it should not carry credentials or private query tokens, because the provider will fetch it and you just handed them the token. `EmptyImageUrlError` and `InvalidImageUrlError` cover the blank and malformed cases before any request leaves. ## What stays out Uploads, hosting, OCR, embeddings, perceptual hashes, local image analysis. None of it is here. Lookup by URL fits because it is a thin adapter over a search API. The rest drags in heavy dependencies and belongs in an image or vision package of its own. # CLI ## Run it ```bash pnpm add -g @agntn/web web "your query" ``` `web ` searches with the first configured provider. The subcommands do the rest: | Command | Description | | ------------------------ | ------------------------------------------------ | | `web ` | Search the web using the default provider | | `web search ` | Search one or more queries | | `web search-image ` | Find matching pages from a public image URL | | `web read ` | Read one or more URLs into normalized content | | `web providers` | List registered providers and their capabilities | | `web mcp` | Run the MCP server over stdio | ## Examples ```bash web --provider brave "your query" --max-results 5 web search "your query" --json web search "first query" "second query" --provider all --json web search "your query" --provider firecrawl --sources web,news --categories research web search "your query" --provider exa --summary --full-text web search "your query" --provider brave --continuation --json web search "your query" --include-domains github.com,stackoverflow.com --start-published-date 2026-01-01 web search-image https://example.com/image.jpg --max-results 5 --json web read https://example.com --format markdown --max-chars 20000 --json web read https://example.com --max-chars 20000 --continuation --json web read https://example.com/one https://example.com/two --json web providers --json ``` ## Flags | Flag | Description | | ----------------------------------- | ------------------------------------------------------------------------------------------------ | | `--provider ` | Provider to use (text: first configured or `all`; image: SerpAPI; read: auto starting with Jina) | | `--max-results ` | Maximum text or image results (default: `10`) | | `--no-highlights` | Disable the passages picked for the query when supported | | `--summary` | Request generated summaries from Exa or an answer from Tavily | | `--full-text` | Request full page text from Exa or Tavily | | `--include-domains ` | Include only these domains in text search | | `--exclude-domains ` | Exclude these domains from text search | | `--sources ` | Source types for providers that support them | | `--categories ` | Categories for providers that support them | | `--category ` | One provider category | | `--start-published-date ` | Earliest publication date | | `--end-published-date ` | Latest publication date | | `--format ` | Preferred read format | | `--max-tokens ` | The provider's own maximum of read tokens when supported | | `--max-chars ` | Portable maximum of page content characters | | `--continuation ` | Continue a search page or a truncated read of one URL | | `--links` | Keep the page's links in a read bounded by `--max-chars` | | `--images` | Keep the page's image URLs in a read bounded by `--max-chars` | | `--json` | Output as JSON | Lists are comma separated. Read commands pick the reader automatically unless `--provider` is set. ## JSON envelopes `--json` prints the same shapes the library and the agent tools return. One parser for all three, that was the point. - A single search prints `{ provider, results, ignoredFilters, undeclaredFilters, pagination, metadata? }`, plus `attempts` and `failures` when the provider was picked automatically. - `--provider all` prints `{ results, successfulProviders, errors, filterReports, providerPagination, providerMetadata? }`, with `providers` and `evidence` on every result. Provider errors are part of the output, not a reason to exit. - A batch of queries prints one item per query, each with its own result or error. - A single read prints `{ result, requestedProvider, provider, attempts, failures }`. A batch adds `url` to each success and keeps the same diagnostics on each exhausted failure. Any failed read item makes the command exit 1, but the successes are still printed. A failure that ends the command prints one line on stderr and exits 1, whether it is a spent key, a bad token or a date that is not ISO 8601. Nothing lands on stdout, `--json` included. ## MCP ```bash web mcp ``` starts a [Model Context Protocol](https://modelcontextprotocol.io){rel=""nofollow""} server over stdio with `web_search`, `web_search_image`, `web_read` and `web_providers`. Register it with any client: ```bash claude mcp add web --scope user -- web mcp ``` [Agents](https://web.agntn.dev/guide/agents) covers what the tools accept and return. # Agents ## Four tools, every host | Tool | Does | Returns | | ------------------ | --------------------------------------------------------------------------------- | ------------------------------------------------------------- | | `web_search` | one query or a batch, one provider, automatic selection, or `all` for fan-out | results with the provider diagnostics | | `web_search_image` | matching pages and images for one public image URL | `ImageSearchResult[]` | | `web_read` | one URL or a batch into normalized content, with reader provenance | `{ result, requestedProvider, provider, attempts, failures }` | | `web_providers` | the running build, configuration, reachability and the complete capability matrix | `listProviders()` plus `packageCapabilities` | Same four on every surface. The descriptions name the built-in providers, but execution checks the name against the live registry, so a custom provider registered before the session works in a tool call too. The host's abort signal cancels the provider request, no zombie fetches after the model gives up. `timeoutSeconds` on `web_search` and `web_read` is the budget the model sets itself: a fan-out or batch returns what finished by then and lists the providers it cut off under `errors`, so one slow provider no longer holds an `all` search past the host's own limit. ## AI SDK ```ts import { generateText } from "ai"; import { readTool, searchImageTool, searchTool } from "@agntn/web/ai"; const { text } = await generateText({ model: yourModel, tools: { web_search: searchTool, web_search_image: searchImageTool, web_read: readTool, }, prompt: "Find the latest TypeScript release notes", }); ``` `ai` and `zod` are optional peers. The main entry never imports them, only the `/ai` subpath does. `providersTool` is the fourth export. ```ts type SearchToolInput = { query: string | string[]; // up to 10 in a batch provider?: string; // a built-in, a custom name, or "all" maxResults?: number; // 1 to 20, default 10 continuation?: string; highlights?: boolean; summary?: boolean; fullText?: boolean; favicon?: boolean; // favicon URLs on results, off by default includeDomains?: string[]; excludeDomains?: string[]; sources?: string[]; categories?: string[]; category?: string; startPublishedDate?: string; endPublishedDate?: string; timeoutSeconds?: number; // whole seconds, at most 3600 }; ``` A scalar search returns `{ provider, results, ignoredFilters, undeclaredFilters, pagination, metadata? }`, automatic ones add `attempts` and `failures`, and `provider: "all"` returns the fan-out envelope with `providers` and `evidence` on each result. `readTool` takes `url` or an array of URLs plus the read options, defaults to 20 000 characters and accepts at most 200 000. Without that default a model would happily ask for a whole site and stuff it into its context. The page's `links` and `images` are off unless the model asks for them, a long page carries thousands and they'd outweigh the content several times over. The `favicon` URLs on search results are off the same way: they were a third of a ten result Brave answer and a model has nothing to do with an icon address. ## MCP ```bash web mcp claude mcp add web --scope user -- web mcp ``` The server uses the SDK's `Server` class with TypeBox schemas. Every tool advertises an output schema and returns its result under `structuredContent.result`, compact JSON stays in `content` for clients that only render text. Tool symbols and titles match the native extensions, and nothing decorative gets written into the JSON-RPC stream. The client owns its TUI, not the server. `createMcpServer()` from the `@agntn/web/mcp` subpath returns the same server when your host brings its own transport. ## Pi and OMP The Pi extension requires `@earendil-works/pi-coding-agent` 0.86.1 or later. OMP keeps its 18.x API, tested with 18.2.6. Updating the package alone won't update either host. ```bash pi install git:github.com/agntn/web ``` The package ships the four tools as a [pi](https://pi.dev){rel=""nofollow""} extension and an OMP extension. Pi also gets two slash commands: `/web [query]` shows results as a selector and pastes the chosen URL into the editor, `/web-providers` shows configuration, reachability and capabilities. Both extensions read the same env vars as the library. Their TUI rows show progress, provider choice, result counts and fallback attempts, with bounded previews. A whole page rendered into the terminal is useless, so it never happens. ## What the answer is The tools return the normalized objects, not prose. A model summarizing them should say what the provider said. The [Explorer](https://web.agntn.dev/explorer) shows the same objects for any query, which is a quick way to check a summary against the source when a model gets creative. ::caution Titles, snippets, highlights and page content are published by whoever owns the page. They are data to report, not instructions to follow. An agent that reads `snippet: "ignore previous instructions"` has learned one thing about the page, and nothing about what to do next. :: ## Rolling your own Framework is none of the above? The four tools are thin wrappers over the helpers anyway: ```ts import { listProviders, readUrlDetailed, searchAllDetailed, searchByImage, searchWithFallback } from "@agntn/web"; export async function run(input: { operation: string; query?: string; url?: string; provider?: string }, signal?: AbortSignal) { switch (input.operation) { case "search": return input.provider === "all" ? searchAllDetailed(input.query!, { signal }) : searchWithFallback(input.query!, { signal }); case "search_image": return searchByImage(input.url!, { provider: input.provider, signal }); case "read": return readUrlDetailed(input.url!, { provider: input.provider, maxChars: 20_000, signal }); case "providers": return listProviders(); } } ``` # Custom Providers ## The contract ```ts abstract class Provider { static readonly providerName: string; static readonly defaultBaseURL: string; static readonly apiKeyEnvVar?: string | null; static readonly capabilities?: readonly ("search" | "searchImage" | "read")[]; static readonly capabilityDetails?: ProviderCapabilityDetails; static readonly searchFilterCapabilities?: SearchFilterCapabilities; protected readonly client: Client; protected readonly baseURL: string; get name(): string; } interface SearchProvider { search(query: string, options?: SearchRequestOptions): Promise; } interface ImageSearchProvider { searchByImage(url: string, options?: ImageSearchRequestOptions): Promise; } interface ReadProvider { read(url: string, options?: Readonly): Promise; } ``` A provider is a class that extends `Provider` and implements one or more of the three interfaces. Capabilities are read off the prototype: a class with a `search` method is a search provider, done. Methods declared as class fields are invisible to that check, so a class that uses them declares a static `capabilities` array instead. ## A minimal search provider ```ts import { Provider, register, searchProviders, type ProviderCapabilityDetails, type ProviderConfig, type SearchResult, } from "@agntn/web"; class InternalSearch extends Provider { static readonly providerName = "internal-search"; static readonly defaultBaseURL = "https://search.example.com"; static readonly apiKeyEnvVar = null; static readonly capabilityDetails = { search: { contentOptions: [], resultLimit: { default: 10, maximum: 50 }, resultFields: [], }, } as const satisfies ProviderCapabilityDetails; constructor(config: Readonly) { super(config, InternalSearch); } async search(query: string): Promise { return [{ url: "https://example.com", title: query, snippet: "Internal result" }]; } } register(InternalSearch); searchProviders(); // [..., "internal-search"] ``` From here `create("internal-search")`, `searchAll`, `searchWithFallback`, the CLI, the AI SDK tools, the MCP server and the Pi and OMP extensions all know about it. One catch: tool descriptions still advertise the built-in names, because they are frozen when a session starts. Execution accepts any registered name, the description just does not brag about yours until the next session. ## Names and keys Provider names are lowercase ASCII letters, digits and single hyphens inside. `apiKeyEnvVar: null` means registering is enough to count as configured. Leave it out and automatic selection expects the derived variable, `INTERNAL_SEARCH_API_KEY` for the class above. An explicit `create()` can always pass `apiKey`. Providers with several required credentials can declare a synchronous static `isConfigured()` check. Keep it local: discovery must not fetch or refresh tokens. `this.client` is the shared HTTP client, `getJSON`, `postJSON` and friends, with the redaction that keeps keys out of error messages. `this.baseURL` is the constructor's `baseURL` or the class default, checked to be `http` or `https`. ## Declare what you can do `capabilityDetails` is what `listProviders()`, `web providers` and `web_providers` report: content options, result limits, the result fields you fill, read formats and options. `searchFilterCapabilities` lists the filters you honour and the categories you accept. Skip it and requested filters come back as `undeclaredFilters` instead of being guessed. Guessing would be worse. ```ts static readonly searchFilterCapabilities = { filters: ["includeDomains", "startPublishedDate", "endPublishedDate"], categories: ["news", "docs"], } as const satisfies SearchFilterCapabilities; ``` ## Pagination A search provider can also implement `searchPage(query, options, continuation?)` and return a `ProviderSearchPage` with its own `continuation` and a `continuationStatus` of `next` or `unknown`. That state belongs to the provider, at most 2 048 characters, and only the same adapter ever reads it. The core helpers wrap it in a public token bound to the provider, query and options. No returned continuation means last page, and `capabilities.search.pagination` turns `true` as soon as the method exists. ## Errors Map raw failures with `normalizeError(error, this.name)`: a 401 becomes `AuthError`, a 429 `RateLimitError` with the `Retry-After` value, anything else stays `HTTPError`. Throw the typed errors yourself for the validation you do before the request. Automatic fallback reads the type to decide whether the next provider is worth trying, so a wrong type means either a wasted quota or a chain that stops too early. ## Reachability Implement `isAvailable(signal?)` when your endpoint can be down without being misconfigured, the way a self-hosted SearXNG can. `listProvidersAsync`, `detectAvailableProvidersAsync` and `resolveDefaultProviderAsync` call it and skip a provider whose probe fails. ## Where the built-ins live `src/providers/.ts`, one file per provider, each exporting its class and nothing else at module scope. The manifest in `src/providers/index.ts` carries what the class statics above carry for a custom provider, the capabilities and filters, next to a lazy `import()` of the file, so `create("brave")` loads Brave and only Brave. A built-in class needs no `register()` and no static capability metadata, only `providerName` and `defaultBaseURL`. `test/unit/providers-manifest.test.ts` checks the entry against the methods the class implements. Brave is the smallest search adapter with pagination, Jina is the template for a reader, SerpAPI is the one with reverse image search. # Explorer The [Explorer](https://web.agntn.dev/explorer) calls the docs worker, the worker calls the library: `searchProviderDetailed` or `searchWithFallback` for a search, `searchAllDetailed` for a fan-out, `readUrlDetailed` for a page, `listProviders` for the matrix. Exactly what a script would do. Nothing on the page is a mock, a slow answer is the engine being slow and a `503` means the worker has no key for what you asked. | Operation | What it answers | CLI equivalent | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | | Search | up to ten results from one provider, or the first configured one with its fallback chain, plus ignored filters and the pagination status | `web search --provider --json` | | Fan-out | every provider the worker has a key for, deduplicated, with the providers behind each URL and the failures kept | `web search --provider all --json` | | Read | one page as Markdown, bounded to 500, 2 000 or 8 000 characters, with the reader that answered and the readers tried | `web read --max-chars --json` | | Providers | `listProviders()` on the worker: capabilities, and whether the worker holds a key | `web providers --json` | Every query is a deep link. `/explorer?op=search&q=nitro&provider=brave` opens the page on that answer. ## Caching and manners Answers are cached on the worker, in KV in production: fifteen minutes for a search or a fan-out, an hour for a page. An answer with a provider failure inside is kept for five minutes only, a thrown failure is never cached, so a typo or an outage does not stick. One address can start twenty new provider queries a minute, cache hits are free. Each operation gets one budget of 25 seconds through `deadline`, fan-out runs three providers at a time, and a `429` is not retried. A rate limit from the engine comes back as a rate limit from the worker, retrying it into a ban would help nobody. Continuation tokens stay on the worker, so nobody pages through a provider's quota from a demo page. The worker holds keys for some providers, not all. The Providers tab says which. A search against a provider without a key is a `503`, not a bug in your code. ## The landing The panels on the [home page](https://web.agntn.dev) start from answers recorded through the library and labelled `sample`. As the page walks through its four queries, each one is replaced by the worker's live answer and relabelled `live`. The recorded answers live in `docs/app/utils/landing-fixtures.ts` and are regenerated with the library, never edited by hand. # Providers Every search provider answers the same `search()` with the same `SearchResult`. Five of them also `read()`, one also `searchByImage()`. What changes is the API behind it, the filters it can honour, which optional fields it fills, and what ends up in `metadata`. The pages are short on purpose. The part worth reading on each one is the gotcha list. | Provider | Env var | Search | Image | Read | Filters | Paging | | ------------------------------------------------------------ | ---------------------------------------------------- | ------ | ----- | ---- | ---------------------------- | ------ | | [Brave](https://web.agntn.dev/providers/brave) | `BRAVE_API_KEY` | yes | | | dates | yes | | [Context.dev](https://web.agntn.dev/providers/context) | `CONTEXT_DEV_API_KEY` | yes | | yes | domains | | | [Exa](https://web.agntn.dev/providers/exa) | `EXA_API_KEY` | yes | | | domains, category, dates | | | [Firecrawl](https://web.agntn.dev/providers/firecrawl) | `FIRECRAWL_API_KEY` | yes | | yes | domains, sources, categories | | | [Jina](https://web.agntn.dev/providers/jina) | `JINA_API_KEY` | yes | | yes | include domains, category | | | [Mojeek](https://web.agntn.dev/providers/mojeek) | `MOJEEK_API_KEY` | yes | | | domains, dates | yes | | [OpenAI Codex](https://web.agntn.dev/providers/openai-codex) | Existing login; optional `OPENAI_CODEX_ACCESS_TOKEN` | yes | | | none | | | [SearXNG](https://web.agntn.dev/providers/searxng) | none | yes | | | category | yes | | [SerpAPI](https://web.agntn.dev/providers/serpapi) | `SERPAPI_API_KEY` | yes | yes | | none | yes | | [SerpBase](https://web.agntn.dev/providers/serpbase) | `SERPBASE_API_KEY` | yes | | | category | yes | | [Tavily](https://web.agntn.dev/providers/tavily) | `TAVILY_API_KEY` | yes | | yes | domains, category, dates | | | [TinyFish](https://web.agntn.dev/providers/tinyfish) | `TINYFISH_API_KEY` | yes | | yes | domains, category, dates | yes | ::card-group :::card --- icon: i-simple-icons-brave title: Brave to: https://web.agntn.dev/providers/brave --- The Brave Search API, one GET per page, with extra snippets joined into text and a continuation that carries the offset. ::: :::card --- icon: i-lucide-book-text title: Context.dev to: https://web.agntn.dev/providers/context --- The Context.dev API for search with Markdown ranked by relevance and for scraping pages into Markdown or HTML. ::: :::card --- icon: i-lucide-sparkles title: Exa to: https://web.agntn.dev/providers/exa --- The Exa neural search API, the richest result shape of the built-ins, with highlights by default and summaries and full text on request. ::: :::card --- icon: i-lucide-flame title: Firecrawl to: https://web.agntn.dev/providers/firecrawl --- The Firecrawl v2 API for search with scraped passages and for scraping a page into Markdown or HTML. ::: :::card --- icon: i-lucide-file-text title: Jina to: https://web.agntn.dev/providers/jina --- Jina's s.jina.ai for search and r.jina.ai for reading, the default reader, with an optional key. ::: :::card --- icon: i-simple-icons-mojeek title: Mojeek to: https://web.agntn.dev/providers/mojeek --- The Mojeek Search API, an independent index with classic SERP metadata, domain and date filters and paging. ::: :::card --- icon: i-lucide-search title: OpenAI Codex to: https://web.agntn.dev/providers/openai-codex --- Hosted web search through the ChatGPT Codex backend using an existing Pi, OMP, Codex or OpenCode login. Native sources, not snippets invented by the model. ::: :::card --- icon: i-simple-icons-searxng title: SearXNG to: https://web.agntn.dev/providers/searxng --- A self-hosted SearXNG metasearch instance, no key, with a reachability probe and the engines behind each result in metadata. ::: :::card --- icon: i-lucide-scan-search title: SerpAPI to: https://web.agntn.dev/providers/serpapi --- SerpAPI's Google endpoints for classic SERP results and Google Lens for reverse image search. ::: :::card --- icon: i-lucide-layers title: SerpBase to: https://web.agntn.dev/providers/serpbase --- SerpBase's Google SERP endpoints, with a category that picks the web, images, news or videos endpoint. ::: :::card --- icon: i-lucide-compass title: Tavily to: https://web.agntn.dev/providers/tavily --- The Tavily search API, with an optional generated answer in response metadata, and Tavily Extract for reading pages. ::: :::card --- icon: i-lucide-fish title: TinyFish to: https://web.agntn.dev/providers/tinyfish --- TinyFish Search for news and research results with publisher metadata, and TinyFish Fetch for reading pages. ::: :: # Brave :provider-facts{provider="brave"} ## Address it ```ts import { create } from "@agntn/web"; const brave = await create("brave"); // BRAVE_API_KEY const results = await brave.search("typescript runtimes", { maxResults: 10 }); const recent = await brave.search("typescript runtimes", { startPublishedDate: "2026-06-01" }); ``` The key goes in the `X-Subscription-Token` header. Pass `apiKey` to `create()` if you do not want env involved. ## What it reads | Call | Endpoint | | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `search` | `GET /res/v1/web/search?q=…&count=…&offset=…&extra_snippets=true&text_decorations=false` on api.search.brave.com, plus `freshness=YYYY-MM-DDtoYYYY-MM-DD` when a date bound is set | `count` is `maxResults`, 20 at most. `offset` comes from the continuation and steps through pages of that size. `query.more_results_available` in the response decides whether the next token is `next` or `end`, so Brave is one of the few engines that actually tells you. ## What comes back `snippet` is the `description` with Brave's HTML escapes decoded. `favicon` comes from `meta_url.favicon`, and `text` is the `extra_snippets` joined with newlines when Brave sends them. `publishedDate` is Brave's `page_age`, the page's published or last modified date as Brave read it, so it can be missing. `startPublishedDate` and `endPublishedDate` go out as `freshness`, the only filter Brave has here. No score, and `includeDomains` and friends come back in `ignoredFilters`. ## Gotchas - A page is 20 results max. Ask for more and you get 20 plus a continuation. - Brave bolds query terms with `` unless you send `text_decorations=false`, and it escapes `description` like HTML (`'`, `&`, `"`). The adapter turns the bolding off and decodes the escapes, so `snippet` is plain text. `title` and `extra_snippets` come as text already. - The free plan is a couple of thousand queries a month and a burst gets you a 429 with `Retry-After`, which is `RateLimitError` here. - `freshness` wants `YYYY-MM-DDtoYYYY-MM-DD`, both days inside the window, the same day twice gives that day's pages. Anything else, a time on the date included, Brave drops without a word and answers unfiltered, so a datetime bound becomes its UTC day here, a missing start becomes `1970-01-01` and a missing end becomes today's UTC date. The open forms `2026-06-01to` and `to2026-06-01` answered correctly when probed, but Brave does not document them, and a filter that fails silently is not the place to lean on that. - Brave only fills extra snippets on a Search plan. A free key still takes `extra_snippets=true` and leaves `text` empty. ## Where it lives `src/providers/brave.ts`. Smallest search adapter with pagination, copy it when you write a new one. # Context.dev :provider-facts{provider="context"} ## Address it ```ts import { create, readUrl } from "@agntn/web"; const context = await create("context"); // CONTEXT_DEV_API_KEY const results = await context.search("nitro cloudflare preset", { includeDomains: ["nitro.build"] }); const page = await readUrl("https://nitro.build/deploy", { provider: "context", format: "markdown" }); ``` Bearer auth against `api.context.dev/v1`. ## What it reads | Call | Endpoint | | -------- | ---------------------------- | | `search` | `POST /web/search` | | `read` | `GET /web/scrape/markdown?…` | ## What comes back Each search result carries the page's Markdown in `text` when Context.dev has it, plus `metadata.relevance` (`high`, `medium`, `low`) and `metadata.markdownCode`. Domain filters work. No categories, sources or dates. Reads give `markdown` or `html` and take `targetSelector`, `removeSelector`, `timeout` and `noCache`. ## Gotchas - Spent credits are not a bad key. Context.dev returns 401 with `error_code: "USAGE_EXCEEDED"`; the adapter reports `PaymentError`, preserving the HTTP status and body. Automatic search and read can try the next configured provider. Other 401 responses remain `AuthError` and stop the chain; an explicit provider never switches. - `maxResults` goes up to 100, default 10. ## Where it lives `src/providers/context.ts`. # Exa :provider-facts{provider="exa"} ## Address it ```ts import { create } from "@agntn/web"; const exa = await create("exa"); // EXA_API_KEY const results = await exa.search("typescript runtime benchmarks", { maxResults: 5, summary: true, fullText: true, category: "research paper", startPublishedDate: "2026-01-01", }); ``` The key goes in the `x-api-key` header, the request is a POST. ## What it reads | Call | Endpoint | | -------- | ------------------------------------------------------------------------------------- | | `search` | `POST /search` on api.exa.ai with `type: "auto"`, `numResults` and a `contents` block | `contents.highlights` follows `highlights` (default true), `contents.summary` follows `summary`, `contents.text` follows `fullText`. ## What comes back `score`, `publishedDate`, `author`, `image`, `favicon`, `highlights[]` by default, `summary` and `text` on request. Domain filters, a `category` forwarded as given, and both date bounds all work. The most complete result shape in the provider list. ## Gotchas - `fullText` returns the whole page for every result. Ten of those is a lot of tokens, stick to highlights unless you will actually read the text. - `category` is passed through as typed, so an unknown value is Exa's problem, not the library's. - Exa is first in the detection order. With `EXA_API_KEY` set it is the default provider, whether you meant that or not. ## Where it lives `src/providers/exa.ts`. # Firecrawl :provider-facts{provider="firecrawl"} ## Address it ```ts import { create, readUrl } from "@agntn/web"; const firecrawl = await create("firecrawl"); // FIRECRAWL_API_KEY const results = await firecrawl.search("vercel ai sdk tools", { sources: ["web", "news"], categories: ["research"], }); const page = await readUrl("https://sdk.vercel.ai/docs", { provider: "firecrawl", targetSelector: "main" }); ``` Bearer auth against `api.firecrawl.dev`. ## What it reads | Call | Endpoint | | -------- | -------------------------------------------------------------------------- | | `search` | `POST /v2/search` with `limit`, `sources`, `categories` and scrape options | | `read` | `POST /v2/scrape` with `formats` | ## What comes back `snippet` is a passage relevant to the query, Markdown included when the source has it. `text` is the scraped Markdown, `image` and `metadata` when present. `highlights: false` turns the passage scraping off. Domain filters, `sources` (`web`, `news`, `images`) and `categories` (`research`, `pdf`, `developer`) work. The singular `category` is not forwarded, Firecrawl wants the plural. ## Gotchas - `searchDetailed` exposes response metadata: request `id`, `warning` and `creditsUsed`. `web search --provider firecrawl --json` prints it. - The `developer` category cannot be mixed with the others. The adapter rejects the combination before the request instead of letting Firecrawl answer with a 400. - The reader rejects `maxTokens` instead of ignoring it. Use `maxChars` for a bound that means the same thing everywhere. ## Where it lives `src/providers/firecrawl.ts`. # Jina :provider-facts{provider="jina"} ## Address it ```ts import { create, readUrl } from "@agntn/web"; const jina = await create("jina"); // JINA_API_KEY, required for search const results = await jina.search("model context protocol", { category: "news" }); const page = await readUrl("https://modelcontextprotocol.io", { format: "markdown" }); // Jina first, no key needed ``` The key is sent as a Bearer token. Search needs it, reading works without one and sends it when present. ## What it reads | Call | Endpoint | | -------- | ------------------------------------------------------------------------------------------------------------------------- | | `search` | `GET https://s.jina.ai/…` with `Accept: application/json` | | `read` | `GET https://r.jina.ai/` with `X-Return-Format`, `X-Target-Selector`, `X-Remove-Selector`, `X-Timeout`, `X-No-Cache` | The reader host is derived from the search host, so a custom `baseURL` for `s.` moves `r.` with it. ## What comes back Search results carry the page content in `text`, plus `publishedDate`, `image` and `metadata`. `includeDomains` and a `category` of `web`, `images` or `news` work. Reads give `markdown`, `text` or `html` and take `maxTokens` natively. ## Gotchas - Without a key r.jina.ai is rate limited per address. With a key, an empty balance answers 402, which is eligible for fallback to the next configured reader. - A 409 from the reader (could not fetch the page right now) is eligible for fallback too. - `text` on search results is the whole page. Ten results can be huge, keep `maxResults` low. ## Where it lives `src/providers/jina.ts`, the template for a reader. # Mojeek :provider-facts{provider="mojeek"} ## Address it ```ts import { create } from "@agntn/web"; const mojeek = await create("mojeek"); // MOJEEK_API_KEY const results = await mojeek.search("independent search index", { excludeDomains: ["reddit.com"], startPublishedDate: "2026-01-01", }); ``` The key is a query parameter on `api.mojeek.com`. ## What it reads | Call | Endpoint | | -------- | ------------------------------------------------------------------------------------ | | `search` | `GET /search?fmt=json&…` with `fi`, `fe`, `since`, `before` and an offset for paging | ## What comes back `score`, `publishedDate`, `image` and `metadata.{confidence, documentSize, lastModifiedDate, crawledDate, moreResultsFromDomain, imageWidth, imageHeight}`. Include and exclude domains and both date bounds work, paging too. ## Gotchas - Dates get converted to Mojeek's own format. Pass ISO dates and let the adapter deal with it. - The free trial is small. Mojeek is last in the detection order for a reason. ## Where it lives `src/providers/mojeek.ts`. # SearXNG :provider-facts{provider="searxng"} ## Address it ```ts import { create } from "@agntn/web"; const searx = await create("searxng", { baseURL: "https://searx.example.com" }); // default http://localhost:8080 const results = await searx.search("query", { category: "general" }); ``` No key. `apiKeyEnvVar` is `null`, so SearXNG always counts as configured. Set `baseURL` to your instance. ## What it reads | Call | Endpoint | | ------------- | --------------------------------------------------- | | `search` | `GET /search?format=json&q=…&pageno=…&categories=…` | | `isAvailable` | `GET /` with a short timeout | ## What comes back `score`, `publishedDate`, `image` and `metadata.{engine, engines, category}`. A `category` is forwarded as given. No domain filters, no date bounds. Paging works through `pageno`. ## Gotchas - The instance has to allow the JSON format (`search.formats` in its settings). Most public instances do not, so bring your own. - Configured by definition means `searchAll` will try it, even when nothing listens on `localhost:8080`. `detectAvailableProvidersAsync` and `resolveDefaultProviderAsync` run the probe and skip a dead instance. The docs worker leaves it out of fan-out for the same reason. - `maxResults` is applied on our side, the instance decides its page size. ## Where it lives `src/providers/searxng.ts`, the only built-in with `isAvailable()`. # SerpAPI :provider-facts{provider="serpapi"} ## Address it ```ts import { create, searchByImage } from "@agntn/web"; const serpapi = await create("serpapi"); // SERPAPI_API_KEY const results = await serpapi.search("query", { maxResults: 10 }); const matches = await searchByImage("https://example.com/image.jpg", { provider: "serpapi" }); ``` The key is a query parameter on `serpapi.com`. ## What it reads | Call | Endpoint | | --------------- | --------------------------------------------- | | `search` | `GET /search?engine=google&q=…&num=…&start=…` | | `searchByImage` | `GET /search?engine=google_lens&url=…` | ## What comes back Search: `publishedDate`, `image` (thumbnail), `favicon` and `metadata.{position, source, displayedLink}`. No filters. Paging through `start`. Image matches carry `pageUrl`, `imageUrl`, dimensions, `source`, `position` and `exactMatch`. ## Gotchas - The only built-in reverse image provider. Everything under [Reverse image search](https://web.agntn.dev/guide/image) applies. - 100 searches a month on the free plan. A burst answers 429 fast. - `maxResults` is applied on our side to whatever Google or Lens returns. Google pays little attention to `num` (a `maxResults: 1` call still brought back nine organic results) and hands a different page for a `start` off the ten grid, so a small `maxResults` walks each Google page in slices: the continuation fetches the same page again, which SerpAPI serves from its cache for an hour without charging, and `start` moves only once the page is used up. - No matches is an empty list, not an error. An image Lens cannot fetch looks the same, so `[]` may also mean the provider never saw the picture. ## Where it lives `src/providers/serpapi.ts`. # SerpBase :provider-facts{provider="serpbase"} ## Address it ```ts import { create } from "@agntn/web"; const serpbase = await create("serpbase"); // SERPBASE_API_KEY const results = await serpbase.search("query", { category: "news", maxResults: 10 }); ``` The key goes in the `X-API-Key` header against `api.serpbase.dev`. ## What it reads | Call | Endpoint | | -------- | ----------------------------------------------------------------------------------------------------- | | `search` | `POST` to the endpoint the category picks: web by default, `images`/`image`, `news`, `videos`/`video` | ## What comes back `publishedDate`, `image`, `favicon` and `metadata.{position, rank, searchType, requestId, elapsedMs, creditsCharged}`. Only `category` is honoured. Paging works, but the next token stays `unknown` until a page comes back empty, SerpBase does not say when it is done. ## Gotchas - `maxResults` is applied on our side to the returned page, the endpoint decides the page size. - Credits are charged per request and reported in `metadata.creditsCharged`. An empty balance answers HTTP 200 with `status: 1020`. That is `PaymentError` here, so automatic search tries the next configured provider instead of stopping. ## Where it lives `src/providers/serpbase.ts`. # Tavily :provider-facts{provider="tavily"} ## Address it ```ts import { create, readUrl, searchProviderDetailed } from "@agntn/web"; const tavily = await create("tavily"); // TAVILY_API_KEY const results = await tavily.search("query", { includeDomains: ["docs.python.org"] }); const recent = await tavily.search("query", { category: "news", startPublishedDate: "2026-06-01" }); const answer = await searchProviderDetailed("tavily", "query", { summary: true, fullText: true }); answer.metadata?.answer; // the generated answer for the query const page = await readUrl("https://docs.python.org/3/", { provider: "tavily", format: "text" }); ``` Search puts the key in the body of a POST to `api.tavily.com`, Extract takes it as a Bearer token. ## What it reads | Call | Endpoint | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `search` | `POST /search` with `max_results`, `search_depth: "basic"`, `include_answer`, `include_raw_content`, `include_published_date: true`, and `topic`, `start_date`, `end_date` when asked | | `read` | `POST /extract` with one URL in `urls`, `extract_depth: "basic"`, `format` and `timeout` | ## What comes back `score`, `publishedDate` and, with `fullText`, `text` from `raw_content`. `summary` asks for the query answer, which lands in response `metadata.answer`, not on a result. Include and exclude domains work, `startPublishedDate` and `endPublishedDate` go through as `start_date` and `end_date`, and `category` set to `general`, `news` or `finance` picks Tavily's `topic`. Reads give `markdown` or `text` in `content`, the page `title` and Tavily's `request_id` in `metadata.requestId`. `timeout` is passed on in seconds, clamped to Tavily's 1 to 60. No links, images or selectors, and `maxTokens` and `noCache` are ignored. ## Gotchas - The answer is one string for the whole query, so it lives on the response, not on a result. `search()` alone cannot show it, `searchDetailed()` can. - `search_depth` is always `basic`. Advanced depth costs more credits and is not exposed, on purpose. - Tavily dates a result only when asked, so every search asks. It writes the date as `Tue, 11 Mar 2025 17:00:00 GMT`, which becomes ISO 8601 here like everyone else's, and a page it couldn't date has no `publishedDate`. Tavily calls the field beta. - The window is by day, a datetime bound loses its time. Pages Tavily couldn't date stay in, three of five on a plain query, so `filter_by_published_date` isn't sent on purpose. - A spent plan or pay-as-you-go cap comes back as HTTP 432 or 433, not 402. Both are `PaymentError` here, so automatic search and read move on to the next configured provider. A wrong key is still 401 and still stops the chain. - A page Extract cannot fetch is not an HTTP error. The response is a 200 with an empty `results` and the reason in `failed_results` (`404 page not found`, `Failed to fetch url`, which is also what a PDF gets). That becomes a `WebError` with the reason, not a fallback. - `url` on a read is the one you asked for. Extract does not report where a redirect ended up. - Extract is `basic` depth, `advanced` (tables and embedded content, twice the credits) is not exposed. Basic costs one credit per five successful pages. ## Where it lives `src/providers/tavily.ts`. # TinyFish :provider-facts{provider="tinyfish"} ## Address it ```ts import { create, readUrl } from "@agntn/web"; const tinyfish = await create("tinyfish"); // TINYFISH_API_KEY const results = await tinyfish.search("transformer inference", { category: "research_paper" }); const page = await readUrl("https://arxiv.org/abs/1706.03762", { provider: "tinyfish" }); ``` The key goes in the `X-API-Key` header. Search talks to `api.search.tinyfish.ai`, reads to `api.fetch.tinyfish.ai`, `readBaseURL` in the config moves the second one. ## What it reads | Call | Endpoint | | -------- | ------------------------------------------------------------------ | | `search` | `GET` on the search host with the query, filters and a page number | | `read` | `POST` on the fetch host | ## What comes back `publishedDate`, `author` and `metadata.{position, siteName, publisher, authors, venue, year, citedByCount, pdfUrl}`, which is why it is the one to use for papers. Domain filters, a `category` of `news` or `research_paper` and both date bounds work. Paging goes up to ten pages, the next token stays `unknown` until a page comes back empty. Reads give `markdown` or `html`. ## Gotchas - `maxResults` is applied on our side to one result page. - Search access has to be enabled on the TinyFish account. A key without it answers with an auth failure, not a helpful message. ## Where it lives `src/providers/tinyfish.ts`. # OpenAI Codex :provider-facts{provider="openai-codex"} This is the search available through a Codex login, not the separately billed OpenAI API. The adapter calls the ChatGPT backend with the hosted `web_search` tool. A model answer without a completed search is an error, not a search result. ## Credentials Log in with Pi, OMP, Codex or OpenCode, then search. No manual token export: ```bash web search "Node.js release notes" --provider openai-codex --json ``` Pi tools resolve credentials through the host's model registry. Pi owns expiry-based refresh, but its extension API can't force renewal of a revoked token. OMP's auth broker supports forced refresh and keeps the calling session's account affinity. Listing providers doesn't resolve a token, and concurrent calls don't share accounts. Outside a host, automatic discovery reads saved logins in this order: | Client | Default store | Overrides | | -------- | ------------------------------------------------ | ----------------------------------------------------------- | | Codex | `~/.codex/auth.json` | `CODEX_HOME` | | Pi | `~/.pi/agent/auth.json` | `PI_CODING_AGENT_DIR` | | OMP | `~/.omp/agent/agent.db`, then legacy `auth.json` | `PI_CODING_AGENT_DIR`, `OMP_PROFILE` or legacy `PI_PROFILE` | | OpenCode | `~/.local/share/opencode/auth.json` | `XDG_DATA_HOME` | Only OAuth access tokens are used. API keys, disabled OMP accounts and expired tokens are skipped. Discovery never writes login files, creates a database or rotates saved refresh tokens. Standalone callers need a usable saved access token; if it expired, open the owning client and let it refresh. Codex logins held only in an OS keyring are not read. OMP SQLite requires Node's optional `node:sqlite` builtin; the native OMP host resolver does not. Choose a store with `codex.authSource` or `OPENAI_CODEX_AUTH_SOURCE`: `auto` (default), `codex`, `pi`, `omp`, `opencode`, or `none` to disable local and host discovery. A named source bypasses other stores and the invoking host. Explicit `codex.credentials` wins over environment overrides, which win over automatic discovery. `OPENAI_CODEX_ACCESS_TOKEN` can still supply a bearer; `OPENAI_CODEX_ACCOUNT_ID` is optional when its JWT contains `chatgpt_account_id`. A partial environment override does not silently switch to a different account. `OPENAI_API_KEY` does not configure this provider. In application code, explicit credentials remain an optional override: ```ts import { createSearchProvider } from "@agntn/web"; const search = createSearchProvider("openai-codex", { codex: { credentials: { accessToken: "your-oauth-access-token", accountId: "your-chatgpt-account-id", }, model: "gpt-5.5", }, }); const results = await search.search("Node.js release notes", { maxResults: 5 }); ``` For refresh, pass a callback instead of the fixed pair. Your login service owns token storage and concurrent refresh coordination: ```ts import { createSearchProvider, type CodexCredentialProvider } from "@agntn/web"; export function createCodexSearch(credentials: CodexCredentialProvider) { return createSearchProvider("openai-codex", { codex: { credentials } }); } ``` The callback receives `{ refresh, signal }` and returns `{ accessToken, accountId? }`, directly or through a promise. The account ID is derived from the access token when omitted. It runs before each search with `refresh: false`. After an authentication rejection it runs once more with `refresh: true`. The adapter retries once only if the bearer or account changed. If the host returns the rejected pair unchanged, search stops and asks for a fresh login. Check expiry even on the first call. The library never handles a refresh token itself. Explicit credentials and model take precedence over the environment. Instance configuration is local to that instance, it does not enroll a callback in the global `auto` or `all` flows. ## Endpoint and results `POST https://chatgpt.com/backend-api/codex/responses`, with Bearer auth, `chatgpt-account-id`, `store: false`, streaming enabled and `tool_choice: { type: "web_search" }`. Results come only from structured search sources and `url_citation` annotations. They have `url`, `title` and an empty `snippet`. Generated prose is not a page excerpt. URLs that appear only in prose never become results. ```ts import { searchProviderDetailed } from "@agntn/web"; const response = await searchProviderDetailed("openai-codex", "Node.js release notes", { maxResults: 5, summary: true, }); console.log(response.results); console.log(response.metadata?.answer); ``` The detailed helper uses the same automatic login discovery and explicit environment overrides. `summary: true` includes the generated answer in `metadata.answer`; it stays absent otherwise. Metadata also carries the returned model, request ID and token usage when available. ## Limits and traps - This is an experimental adapter for the Codex backend, not a stable public search API. Model access and subscription limits still apply. - The default model is `gpt-5.5`. Override it with `codex.model` or `OPENAI_CODEX_MODEL` if your account needs another model that supports hosted web search. There is no hidden model fallback. - `maxResults` caps collected sources locally, not upstream search work. Default 10, maximum 100. It does not promise that many sources. - No paging, domain/date/category filters, highlights, full page text, reverse image search or URL reading. Detailed helpers report ignored filters. - `auto` tries Codex after the existing API providers, before custom providers and SearXNG. `all` includes it when a usable saved login, a native host resolver or explicit environment credentials are available. Discovery does not check subscription status or refresh tokens. - A search has a 90-second ceiling, including credential resolution and its one possible refresh. Caller cancellation and earlier deadlines still win. The transport rejects streams above 8 MiB and events above 1 MiB. - The adapter does not retry metered requests after HTTP 429 or server errors. Automatic search may move to another configured provider under the library's normal fallback rules. - OAuth credentials can only go to the fixed ChatGPT endpoint. Custom `baseURL` values and HTTP redirects are refused. Error messages omit upstream bodies and credential callback diagnostics. - Do not put personal subscription credentials on a public explorer worker. Local configuration is enough to use the library and its tools. Implementation: `src/providers/openai-codex.ts`. Protocol reference: the Codex search adapter in [oh-my-pi](https://github.com/can1357/oh-my-pi/blob/main/packages/coding-agent/src/web/search/providers/codex.ts){rel=""nofollow""}. # @agntn/web :landing-home