Reading

Read a URL

import { readUrl } from "@agntn/web";

const page = await readUrl("https://example.com/article", {
  provider: "jina",
  format: "markdown",
  maxChars: 20_000,
});

page.title;
page.content; // Markdown, at most 20 000 code points
page.truncated; // true when more remained
page.continuation; // opaque token for the next slice

interface ReadResult {
  url: string;
  title?: string;
  description?: string;
  content: string;
  text?: string;
  html?: string;
  publishedDate?: string;
  image?: string;
  links?: string[];
  images?: string[];
  metadata?: Record<string, unknown>;
  truncated?: boolean;
  continuation?: string;
}

Readers

readProviders() lists them at runtime, custom readers included, isReadProvider(provider) is the type guard.

maxTokens is Jina's token budget, and it's a tripwire, not scissors. A page over it doesn't come back cut, Jina refuses it with a 409. Firecrawl rejects maxTokens on an explicit read instead of ignoring it. You asked for a token bound and got an unbounded page, that is a lie, not a feature.

Automatic selection

Without a provider, readUrl starts with Jina Reader, which needs no key for a basic read, and tries the other configured readers after a payment, rate limit, timeout, connection or server failure. Jina's HTTP 409 counts too, it just means the page could not be fetched right now. So does PageFetchError, a reader whose own fetcher hit a bot check, a login wall or a timeout, since the next reader fetches the page its own way. Auth failures and invalid requests stop the chain.

When the reader matters, ask for the detailed answer:

import { readUrlDetailed } from "@agntn/web";

const { result, requestedProvider, provider, attempts, failures } = await readUrlDetailed("https://example.com/article");

requestedProvider; // "auto"
provider; // "firecrawl", the reader that answered
attempts; // ["jina", "firecrawl"]
failures; // [{ provider: "jina", error: "HTTP 402: …" }]

ProviderFallbackError is thrown when every eligible reader fails, with the same attempts and failures on it.

A reader that just ran out of credits waits at the back of the chain for 30 minutes and shows up in skipped, the same as in automatic search.

A maxTokens meant for Jina doesn't follow the chain. Automatic reads leave it out for every reader that doesn't list it. So Firecrawl reads the page instead of refusing the option, and the detailed answer says so with ignoredOptions: ["maxTokens"]. A page Jina refused over budget falls through the same way. The next reader returns it whole, so pass maxChars when the size matters. A custom reader that declares no options still gets maxTokens.

The output bound

maxChars is counted in Unicode code points after the provider answers, so it means the same thing on every reader and for every script. Token counts do not, that is why this is not maxTokens. When more remains, truncated is true and continuation carries an opaque token. Pass it back with the same URL and the same native options:

const first = await readUrl(url, { maxChars: 20_000 });

if (first.truncated) {
  const second = await readUrl(url, { maxChars: 20_000, continuation: first.continuation });
}

The token is pinned to the reader that answered and to a fingerprint of the content. Page changed between slices, you get StaleReadContinuationError. URL or options differ, InvalidReadContinuationError. A paginated slice also drops the provider's text and html duplicates, otherwise they would smuggle the whole page past the bound. links and images stay out of a bounded read too unless you pass links: true or images: true: a Wikipedia article through TinyFish is 20 000 characters of content next to 138 kB of links, and a bound that lets that through isn't one. TinyFish is asked for the link list only when the read will keep it. Fetch leaves that list off unless you ask. Image URLs stay on the request, because the page image is the first one and a bounded read still returns it.

The library and the CLI are unbounded unless maxChars is set. The agent surfaces default to DEFAULT_AGENT_READ_MAX_CHARS (20 000) and accept at most MAX_AGENT_READ_CHARS (200 000). maxTokens is the provider's own request option, not an approximation of this bound.

Batches

import { readBatch, readBatchDetailed } from "@agntn/web";

const pages = await readBatch(["https://example.com/one", "https://example.com/two"], { maxChars: 8000 });
// [{ url, result }, { url, error }]

const detailed = await readBatchDetailed(["https://example.com/one"]);
// [{ url, result, requestedProvider, provider, attempts, failures }]

Up to ten URLs, three at a time by default, order preserved, one failure does not throw away the others. A continuation is rejected for a batch.