Reading

Read a URL

import { readUrl } from "@agntn/web";

const page = await readUrl("https://example.com/article", {
  provider: "jina",
  format: "markdown",
  maxChars: 20_000,
});

page.title;
page.content; // Markdown, at most 20 000 code points
page.truncated; // true when more remained
page.continuation; // opaque token for the next slice

interface ReadResult {
  url: string;
  title?: string;
  description?: string;
  content: string;
  text?: string;
  html?: string;
  publishedDate?: string;
  image?: string;
  links?: string[];
  images?: string[];
  metadata?: Record<string, unknown>;
  truncated?: boolean;
  continuation?: string;
}

Readers

readProviders() lists them at runtime, custom readers included, isReadProvider(provider) is the type guard. Firecrawl rejects maxTokens instead of ignoring it. You asked for a token bound and got an unbounded page, that is a lie, not a feature.

Automatic selection

Without a provider, readUrl starts with Jina Reader, which needs no key for a basic read, and tries the other configured readers after a payment, rate limit, timeout, connection or server failure. Jina's HTTP 409 counts too, it just means the page could not be fetched right now. Auth failures and invalid requests stop the chain.

When the reader matters, ask for the detailed answer:

import { readUrlDetailed } from "@agntn/web";

const { result, requestedProvider, provider, attempts, failures } = await readUrlDetailed("https://example.com/article");

requestedProvider; // "auto"
provider; // "firecrawl", the reader that answered
attempts; // ["jina", "firecrawl"]
failures; // [{ provider: "jina", error: "HTTP 402: …" }]

ProviderFallbackError is thrown when every eligible reader fails, with the same attempts and failures on it.

The output bound

maxChars is counted in Unicode code points after the provider answers, so it means the same thing on every reader and for every script. Token counts do not, that is why this is not maxTokens. When more remains, truncated is true and continuation carries an opaque token. Pass it back with the same URL and the same native options:

const first = await readUrl(url, { maxChars: 20_000 });

if (first.truncated) {
  const second = await readUrl(url, { maxChars: 20_000, continuation: first.continuation });
}

The token is pinned to the reader that answered and to a fingerprint of the content. Page changed between slices, you get StaleReadContinuationError. URL or options differ, InvalidReadContinuationError. A paginated slice also drops the provider's text and html duplicates, otherwise they would smuggle the whole page past the bound.

The library and the CLI are unbounded unless maxChars is set. The agent surfaces default to DEFAULT_AGENT_READ_MAX_CHARS (20 000) and accept at most MAX_AGENT_READ_CHARS (200 000). maxTokens is the provider's own request option, not an approximation of this bound.

Batches

import { readBatch, readBatchDetailed } from "@agntn/web";

const pages = await readBatch(["https://example.com/one", "https://example.com/two"], { maxChars: 8000 });
// [{ url, result }, { url, error }]

const detailed = await readBatchDetailed(["https://example.com/one"]);
// [{ url, result, requestedProvider, provider, attempts, failures }]

Up to ten URLs, three at a time by default, order preserved, one failure does not throw away the others. A continuation is rejected for a batch.