TinyFish

Address it

import { create, readUrl } from "@agntn/web";

const tinyfish = await create("tinyfish"); // TINYFISH_API_KEY
const results = await tinyfish.search("transformer inference", { category: "research_paper" });
const page = await readUrl("https://arxiv.org/abs/1706.03762", { provider: "tinyfish" });

The key goes in the X-API-Key header. Search talks to api.search.tinyfish.ai, reads to api.fetch.tinyfish.ai, readBaseURL in the config moves the second one.

What it reads

What comes back

publishedDate, author and metadata.{position, siteName, publisher, authors, venue, year, citedByCount, pdfUrl}, which is why it is the one to use for papers. Domain filters, a category of news or research_paper and both date bounds work. Paging goes up to ten pages, the next token stays unknown until a page comes back empty. Reads give markdown or html.

Gotchas

• maxResults is applied on our side to one result page.
• A bounded read asks for links only when it will keep them. image_links stays on, because image is the first image URL and a bounded read still returns it.
• TinyFish parses a plain-text file as HTML, whether the read asks for Markdown or HTML. Every <...> comes back empty, and <script/script.h> swallows the rest of the file. So a read with no title, description or language asks again for the JSON tree. When that tree holds one code node, its text is the file byte for byte, and the read returns it. An html read gets it escaped inside <pre>, the way a browser shows a text file. Asked right after the Markdown, the tree can come from cache as text: null, so the read asks once more with ttl: 0. A noCache read skips it, since that tree never came from cache. A failed tree request keeps what came first.
• A page Fetch could not get is a 200 with the reason in errors. Reasons about TinyFish's own fetcher (bot_blocked, login_required, timeout, target_http_error and the like) become PageFetchError, so an automatic read moves on to Tavily or Exa. The status inside target_http_error is what TinyFish got back, and once it was a 401 on a public raw GitHub file that curl read fine. page_not_found stays an HTTPError with its 404 or 410, and its url names the missing page, not an empty string. A bad URL or selector stops the chain.
• The window is by day, a datetime bound becomes its UTC day. With research_paper only the year of each bound goes through, as pub_year_min and pub_year_max.
• Search access has to be enabled on the TinyFish account. A key without it answers with an auth failure, not a helpful message.

Where it lives

src/providers/tinyfish.ts.