Scheduled crawls with a ceiling on every single request

Where it goes wrong

Crawl code drifts: one script follows redirects, another has no timeout, a third writes a 40 MB page into memory. Cost is discovered at the end of the month instead of at the top of the run.

What Backfire handles

• Every page is one trip, so a 4,000-page run costs 4,000 trips and nothing else.
• Request and response sizes are bounded per call, so one huge page cannot blow up a worker.
• Redirects are returned rather than followed, which keeps the crawl graph explicit.
• The exact URLs fetched sit in the activity log, exportable as CSV or JSON for reconciliation.

The request

for (const page of pages) {
  const res = await fetch('https://backfireapi.base44.app/functions/forward', {
    method: 'POST',
    headers: { Authorization: `Bearer ${process.env.BACKFIRE_KEY}`, 'Content-Type': 'application/json' },
    body: JSON.stringify({ url: page, method: 'GET', timeout_ms: 10000 }),
  });

  const envelope = await res.json();
  if (!envelope.ok) { skipped.push([page, envelope.error.code]); continue; }
  if (envelope.upstream.status === 200) save(page, envelope.body.text);
  if (envelope.quota.remaining < 100) break; // stop cleanly before the balance runs out
}

All use cases