Turn any web page into clean Markdown.

A self-hosted scraper with a Firecrawl-compatible API, in one Go binary and one container. It renders JavaScript with Lightpanda and strips ads, trackers and cookie banners. No Redis, no Playwright, no Chrome.

Get started GitHub

Pages scraped at once

SmallMediumLarge

2 vCPU · 4 GB memory

Huntsman45 pages at once4.58 pages/s · 90.9% scraped
Firecrawl8 pages at once0.91 pages/s · 92.5% scraped
Crawl4AICrashed or ran out of memory even at 2 pages at once

5.6× the pages in flight of Firecrawl, and 5.0× the pages per second.

4 vCPU · 8 GB memory

Huntsman75 pages at once7.74 pages/s · 91.8% scraped
Firecrawl15 pages at once1.58 pages/s · 92.2% scraped
Crawl4AI15 pages at once1.35 pages/s · 80.6% scraped

5.0× the pages in flight of Firecrawl, and 4.9× the pages per second.

8 vCPU · 16 GB memory

Huntsman129 pages at once11.54 pages/s · 90.8% scraped
Firecrawl43 pages at once2.90 pages/s · 92.3% scraped
Crawl4AI22 pages at once1.62 pages/s · 81.8% scraped

3.0× the pages in flight of Firecrawl, and 4.0× the pages per second.

How we measured

1,000 random URLs from the test split of huntsman-scrape-dataset-v1, scraped live on ARM cloud machines with each engine self-hosted alone. A binary search found each engine’s exact limit: a level passes if the engine doesn’t crash, success stays within 10 points of its best level, and the median page takes under 3× as long as at 8 pages. Firecrawl ran in scrape-only mode on the small machine. One run per level; methods and raw results.

Try it on any page

Markdown appears here

Paste a link and Huntsman fetches it live, renders any JavaScript, and strips ads, trackers and cookie banners. You get back just the page’s content as Markdown.

Get started

From an empty machine to your first page of Markdown in about a minute.

Run the container
The image is published for linux/amd64 and linux/arm64. The volume keeps the job queue and result cache across restarts.
Run Huntsman with Docker
```
docker run -d --name huntsman -p 127.0.0.1:8080:8080 \
  -v huntsman-data:/data ghcr.io/tulipfarm/huntsman:latest
```

Scrape a page

Send a URL to /v1/scrape and get back the page’s main content as Markdown, plus its links.

Scrape example.com

curl -s localhost:8080/v1/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://example.com"}'

Example response, shortened

{
  "success": true,
  "data": {
    "markdown": "This domain is for use in documentation examples …",
    "links": ["https://iana.org/help/example-domains"],
    "metadata": {
      "title": "Example Domain",
      "statusCode": 200,
      "engine": "lightpanda"
    }
  }
}

Size it for your machine
HUNTSMAN_WORKERS sets how many pages are scraped at once. By default Huntsman sizes it to the machine: 16 per CPU core, capped at one per 96 MB of memory, so 2 vCPU and 4 GB gets 32. Set it yourself to go further; these are the highest values that held up in the benchmark above. Memory grows over runs of many hours, so leave headroom for long-running servers.
Highest tested HUNTSMAN_WORKERS per machine size
Machine Tested max
Small 2 vCPU · 4 GB 45
Medium 4 vCPU · 8 GB 75
Large 8 vCPU · 16 GB 129
Run Huntsman with 64 workers
```
docker run -d --name huntsman -p 127.0.0.1:8080:8080 \
  -v huntsman-data:/data \
  -e HUNTSMAN_WORKERS=64 \
  ghcr.io/tulipfarm/huntsman:latest
```

Highest tested HUNTSMAN_WORKERS per machine size
Machine	Tested max
Small 2 vCPU · 4 GB	45
Medium 4 vCPU · 8 GB	75
Large 8 vCPU · 16 GB	129

Lock it down for production

Pin a version and set HUNTSMAN_TOKEN. Every endpoint except /health then needs the token as a bearer header. Private, loopback and link-local addresses are blocked by default.

Run a pinned version with a token

docker run -d --name huntsman -p 8080:8080 \
  -v huntsman-data:/data \
  -e HUNTSMAN_TOKEN="$(openssl rand -hex 32)" \
  ghcr.io/tulipfarm/huntsman:0.1.0

Scrape with the token

curl -s https://scraper.example.com/v1/scrape \
  -H "Authorization: Bearer $HUNTSMAN_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://example.com", "onlyMainContent": true}'

API

Requests and responses follow Firecrawl’s /v1 shapes, so most clients only need a new base URL. Unknown fields are rejected, so typos fail loudly.

Endpoints
Endpoint	What it does
POST `/v1/scrape`	Scrape one URL and return Markdown and links
POST `/v1/batch`	Queue 1–1,000 URLs with shared scrape options
GET `/v1/jobs/{id}`	Status and result of a queued job
POST `/v1/crawl`	Crawl a site up to a page limit and depth
GET `/v1/crawl/{id}`	Crawl progress and pages, paged with ?offset=
GET `/health`	Liveness and storage check

Coming from Firecrawl? Keep calling /v1/scrape. Huntsman deliberately does less: it returns Markdown and links only, with no screenshots, HTML, JSON extraction, actions or search.

Crawl a site

Start a crawl and read its pages

curl -s localhost:8080/v1/crawl \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://example.com", "limit": 50, "maxDepth": 2}'
# → {"success": true, "id": "…"}

curl -s 'localhost:8080/v1/crawl/<id>?offset=0'

Configuration

Everything is an environment variable. The most used ones are below; the full list is in the README.

Common settings
Variable	Default	Meaning
`HUNTSMAN_WORKERS`	`16 per CPU`	Pages scraped at once, capped at one per 96 MB of memory. Container limits count (see step 3 above).
`HUNTSMAN_PER_HOST`	`2`	Most pages scraped at once from a single site, so one site isn’t flooded. Max 64.
`HUNTSMAN_TOKEN`	`empty`	Bearer token; empty turns auth off
`HUNTSMAN_TOKENS`	`empty`	Named keys, name:token,... One per client, so you can revoke one without the others
`HUNTSMAN_CACHE_TTL`	`1h`	Default result cache age, up to 168h
`HUNTSMAN_OBEY_ROBOTS`	`true`	Respect robots.txt
`HUNTSMAN_QUEUE_CAPACITY`	`10000`	Queued jobs before 429