Recursive Spider (in-process)

action_recursive_spider · action · Utility · Available · v1.0.0

Description

In-process recursive crawler for URL discovery and site mirroring WITHOUT hosting an external service: bounded BFS with dedup, depth limit, hard page cap, worker pool, per-host token-bucket rate limiting and robots.txt caching.

⚙️ Configuration parameters

Fields shown in the editor when configuring the node. Generated directly from the NodeDefconfigFields.

Field	Type	Required	Default	Description
`seeds` Seed URLs	`string`	yes	— https://example.com/, https://example.com/blog	Lista URL di partenza separati da virgola o newline. Devono essere http(s)://. Lo spider farà BFS partendo da questi.
`maxDepth` Profondità massima	`number`	no	`3`	Quanti livelli di link seguire dai seed. 0 = solo i seed. 3 = seed → click → click → click. Max 50.
`maxPages` Pagine massime (hard cap)	`number`	no	`100`	Stop al raggiungimento. Le pagine non visitate restano in `frontier` (output) per resume futuro. Max 5000.
`sameOriginOnly` Solo same-origin	`boolean`	no	`true`	Se ON, i link verso altri host vengono ignorati. Combinare con allowDomains per multi-host controllato.
`allowDomains` Domini consentiti (whitelist)	`string`	no	— example.com, sub.example.com	Override di sameOriginOnly: se valorizzato, accetta link verso questi host (anche sotto-domini). Vuoto = solo origin del seed.
`denyPatterns` Pattern URL da escludere (regex)	`string`	no	— /login, \.pdf$, /admin/	Regex valutate sull'URL assoluto. Separati da virgola o newline. Es. "/admin/, /api/, \.pdf$".
`concurrency` Concorrenza (fetch in parallelo)	`number`	no	`4`	Worker simultanei. 1=seriale, 4=default sano per single tenant, max 16. Più alto = più veloce ma più aggressivo sul server target.
`perHostMinDelayMs` Delay minimo per host (ms)	`number`	no	`500`	Token bucket per-host: garantisce >= N ms fra due request allo stesso host. 0 = nessun rate limit. Default 500ms = ~2 req/sec.
`respectRobots` Rispetta robots.txt	`boolean`	no	`true`	Fetch /robots.txt una volta per host, applica Disallow + Crawl-Delay. Spegnere SOLO se hai diritti sul target.
`timeoutMs` Timeout per pagina (ms)	`number`	no	`20000`	Hard timeout sulla singola fetch. Pagine timeout vanno nei result con error="timeout..." e link=[].
`userAgent` User-Agent	`string`	no	— FlowForge-Spider/1.0 (+https://flowforge.automazionezeli.com)	User-Agent dichiarato. Default RFC-compliant. Override SOLO per propri siti che bloccano UA generici.

💡 Configuration example

JSON snippet of the node as it appears in the workflow. Values are derived fromdefaultValue and from required parameters.

{
  "id": "node-action_recursive_spider-1",
  "defId": "action_recursive_spider",
  "label": "Recursive Spider (in-process)",
  "config": {
    "seeds": "https://example.com/, https://example.com/blog",
    "maxDepth": 3,
    "maxPages": 100,
    "sameOriginOnly": true,
    "concurrency": 4,
    "perHostMinDelayMs": 500,
    "respectRobots": true,
    "timeoutMs": 20000
  }
}

🔗 Related nodes in the same category

Ready to use Recursive Spider (in-process)?

Available now on all FlowForge plans. Try it free without a credit card.

Start free Browse all nodes