FPP Scraper Prototype
ScrapingBee → Google search → property page → DB (property + html + pivot + parsed comps).
Address
Site
redfin
xome
Find
Scraper mode
Eager
— Fetch all comps inline (parallel)
Queued
— Background jobs
(coming soon)
Lazy
— Defer comps until clicked
Eager mode.
Fetches every comparable home in parallel right after the target page is parsed, persists each as a first-class property, and returns once all data is in the DB. Slowest user-perceived latency (~10–15 s typical) but the richest single-call API. Reuses any property already in the DB whose data_updated_at falls inside the freshness window.
Queued mode.
Returns the target as soon as it's parsed and dispatches a background job per comp. The form would return in ~5 s; comps fill in over the following seconds as workers pick them up. Best UX for high-traffic deployments. Requires a queue worker (Redis or database driver). Selecting this mode today raises a LogicException — the picker is locked until the worker side ships.
Lazy mode.
Returns the target right after it's parsed and leaves comps un-enriched (parsed summary cards stay in the payload but no Property row is created for each comp). Fastest and cheapest path when you only need the target up front. Each un-enriched comp in the table renders with a "↓ fetch" chip — clicking it triggers an on-demand single-property fetch, persists the comp, and lands on its detail page. Subsequent clicks on the same comp hit the freshness cache.
Cache
Enabled
— Reuse fresh DB rows; skip ScrapingBee when within the freshness window
Disabled
— Force a live fetch for the target and every comp
Comps
Include
— Process per the selected mode
Skip entirely
— No comp data in payload (mode is moot)
In progress…