What it is
A UK property search engine that aggregates listings from OnTheMarket, Rightmove, and Zoopla into one searchable dataset, with geospatial search on top. Built and run solo, end to end.
What I built
- Three scraping pipelines, one per portal, with anti-bot engineering to keep them running unattended.
- Geospatial search on PostGIS with H3 reverse-geocoding, so queries work by area rather than by postcode string matching.
- Watermark-based incremental sync: daily refreshes fetch only changed listings, partitioned and retry-safe across every UK outcode.
- Dagster orchestration with dbt models over the raw listings, landing in ClickHouse for the analytical side.
Notes on the build
The hard part is not the scraping, it's the refresh. A naive full crawl of every UK outcode is both slow and hostile to the source sites, so the pipeline tracks a watermark per outcode and only pulls what moved. Partitioning by outcode means a failed region retries on its own without re-running the country.
Source is private.