Real Estate Scraper
Python tool for collecting and normalizing Storia listings, with parallel scraping, deduplication by listing ID, relational storage, a filterable REST API, and walking, driving, and public-transit distance calculations.

Core stack
The challenge
The challenge was turning the semi-structured data of an external Next.js application into a stable, filterable model without relying exclusively on fragile CSS selectors. At the same time, re-scraping had to avoid duplicates, update existing data, and enrich each property with travel distances useful for selection.
The solution
The solution reads Storia's data from __NEXT_DATA__, normalizes the relevant fields, and processes detail pages concurrently through a ThreadPoolExecutor. The Django ORM uses ad_id as the unique identifier, update_or_create for re-scraping, and normalized tables for characteristics, amenities, images, and distances. DRF provides pagination, filtering, and sorting, while the distance service combines Haversine, OSRM, and optionally Google Maps Directions.
Notable complexity
Real Estate Scraper is a study project built to turn real estate listings into a queryable dataset rather than a one-off export. The Django pipeline starts from Storia's result pages, automatically detects the number of pages from __NEXT_DATA__, extracts listing previews, and then opens the detail pages to fetch the title, description, price, surface area, floor, coordinates, neighborhood, street, the seller's public data, characteristics, amenities, images, and the timestamps supplied by the source.
The scraper uses a requests session with connection pooling and retries for 429 responses and 5xx errors, short randomized delays, and a ThreadPoolExecutor with a configurable number of workers for parallel processing of detail pages. Interruption via Ctrl+C is handled explicitly in both the standalone variant and the Django command.
The main persistence layer is modeled relationally in Django: Listing keeps the frequently queried fields, while Characteristic, Feature, Image, and Distance are separate tables with relationships, uniqueness constraints, and dedicated indexes. Re-scraping uses update_or_create on ad_id, so the same listing is updated instead of duplicated. Associated characteristics, amenities, and images are rebuilt and bulk-inserted to keep a coherent representation of the current state.
In addition to Haversine straight-line distance, the distance service can query OSRM for walking and driving routes and, if a key is configured, Google Maps Directions for public transit. Results are persisted per listing and reference point and can be used directly in the API filters.
Django REST Framework exposes paginated listing and full detail, filtering by seller type, advertiser type, neighborhood, price range, and travel durations, plus multi-select filters for amenities and characteristics. Separate endpoints return the available filter values, recalculate a listing's distances, and can manually trigger a new scraping run. This trigger runs synchronously in the API process; the repository does not contain an active Celery worker.
The repository also keeps an earlier stage of the project in the base/ directory: a BeautifulSoup scraper, parallel processing, and SQLite persistence, with JSON/CSV export and a shell administration script. The script references a local web interface, but the start_web.py file is not present in the backup, and the current repository contains no frontend components. The API and filter structure suggests the intent of a home-selection tool by cost, area, amenities, and travel time, but that UI is not claimed as implemented.
The analyzed version is a development project, not a production-ready configuration: the Django settings keep DEBUG on, a development secret and MySQL credentials in code, there is no authentication or authorization for POST actions, and OSRM requests disable TLS verification. These are security limitations that must be addressed before exposing the service to the internet.
Key results
- Storia scraper that automatically detects pagination and extracts previews directly from __NEXT_DATA__
- Configurable parallel processing of detail pages, with connection pooling and retry for 429 and 5xx errors
- Relational normalization for listings, characteristics, amenities, images, and distances
- Upsert on ad_id to update already-known listings without duplicating the main record
- Bulk insertion for characteristics, amenities, and associated image variants
- Calculation and persistence of straight-line, walking, driving, and public-transit distance when the integration is available
- Paginated REST API with listing, full detail, sorting, and filters by price, neighborhood, seller, amenities, characteristics, and travel duration
- Endpoint for available filter values and an endpoint for recalculating a listing's distances
- Django CLI command with configurable URL, page count, and workers, plus controlled shutdown on interruption
- Standalone prototype kept separately, with SQLite, local statistics, and JSON/CSV export
Related projects
- Time TrackerReact + Express application that centralizes imported time-tracking data, adds manual hours, applies reporting rules, and generates filters, charts, hour reports, and billing calculations for project groups.
- PDF Auto-PrintPython desktop utility that periodically polls an endpoint, downloads PDF documents, keeps a local history and status for them, and sends them automatically or manually to Windows printers via SumatraPDF.
- QR GeneratorSimple full-stack tool for app distribution through a single QR code: define the iOS/Android links, generate a QR code pointing to a stable address, and redirect the mobile user to the appropriate store.
Want something similar?
Start with the estimate wizard — answer a few questions and get a realistic price range and timeline.