Back to portfolio
ToolStudy / personal

Real Estate Scraper

Python tool for collecting and normalizing Storia listings, with parallel scraping, deduplication by listing ID, relational storage, a filterable REST API, and walking, driving, and public-transit distance calculations.

Real Estate Scraper

Core stack

Python
Django
Django REST Framework
django-filter
MySQLSQLiterequestsBeautifulSoup4lxmlThreadPoolExecutorOSRMGoogle Maps Directions API

The challenge

The challenge was turning the semi-structured data of an external Next.js application into a stable, filterable model without relying exclusively on fragile CSS selectors. At the same time, re-scraping had to avoid duplicates, update existing data, and enrich each property with travel distances useful for selection.

The solution

The solution reads Storia's data from __NEXT_DATA__, normalizes the relevant fields, and processes detail pages concurrently through a ThreadPoolExecutor. The Django ORM uses ad_id as the unique identifier, update_or_create for re-scraping, and normalized tables for characteristics, amenities, images, and distances. DRF provides pagination, filtering, and sorting, while the distance service combines Haversine, OSRM, and optionally Google Maps Directions.

Notable complexity

Real Estate Scraper is a study project built to turn real estate listings into a queryable dataset rather than a one-off export. The Django pipeline starts from Storia's result pages, automatically detects the number of pages from __NEXT_DATA__, extracts listing previews, and then opens the detail pages to fetch the title, description, price, surface area, floor, coordinates, neighborhood, street, the seller's public data, characteristics, amenities, images, and the timestamps supplied by the source.

The scraper uses a requests session with connection pooling and retries for 429 responses and 5xx errors, short randomized delays, and a ThreadPoolExecutor with a configurable number of workers for parallel processing of detail pages. Interruption via Ctrl+C is handled explicitly in both the standalone variant and the Django command.

The main persistence layer is modeled relationally in Django: Listing keeps the frequently queried fields, while Characteristic, Feature, Image, and Distance are separate tables with relationships, uniqueness constraints, and dedicated indexes. Re-scraping uses update_or_create on ad_id, so the same listing is updated instead of duplicated. Associated characteristics, amenities, and images are rebuilt and bulk-inserted to keep a coherent representation of the current state.

In addition to Haversine straight-line distance, the distance service can query OSRM for walking and driving routes and, if a key is configured, Google Maps Directions for public transit. Results are persisted per listing and reference point and can be used directly in the API filters.

Django REST Framework exposes paginated listing and full detail, filtering by seller type, advertiser type, neighborhood, price range, and travel durations, plus multi-select filters for amenities and characteristics. Separate endpoints return the available filter values, recalculate a listing's distances, and can manually trigger a new scraping run. This trigger runs synchronously in the API process; the repository does not contain an active Celery worker.

The repository also keeps an earlier stage of the project in the base/ directory: a BeautifulSoup scraper, parallel processing, and SQLite persistence, with JSON/CSV export and a shell administration script. The script references a local web interface, but the start_web.py file is not present in the backup, and the current repository contains no frontend components. The API and filter structure suggests the intent of a home-selection tool by cost, area, amenities, and travel time, but that UI is not claimed as implemented.

The analyzed version is a development project, not a production-ready configuration: the Django settings keep DEBUG on, a development secret and MySQL credentials in code, there is no authentication or authorization for POST actions, and OSRM requests disable TLS verification. These are security limitations that must be addressed before exposing the service to the internet.

Key results

  • Storia scraper that automatically detects pagination and extracts previews directly from __NEXT_DATA__
  • Configurable parallel processing of detail pages, with connection pooling and retry for 429 and 5xx errors
  • Relational normalization for listings, characteristics, amenities, images, and distances
  • Upsert on ad_id to update already-known listings without duplicating the main record
  • Bulk insertion for characteristics, amenities, and associated image variants
  • Calculation and persistence of straight-line, walking, driving, and public-transit distance when the integration is available
  • Paginated REST API with listing, full detail, sorting, and filters by price, neighborhood, seller, amenities, characteristics, and travel duration
  • Endpoint for available filter values and an endpoint for recalculating a listing's distances
  • Django CLI command with configurable URL, page count, and workers, plus controlled shutdown on interruption
  • Standalone prototype kept separately, with SQLite, local statistics, and JSON/CSV export

Related projects

Want something similar?

Start with the estimate wizard — answer a few questions and get a realistic price range and timeline.