A US hotel-intelligence company
Market data collected every night for a hotel-intelligence company
Collectors with proxy rotation feeding MariaDB, running every night since 2025. 8M+ requests a month against the sources the collectors watch.
Our own engineering. September 2024 to February 2025.
A retail catalogue kept in step with Shopify, price by price
A store reselling retail products needs its Shopify catalogue to track the retailer's prices and stock, and by hand that work drifts the moment anyone stops. We built the pipeline ourselves: extraction into MongoDB, a threaded price and stock refresh, and products created and updated in Shopify by SKU.
Source catalogueExtracted with Selenium and BeautifulSoup
MongoDBEvery product, every price seen
Threaded schedulerPrice and stock refresh, with retries
ShopifyCreate and update by SKU: variants, images, collections, inventory
Screens from this system are being added.
The situation
A store that resells retail products needs its Shopify catalogue to track the retailer's prices and stock. Done by hand that work is exact, repetitive and endless, and it drifts the moment anyone stops. We built the pipeline on our own account so the sync could run on a schedule instead of a person.
What we built
- Product extraction from Costco with Selenium and BeautifulSoup
- Normalised storage in MongoDB, with a threaded scheduler refreshing price and stock
- Shopify product creation and updates keyed by SKU, with variants, images, metafields, collections and inventory levels
- Retries and bulk operations, with rate-limit handling against the Shopify Admin API
- JSON export for downstream use
- An earlier wholesale-catalogue prototype, begun in December 2022: crawl vendor catalogues, normalise them, make them searchable, and handle pricing, stock and ordering, with an ecommerce website requirements specification written in October 2023
What changed
The store's catalogue moves when the retailer's does, and a failed refresh retries itself rather than waiting for someone to notice.
Along the way
Wholesale-vendor catalogue prototype begun: a crawler that pulls vendor catalogues, normalises them and makes them searchable.
Requirements written: catalogue, price and stock tracking from a retailer into a Shopify store.
Pipeline repository created; commits run through to February 2025.
Last push: threaded scheduler, product extraction, MongoDB storage, price and stock refresh, retries, bulk create and update, inventory sync and JSON export all in place.
Read next
A real-estate researcher working Montana property records
Every lot in the Yellowstone Club, as one dataset
Every house and lot in the Yellowstone Club community, extracted from Montana's cadastral API and its HTML pages into a single CSV, with a notebook that measured how the extraction behaved. 3 days from the approved sample to the full dataset, November 2023.
A publisher of scraping tutorials
A Zillow scraper, and the tutorial written around it
A commented Python scraper for Zillow listings with incremental CSV output, and a tutorial article that walks beginner and intermediate developers through it.
Elsewhere on the site
Related
- PracticeThe Data practiceWhat the practice covers, what it costs to start, and the work behind it.
- The workDemosThe demos you can open now: the two that run in your browser, the hosted prototypes, a walkthrough on video and a live cafe site.
- The workThe five practicesBuild, Automate, Data, Growth and Talent: what each one covers, the work behind it, and what it costs.
Where to next
Have a problem in this shape?
Say so in a few sentences. The reply comes from the person who built this one.