A US hotel-intelligence company
Market data collected every night for a hotel-intelligence company
Collectors with proxy rotation feeding MariaDB, running every night since 2025. 8M+ requests a month against the sources the collectors watch.
Baba International, a Turkish ecommerce sourcing business. August to October 2023.
Product-import tools for a Turkish sourcing business
Baba International buys products through Trendyol and needed product lists pulled from it in a shape its catalogue import could take. We built a Flask API that returns each product as JSON or CSV with a per-item result, and specified the Chrome extension to sit on top.
Baba International in figures
- 100product URLs per batch
- 5documented response contracts, with a result per item
- 3paid milestones, August to October 2023
- Batch of product URLs
- Flask scraping API
- JSON or CSV per product
- Per-item results
- Chrome extension specification
Screens from this system are being added.
The situation
Baba International buys products through Trendyol, a Turkish marketplace, and needed product lists pulled from it in a shape its catalogue import could take. Bulk catalogue jobs fail halfway and report a single status, so nobody can tell which items came back clean and which need a retry.
What we built
- A Flask API that scrapes requested product URLs and returns url, title, description, price, attributes, barcode and images as JSON or CSV
- Batches of up to 100 URLs per request
- Per-item success, partial and failure results rather than one job-level status, across five documented response contracts
- CSV attachments for Baba International's catalogue import
- A requirements specification for a Chrome extension: product-page detection, CSV export, Turkish to English, French and German translation, retry and error handling
What changed
A batch that stops halfway now reports which products are usable, and the import backend was accepted as the third milestone at the end of October 2023.
Along the way
Fixed-price contract started for a scraping application delivering product details as CSV.
First milestone approved: the initial scrape.
Repository created for the batch product scraper.
Planning and initial set-up milestone approved.
Backend for product import approved as the third milestone.
Contract closed.
Read next
Our own engineering
A retail catalogue kept in step with Shopify, price by price
A scheduled Python pipeline that reads Costco product pages, stores them in MongoDB and creates and updates the matching Shopify products, prices and inventory by SKU.
A real-estate researcher working Montana property records
Every lot in the Yellowstone Club, as one dataset
Every house and lot in the Yellowstone Club community, extracted from Montana's cadastral API and its HTML pages into a single CSV, with a notebook that measured how the extraction behaved. 3 days from the approved sample to the full dataset, November 2023.
Elsewhere on the site
Related
- PracticeThe Data practiceWhat the practice covers, what it costs to start, and the work behind it.
- The workDemosThe demos you can open now: the two that run in your browser, the hosted prototypes, a walkthrough on video and a live cafe site.
- The workThe five practicesBuild, Automate, Data, Growth and Talent: what each one covers, the work behind it, and what it costs.
Where to next
Have a problem in this shape?
Say so in a few sentences. The reply comes from the person who built this one.