Our engineering
A retail catalog pipeline that keeps a Shopify store current
Scrapes Costco product data on a schedule, normalizes and stores it, and syncs price and stock to Shopify.
Source not public
Our own engineering. We will screen-share the code, the architecture, and the tests with anyone seriously evaluating us; publishing the repository invites the scrapers we spend our days defeating.
- Recorded outcome
- Complete pipeline on record: scrape, normalize, store, schedule, and sync, with retries and bulk operations
- Status
- Delivered
- Practice
- Automate · Data
- Sources
- 1 figure
Qelv engineering record
Our engineering
Retail ecommerce
engineering record
The problem.
A store selling retail products needs its Shopify catalog to track a source retailer's prices and stock. Done by hand, that work is exact, repetitive, and endless.
What was built.
- Product scraping from Costco with Selenium and BeautifulSoup
- Normalized storage in MongoDB with scheduled refresh of price and stock
- Shopify product creation, updates, and inventory sync, with retries and bulk operations
- A threaded scheduler for the refresh
- JSON export for downstream use
- An earlier wholesale-catalog prototype, December 2022 to April 2024: crawl vendor catalogs, normalize them, make them searchable, and handle pricing, stock, and ordering, with an e-commerce website requirements specification written in October 2023
Timeline.
Wholesale-vendor catalog prototype begun: a crawler that pulls vendor catalogs, normalizes them, and makes them searchable
Requirements written: catalog, price, and stock tracking from a source retailer into a Shopify store
Pipeline repository created; commits run to February 2025
Last push: threaded scheduler, product extraction, MongoDB storage, price and stock refresh, retries, bulk create and update, inventory sync, and JSON export in place
What this case claims, and what it does not.
On the record
- The workflow and stack, as built
Withheld, and why
Figures that exist internally and are not on this page, with the reason.
- How many products it keeps current, or how often
- A product count appears in the original requirements and nothing in the engineering record confirms it, so no count is published.
- Sales impact
- No store analytics are attached. The pipeline is described as built, which is what the record supports.
Elsewhere on the site
Read next
Where to next
Have a problem in this shape?
Send a short description. The form opens with this case already named, so you only add the part that is yours. We will tell you whether it is worth building at all.