Every scraper breaks.
The question is what happens next.

The problem

The retailer sells refurbished phones and buys customers' old handsets through a trade-in flow. Both halves depend on one number: what the rest of the market is charging today.

List a refurbished handset €40 above the market and it sits in stock. Offer a customer €40 more than the high-street chains pay for the same trade-in, and the margin is gone before the device has even been graded.

Until this was built, keeping track meant somebody opening a dozen competitor product pages, copying prices into a spreadsheet, then doing it again the following week. That stops working past a few dozen SKUs. It also leaves no history, and it tells nobody when a competitor moves.

What it does

Every night the service visits the competitor page for each tracked device on each marketplace, reads today's price off the page, files it, and works out which marketplace is currently cheapest. If a price jumps sharply, the pricing team gets an email the same night.

The answer is also available over an API, so the retailer's own operational systems can ask what a handset is worth right now and get a live number instead of a stale spreadsheet cell.

The hard part is not the collecting

Anyone can fetch a page and read a number off it. The difficulty in this kind of system is that you control none of your data sources, and they change without telling you.

A marketplace redesigns its product page, your selector matches nothing, and overnight one column of your pricing table quietly goes empty.

So the failure handling is the product. It has three parts.

Reading the price without writing a rule per retailer

The obvious implementation is a hand-written extraction rule per marketplace. Ten marketplaces, ten brittle selectors, and ten things to fix every time one of them redesigns a page.

Instead, each connector sends the collection service the same instruction in plain English: extract the product price as a decimal number, return only the numeric value, use a dot as the decimal separator. Alongside it goes a hint about where on the page to look.

The differences that remain between marketplaces are real ones. One source keeps its price in the structured data embedded in the page. Another needs a couple of seconds for the page to settle before the number appears at all.

Adding the eleventh marketplace is one class

The parts of this system are separated by interchangeable connectors rather than by a network boundary. Each connector registers itself at start-up.

A connector declares which catalogues it belongs to, how to fetch a price, and how it should appear in tables, exports and charts. Everything downstream simply loops over the registered connectors.

So adding a marketplace does not mean editing eleven screens. Each connector contributes its own table column, export column and chart series on its own. The whole change fits in one small class.

Cost is a first-class metric

The retailer buys collection instead of running it. A commercial service fetches the pages through managed proxies and charges per request.

People usually frame that decision as convenience. The bigger reason is visibility. Every fetch has a known price, so the console can show how much of the month's collection budget is gone, and turn the figure red once the quota is nearly used up.

A background job that silently costs more every month is one of the easiest ways to lose money in software. Here the number sits on the same screen as the data it paid for.

The console is fast because nothing is computed when you look at it

Once prices come back, the work runs through a chain of internal events. The system writes prices to history in bulk, recomputes the device's summary of latest values and cheapest marketplace, then runs the volatility check against that summary and sends any alerts. The chart data and every cached table cell for that device are rebuilt in the background.

By the time anyone opens the console, the expensive work has already been done. The screen reads almost entirely from cache.

Every price row gets a time-ordered unique identifier, which keeps a fast-growing history table quick to insert into and quick to read back in order. It is the unglamorous decision that stops a price-history product from getting slower every month it succeeds.

The security gate really blocks

Static analysis, dependency scanning and container-image scanning all run inside the deployment pipeline and report into a central register.

The check is a gate rather than a report: a High or Critical finding fails the build. At deploy time the pipeline pulls secrets from a managed parameter store, separately per project and environment, and keeps the most sensitive values under a restricted path. Nothing sensitive is ever baked into an image.

That is unusual for a system this small, and it is deliberate. An internal tool with an API that other systems trust is exactly the kind of thing that gets forgotten and then exploited.