Scaling Large Catalogs

For catalogs around 500k SKUs, keep every workflow cursor-based, queue-based, and aggregate-aware.

Database

Queues

Use composite indexes, partition-friendly time-series tables, and daily aggregates.

Separate discovery, scrape, AI, and notification lanes.

Cache discovery results and avoid re-discovering confirmed matches unless evidence changes.

The practical goal is to make dashboards query aggregates, workers process bounded batches, and humans review only the uncertain subset.

Last updated: