Product Image Recognition for Retail Shelves: How We Reached 89% SKU Classification Accuracy

Product image recognition, identifying the exact SKU in a shelf photo by its packaging, colors, and label design, is the workhorse artificial intelligence task behind retail execution audits, planogram compliance, and cashierless stores. This outline and case study shows how we built a production product recognition system for exactly that task: 89% top-1 accuracy on real retail shelf photos, evaluated on the RP2K dataset. For context, base CLIP hits 41% on the same benchmark and Fashion CLIP hits 50%. What follows is the architecture, the training approach, and the evaluation methodology behind that 48-point gap.

Width.ai created a new retail product image classification model that outperforms the SOTA results from CLIP and Fashion CLIP on the most popular dataset in the domain. These models are commonly used in product matching use cases where photos are taken with lower resolution and zero control for noise and image angles.

If you are unfamiliar with product matching on retail shelves, I highly recommend you start with our breakdown of how AI product recognition works in 2026 to understand the entire pipeline, and why we use a similarity based approach for SKU image matching on store shelves over a class based system.

‍

I also recommend taking a look at our previous version of this product SKU image classification model that does a great job of outlining the process for reaching these benchmarks. This workflow is where we went from CLIP to Fashion CLIP with customizations to the architecture to fit this domain.

When to Use Similarity-Based Product Image Recognition

Similarity-based product image recognition, embedding each SKU as a vector and running a visual search against the catalog of multiple products, is the right architecture for retail catalogs that change on a consistent basis. Such a system sidesteps the retraining cost of class-based classifiers, handles new SKUs by adding multiple images per SKU to the database instead of retraining the model, and scales to catalogs in the millions. This is the approach behind cashierless stores, smart carts, shelf-audit systems, and every serious CPG retail execution platform.

‍

The distinction matters most when the photos are low quality. If your products sit in controlled studio conditions, simpler image recognition technology can look fine. The moment image data comes from real shelves, phone cameras, and field reps, with blur, odd angles, and occlusion, visual search on fine-tuned deep learning embeddings is the approach that holds up, and the numbers in this case study show exactly how large that difference is. Traditional solutions struggle with this, and anyone who has built this knows the sku matching portion is the big lift. 

A class based system can still be the right choice when the catalog is small and frozen: a few hundred SKUs that never change, photographed under consistent conditions. Retail rarely looks like that. Assortments churn weekly, packaging refreshes make the same product look different mid-season, and promo variants appear without warning, so the database-backed approach is what keeps retail product recognition accuracy stable while the catalog moves underneath it.

How a Product Recognition System Combines Object Detection and Visual Search

This case study focuses on the specific sku recognition step, which in production sits inside a larger product recognition system built from deep neural networks with distinct jobs. Product detection comes first: a full shelf photo arrives as raw image data, and object detection models scan the frame and crop out every individual product. Object recognition comes second, and it is the step this article benchmarks: each cropped product runs through our fine-tuned embedding model, which encodes the image features that identify a SKU (packaging shape, colors, label design), places it in the same embedding space as the catalog, and runs a visual search to find the closest match.

The reason to understand the split is that the two halves fail differently. Object detection techniques are largely a solved problem, with modern deep learning object detectors working well off the shelf after light tuning. Object recognition at the SKU level is the challenging task where generic models collapse, which is why the rest of this case study focuses on that specific aspect. When artificial intelligence platforms advertise image recognition technology for the retail industry, the product detection half is usually commodity; the visual search half is where accuracy is won or lost.

Where OCR Fits Alongside the Image Model

The image is the primary signal in product image recognition, but it is not always the deciding one. When two variants are visually near-identical, the same product in regular and zero sugar, the difference often lives in a few characters of printed text, and optical character recognition settles what the packaging alone cannot. Google is the only major platform that formalizes this split, shipping a Product Recognizer for visual identification and a separate Tag Recognizer for reading price tags and labels. In our builds the same idea shows up as a disambiguation step: the deep learning embedding model proposes the closest SKUs from their image features, and text on the packaging breaks ties between look-alike variants before the match is finalized.

‍

Let's start with a bit of an introduction to the key pieces of the equation.

What is Fine-Tuning?

Fine-tuning in deep learning involves slightly adjusting the parameters of pre-trained machine learning models for a new, related task. It's beneficial for several reasons:

1. Resource Efficiency: Fine-tuning reduces the need for extensive computational resources and time as it leverages pre-existing learned patterns.
2. Transfer Learning: It capitalizes on the concept of applying knowledge from one problem to a related one, such as using a model trained on general images to recognize specific items.
3. Limited Data Handling: Fine-tuning helps prevent overfitting when dealing with smaller datasets and fewer images per class, as the model has already learned general image features from its initial larger dataset training.
4. Iteration Speed: Allows you to quickly iterate models in an evaluation framework to understand where they stand for a specific task.
5. No need for hand-crafted features: Old school methods require you to build for recognizing different objects and training for recognizing specific features. This is an awful approach that does not scale into different categories of images, and is extremely time consuming.   

The CLIP model, known for its capacity to comprehend and link text and images, is trained on a vast internet corpus. However, its generalized training may not fully equip it to handle specific or specialized content. To maximize CLIP's potential for a particular task or domain, fine-tuning is essential.

‍

Fine-tuning is a pretty standardized part of building product recognition systems. Most use cases have unique variables such as setting, camera quality, and product count that create variations of how to accomplish the task. The goal of fine-tuning in this specific part of product matching is to better match the cropped products in the environment to product records in the database. This database has the classes of SKUs we know exist.

Dataset for Evaluation

The dataset we focused on optimizing for is RP2K: A Large-Scale Retail Product Dataset for Fine-Grained Image Classification. It contains more than 500,000 images of retail products on store shelves belonging to 2000 different products.

This dataset is awesome for product matching use cases as the images used come from a high noise environment in a cropped from the shelf format. Most of the datasets companies use for training and testing their product matching systems come from Open Food Facts, which are stale product images with very little noise. While these are pictures of the products you want to recognize, they do not look like the products that come from the shelf in terms of angle, size, and brightness. You might see a super high accuracy on the dataset from Open Food Facts, then move to a real product use case and see your accuracy evaporate.

You can see there is a ton of variation in each product's appearance: color, angle, camera quality, blur and other features that will show up in real retail product recognition. The logos are way easier to see in simple hand held products or ecommerce product images. We used to try to augment this noise to improve the mapping of the clean training dataset to how the images will actually look when compared for similarity. Here's an example of the image used on Open Food Facts, and it's clearly cleaner to view than what would be seen on a shelf.

‍

Comparing our result to the RP2K dataset will be a much better representation of the real world use of the product matching. It also matters that all the images in the evaluation come from a real retail store environment rather than a mix of studio and field shots, so the same product shows up at different angles, distances, and lighting levels, exactly as it would across store visits.

Model Performance & Comparison

Before we get into the details of the results of our new model let's look at the results of the most common models used for product similarity in the product recognition use case. We want to see how well CLIP and Fashion CLIP are at finding matching SKUs in the database. Take a look at the chart below to see how they perform in this task.

‍

We can clearly see that for the kind of images that are present in the RP2K dataset, and real product matching use cases, it's very challenging for the baseline CLIP model and the fine-tuned Fashion CLIP model to perform well. In our evaluation CLIP reached 41% while Fashion CLIP reached 50% Top-1 Accuracy. This means that when we provide an input cropped product image the correct result is returned as the top result 41% & 50% of the time. As you can imagine, these numbers go up as we expand the parameter to the correct result being in the top 3 or top 5 returned results. That being said, it's concerning that the correct result appears in the top 3 results under 65% of the time with either model, and CLIP never moves past 60%.

Let's break down the evaluation method step by step (Top-K Ranking):

1. Creating Embeddings for Training Images:

• First, we have a set of images in our training dataset, each belonging to a specific class or SKU
• We process each image through a neural network to obtain a unique numerical representation called an "embedding" for that image.
• These embeddings are then stored, along with their respective class labels. So, for each training image, we know both its embedding and its class label.

2. Computing Embeddings for Test Images:

• Next, we have a separate set of images in our test dataset, and we want to evaluate how well our model can recognize and classify these test images.
• Similar to the training images, we process each test image through the same neural network to obtain an embedding for each of them.

3. Calculating Cosine Distances:

• Now, for each test image, we calculate the cosine distance between its embedding and all the embeddings of the training images that we computed and stored earlier.
• The cosine distance is a measure of similarity between two embeddings, with lower values indicating greater similarity.

‍

4. Finding the Nearest Neighbor:

• Among all the training image embeddings, we identify the one with the smallest cosine distance to the embedding of the test image. Essentially, we are finding the "nearest neighbor" in the training set for each test image.

5. Comparing Class Labels:

• After finding the nearest neighbor, we check the class label associated with that training image.
• We then compare this predicted class label with the actual class label of the test image.

6. Evaluating Correctness:

• If the predicted class label matches the actual class label of the test image, we consider this a correct classification or ranking for that test image.
• We repeat this process for all test images and keep track of how many times the predicted label matches the true label.

In summary, this evaluation method assesses the performance of a model by checking how well it can recognize and classify images in a test dataset based on the similarity of their embeddings to those of the training dataset. It measures accuracy by comparing the predicted labels to the true labels of the test images. This approach is often used in tasks like image retrieval or image classification to evaluate the quality of a model's representations and its ability to generalize to new data.

These embedding models can sometimes give the illusion of excellent performance if the same image is used both during the initial embedding calculation and in a subsequent reverse image search. In such cases, there would often be a 100% match, making it challenging to accurately assess the model's true capabilities.

Fortunately, the dataset we selected for evaluation has a balanced distribution of classes, approximately 2,388 classes in both the training and test sets. Moreover, on average, there are five different image samples available for each class. This balanced and diverse dataset helps us avoid the issue of overestimating the model's performance due to exact image matches during testing, allowing for a more reliable evaluation.

Our results with our new model

Our brand new model reached an accuracy of 89% in terms of Top-1 retrieval compared to the 41% & 50% above. This comes from breakthroughs in how we utilize the weights of the model and how we set up our hyperparameters. The ability to train this architecture without the need for large amounts of data augmentation and evaluate on a dataset that fits with the real world use case makes it easier for us to iterate this accuracy forward than with the other two models.

‍

The new model has a much deeper understanding of the products in a retail environment where the images are not clean and the quality is not always high. This is the point where most of these products fall apart if they don't already have enough data to train on through real world image collection. This model gives us an elite starting point that only improves from there.

It is worth being precise about what 89% top-1 means operationally, because production systems are not built on the top-1 number alone. The similarity score attached to every match, its distance in the embedding space, is a confidence measure, so such a system sets a threshold: high-confidence matches flow straight through, and the small band of low-confidence results routes to a top-3 or top-5 pick list for a quick human confirmation. Those corrections feed back into the catalog as new reference image data, which is how accuracy keeps climbing after launch. A model that starts at 89% with this loop in place behaves very differently over a year than an off-the-shelf API frozen at 41%.

Where This Technique Runs in Production

The pattern in this case study is not a lab exercise; this product recognition technology is the core behind the retail store systems shoppers already use. Image recognition software platforms like Trax and Focal Systems run product detection and retail product recognition on fixed cameras and associate photos inside retail stores to measure out-of-stocks, share of shelf, and planogram compliance. Cashierless stores run it continuously against live camera feeds (like cctv cameras), from Amazon's Just Walk Out to Trigo's retrofits for grocers like REWE and ALDI Nord. Smart carts like Instacart's Caper Cart and the Amazon Dash Cart run it at the basket edge, fused with weight sensors. Different products, same job: match a messy, real-world product photo to the right SKU in a large catalog, fast and at high accuracy.

The surfaces differ in their constraints more than their mechanics. Product detection and matching for shelf audits can run as batch jobs in the cloud, while cashierless stores and smart carts need answers in milliseconds on edge hardware, running across various electronic devices from fixed cameras to handhelds and phones, which is where the size of object detectors and embedding models starts to matter as much as raw accuracy. The object recognition core stays the same across all of them, which is why the evaluation approach in this case study translates directly to each surface.

From SKU Matches to Retail Execution Metrics

For CPG and retail execution teams, the SKU match is the raw material, not the deliverable. Every metric the category talks about is an aggregation of what the image recognition technology outputs, and this case study measures that output directly. Object detection gives each product's shelf position and object recognition gives its identity; facings are counted by grouping matched SKUs along a shelf row. Share of shelf is your brand's facings divided by the category's, and brand recognition rolls the same matches up a level when the exact SKU is uncertain but the brand marks are clear. Out-of-stock detection is a planogram position where no matching SKU appears. Planogram compliance is the diff between the realogram, what the photo says is actually on the shelf, and the planogram the retailer agreed to. Promo compliance is the same comparison run against displays and price tags.

That chain is exactly why the product recognition accuracy underneath matters so much. Every point of top-1 error propagates into every downstream metric at once: phantom out-of-stocks, wrong share-of-shelf numbers, false compliance flags that burn field reps' time. A retail industry platform reporting execution metrics off a 50% accurate matcher is generating noise with a dashboard on top. The 89% number, and the human-in-the-loop routing described above, is what makes the reporting layer trustworthy.

Interested in implementing product recognition?

Width.ai builds custom product recognition systems that handle product detection, object recognition, and SKU matching in retail environments, for products, displays, and other items. We've reached 90%+ accuracy in a ton of domains and have scaled these systems up to 3.5 million SKUs. We'd love to chat about your product matching or warehouse automation use case!

Lets Talk

Frequently Asked Questions

What is product image recognition?

Product image recognition is the computer vision task of identifying the exact product or SKU in a photo by its visual appearance: product packaging shape, colors, label design, and a brand’s logo, with no barcode required. It is the image recognition technology behind shelf audits, planogram compliance, cashierless checkout, and product matching.

How is product image recognition different from generic image recognition?

Generic image recognition software identifies categories: a bottle, a snack bag. Product image recognition identifies the specific SKU among thousands of near-identical variants, a fine-grained deep learning problem and a challenging task for generic pre-trained models. In our evaluation, base CLIP managed just 41% top-1 on real shelf photos.

What accuracy can product image recognition achieve on retail shelf photos?

On RP2K, real shelf photos with blur, angles, and occlusion, base CLIP reached 41% top-1 accuracy and Fashion CLIP reached 50%, while our custom deep learning model reached 89%. Accuracy is decided less by the base architecture and more by whether the model was tuned on data that looks like your actual deployment conditions.

Can product image recognition handle new SKUs without retraining?

Yes, when the product recognition system is built as visual search over embeddings. New SKUs are added by embedding a few photos into the catalog database, with no retraining, which is what makes this architecture practical for the retail industry's weekly assortment changes.

Can I identify a product from a picture?

Yes. Consumer artificial intelligence tools like Google Lens identify well-known products from a single photo, and they work well for popular items with distinctive packaging. Business use cases are harder: identifying one SKU out of a private catalog of thousands, from a blurry shelf photo, needs a product recognition system tuned on that catalog, which is the system this case study describes. Systems like Google Lens are not fine-tunable, so your ability to improve the system in production is extremely limited. 

Can ChatGPT or other vision models do product image recognition?

Vision-language models like GPT-4V, Claude, and Gemini can describe a product and often name well-known items, which makes them useful for long-tail, low-volume identification. They fall short for catalog-scale work: they cannot reliably pick one SKU out of thousands of near-identical private-catalog variants, and per-image cost and latency rule them out for continuous shelf or cart monitoring. In production they complement, rather than replace, a fine-tuned deep learning model. Creating an image search engine with real images of distinct products allows you to retrieve the similar products quickly, and does not need fine-tuning every time you add new skus.

‍