How we set up bread counting and recognition on the conveyor belt with a video camera
Summary
Customer. Bread manufacturer. The company produces 14 popular types of bread: baguettes, 8 grains, Finnish loaf, rustic bread and others.
Task. Count and recognize the released products on the conveyor. At the end of the day display a table with the results.
Result. CamContador recognizes and counts all types of bread, including rejects, with an accuracy of 99.8%. A report on the output is built in real time and sent to the accounting department at the end of the shift.
Situation
A bakery works 24 hours a day without weekends. Every day the enterprise ships 14 kinds of bread to the counters of trade outlets and retail chains.
With such production volumes, discrepancies in product counting are inevitable. For example, the raw materials were used to produce 1000 units of bread, but the package showed 980 pieces. In order to understand the exact number of products produced, the bakery managers decided to install a video product counter.
Previously, technology did not allow counting products on the conveyor belt with the help of a video camera. Traditional solutions: laser, infrared and pulse counters require additional budget for the purchase of equipment, its installation and customization. Therefore, the customer counted the released products at the packaging stage.
Javi Martínez, Project Manager
Today, computer vision technology makes it possible to count moving products right on the conveyor belt. It doesn’t matter whether the bread comes out of the oven in rows on pallets or chaotically scattered on the belt. CamContador counts and recognizes products in any form.
The challenge
The company has two production lines. The first produces 11 types of bread, the second three. The products on the belt are constantly changing: first one type of bread goes down the conveyor belt, a few minutes later another, then a third.
The customer has set the task to recognize and count all the released products. The counter should determine which type of bread goes down the conveyor belt, count its quantity, and at the end of the day output a table with the results.
Solution
The customer sent videos for each type of bread. At the beginning there were 9, but gradually the number grew to 14 types. We selected 30 screenshots for each product, marked them for counting and trained the algorithm to recognize the products.
It took 3 days to prepare and train the neural network. When everything was ready, we launched the video counter in test mode and together with the customer started catching errors.
Errors due to the color of the bread
Because of baking conditions, bread sometimes changes color. One batch can turn out pale, while the next batch is ruddy. An algorithm trained on the pale products may make mistakes on the ruddy ones. They have the same shape but different crust colors.
The customer recorded such errors and sent us the video. We re-labeled the frames and re-trained the model. After that, the errors disappeared.
Similar breads in the frame
When the customer sent a video of the 14th type of bread, it appeared to be similar to the other two types, only it was slightly lighter. If there are several units of bread walking down the line, the differences are immediately visible. But when a single piece was in the frame, the algorithm became confused.
Additional training of the neural network would not solve the problem. Therefore, we implemented a refinement at the counting level.
Several minutes pass between different batches of bread. This means that if, for example, 24 products are visible in the frame, they are all the same. There cannot be different breads on the conveyor belt at the same time. If the system found one loaf in a batch that was different from the others, we eliminated this error programmatically.
Thanks to this modification, CamContador has learned a logic close to human logic. The counter has memorized that there cannot be different products in a homogeneous mass of bread. If the algorithm finds several different loaves, the counter will still count them together with the main mass of bread.
A pile of bread on a ribbon
Sometimes the bread goes down the line piled in a heap. The loaves lie on top of each other – one lower, one higher, and a third hides underneath. Such piles of bread have to be removed. The camera can’t see the items underneath.
Fortunately, this problem is quickly solved by installing a bar above the conveyor. This is exactly what the customer did. Now, when the bread rides on top of each other, the crossbar drops the top loaf onto the belt. The result is a single layer of bread that is easy to count and recognize.
Technical difficulties
Capture Quality. Usually, when there are many cameras in a production facility, the CamContador simply counts the products – without recognizing them. For example, this is the case when counting eggs or poultry carcasses. But on this project, the situation was more complicated.
Although there were only two cameras, we had to not only count but also recognize 14 types of products. First, to avoid overloading the network and the processor, we drastically reduced the quality of the video stream. We cut the bitrate and frame rate of the cameras. However, later together with the customer’s representative we decided to improve the picture quality. They raised the bitrate to 4000 kbps and FPS to 20 frames. Thanks to this, images became clearer and counting accuracy increased.
Shooting angle. Initially, one camera was hanging too far away from the production line. This angle made it difficult to recognize the products. There were three types of bread on the tape: white, black and gray. The algorithm made a mistake on the gray bread: the counter assigned it to the black bread and then to the white bread.
The customer lowered the camera so that the loaves could be seen better. We retrained the neural network on the frames from the new angle and restarted the counter. After that, the gray bread recognition errors disappeared and the counting accuracy exceeded 99%.
Problems with Windows 11. Typically, customers use Windows 10 and Ubuntu operating systems. CamContador is fully compatible with these systems. The bakery was the first customer who came with Windows 11. We thought that technically version 11 was not much different from version 10, but it turned out to be different.
There was an unforeseen problem with the drivers for the NVIDIA graphics card and CUDA library. For some unknown reason, the graphics card load was constantly jumping from 30 to 100%. At the moments of peak load it did not have time to process the stream. This caused counting errors.
For a month we changed settings and reinstalled drivers. The problem did not disappear. Then, after consulting with the customer, we decided to switch to the free Ubuntu OS.
The transition was quick and easy. After installing Ubuntu and restarting the counter the video card errors disappeared. For today the program has been running stably and reliably for six months.
The result
Today CamContador successfully recognizes and counts 14 types of bread. Technically, the algorithm even counts 15 types of products, including production rejects.
The cameras are installed on two production lines and count both product flows. The counting accuracy is 99.8%.
At the end of the shift, the customer receives a report on the output by product type. These reports help to control the quantity of bread produced. This information then goes to the customer’s accounting department:
From the first communication with the customer to the launch of the counter on the production line took a month. This is longer than usual projects for counting products on the conveyor belt which are launched in a few days. However in this case we did not just count the same products but recognized 14 different types of bread and defects.
Javi Martínez, Project Manager
Bread is a rather specific product whose appearance changes due to baking conditions. After the launch, we retrained the neural network 12 times to reduce the error rate, and as a result the counting accuracy was 99.8%. That is, there are only 1-2 errors per thousand units of bread. This is a very high percentage of accuracy compared to conventional counting equipment. However, for the CamContador meter, 99.7-99.9% accuracy is the usual figure that we show in every project and guarantee in the contract.
Video example of how the counter works:
Do you want to count your products? Write at info@camcontador.com