Bin picking and 3D machine vision – when does part handling need AI?

Reading time: approx. 8 minutes

To a person, a bin full of loose parts is nothing special. You reach in, pick one up, turn it around in your hand and put it into the machine. For a robot, the same task is one of the hardest jobs on the shop floor. It does not know where the part is, how it lies, what is covering it, or whether it can reach it at all. This is what bin picking is for: robotic picking from an unsorted bin, which requires 3D machine vision and, more and more often, artificial intelligence.

I wrote this article for production managers, development engineers and anyone facing an automation investment who wants a clear picture of what makes a bin picking cell work in practice. When is classic, CAD-based recognition enough, and when is it worth adding AI to the system? At the end you will find a decision chart and a short list of questions worth going through before you ask for a quote.

The bin will not get any tidier

In most plants, parts arrive the way it suits the supplier or the previous process step: in crates, mesh boxes, KLT containers, piled on top of each other. Someone has to take them out and place them in the right orientation into a machine, onto a tray or onto the assembly line. In many places an operator still does this. It is monotonous, ergonomically demanding with heavier parts, and typically the first spot where staffing becomes a problem at shift changes or during holidays.

The classic automation answer has been to order the parts before the robot touches them. Vibratory feeders, trays, conveyor separation, special supplier packaging. These work well as long as the part is small, the number of variants is low and someone is willing to pay for the trays. For larger, heavier parts with complex geometry, ordering them is expensive and takes up space, and every new product means new tooling.

This is where bin picking comes in. The robot does not wait for an ordered part. It looks into the bin with a 3D camera, finds a part it can pick, calculates its pose, grips it and places it in a known position. It sounds simple on paper. Anyone who has tried to commission one knows there is a lot more work involved in reality.

How does a bin picking cell work?

It is worth breaking the process into steps, because problems show up step by step, and technology decisions are also made step by step.

3D imaging: the camera creates a point cloud of the bin contents, meaning every pixel also has spatial coordinates. The most common options are structured light (pattern projection) cameras, stereo cameras, ToF (time-of-flight) sensors and laser scanners. Structured light is one of the most widely used for bin picking, because it gives good accuracy at medium distances. ToF is faster but typically coarser, and with laser line scanning either the sensor or the part has to move.
Object recognition and segmentation: the system has to decide which points in the cloud belong to one part. When parts touch, lie on top of each other or get tangled, this is the hardest step.
Pose estimation: the position of the part has to be determined in all six degrees of freedom (three translations, three rotations). The classic way is to match the part's CAD model or a reference point cloud to the measured points, for example with the ICP (Iterative Closest Point) algorithm.
Grasp selection: out of the recognised parts, the system picks the one that can actually be gripped, is not covered, and where the gripper will not collide with the bin wall or a neighbouring part.
Path planning: the robot has to reach the part and pull it out without collisions. The last parts sit in the corners and along the walls of the bin. This is where it is decided whether the cell really empties the bin, or whether a few parts always remain that have to be taken out by hand.

If any one of these steps performs poorly, cycle time goes out of the window. A failed grip means a retry, an inaccurate pose estimate means a bad placement, and a collision means a stop. You rarely see this at trade show demos, because there the bin is nicely lit, half full, and the parts have a matte surface.

When is classic 3D matching enough?

Not every bin picking task needs artificial intelligence. If the part is rigid, an accurate CAD model exists, the surface is matte or at least measurable, and the bin does not hold too many different parts, classic model-based matching gives a stable and transparent solution. Typical examples are a raw casting or forging, a plastic housing, or a sheet metal part with a distinctive shape.

This approach has real advantages. It needs no training data, the parameters are understandable, and if something does not work, the engineer can see why. For a new product variant it is often enough to load the new CAD model and set the grip points.

We also recommend not overcomplicating a task that can be solved this way. AI has no value in itself. An unnecessary round of training and validation only costs time and money.

When does bin picking need AI?

Classic matching starts to struggle where the point cloud is incomplete or misleading, and where the very notion of a “part” is uncertain. In practice, these are the most common signs:

Shiny, reflective surfaces: machined steel, aluminium, chrome-plated parts. The projected pattern is reflected, holes and false points appear in the point cloud, and matching can easily jump to the wrong place.
Tangled parts: springs, hooks, bent tubes, cable harnesses. The difficulty is telling where one part ends and the next one begins.
Parts that look like one: when two or more parts touch closely, nest into each other or sit flush side by side, they can appear as a single, larger object in the point cloud. I consider this a particularly treacherous case, because classic matching does not report an error here; it confidently returns a wrong result, fitting across the boundary of the two parts, and the robot grips in the wrong place. It is common with identical flat sheet parts, stackable housings and rings.
Deformable or variable shapes: bagged goods, rubber and textile parts, food. There is no single CAD model to match against.
Many variants, frequent changeovers: if a new part number appears every week, per-model parameter tuning becomes slower than a well-trained recogniser.
Missing or inaccurate CAD: common with legacy products and supplier parts.

In these cases AI typically helps with segmentation. A trained neural network recognises from 2D and 3D data which part of the image belongs to which part, even when the point cloud has gaps. After that, classic geometry works better too, because it only has to match the points of a single part, not the entire bin.

That is why most well-functioning industrial systems are hybrid. AI tells you what the camera sees, while classic matching and robot control calculate exactly where the robot should reach. We have seen the same in visual quality inspection: the two methods are not competitors, together they deliver a stable result.

There is also a middle ground that many people overlook. In principle the task can be solved with classic methods, but only with a lot of fine-tuning, exception handling and parameters that change from bin to bin. Such a system still works at handover, but six months later nobody dares to touch it. In this situation AI is the better choice because it makes operation simpler.

Robot-Service experience: when the chaos is in the machine, not the bin

One of our projects shows well that the same technology chain that is at the heart of bin picking can be needed in a seemingly well-ordered production process too.

The task was spot welding industrial fan housings. The robot grips the housing and brings it to a fixed spot welding head. For a good weld, the part has to be held very precisely against the head. The problem was that the high currents needed for spot welding cause the welding equipment to distort continuously during operation. Over time, the fixed points taught into the robot were no longer where they needed to be in reality. The chaos was not in a bin, it was in the machine itself.

For the solution we took a 3D image and used AI to recognise the relevant segments. We fitted a reference point cloud, captured in advance from the product, onto the segment area, which gave us the actual pose of the part. Using a matrix transformation, we converted this into the coordinate system of the spot welding head, and the robot corrected its position accordingly. So the robot did not go blindly to the taught point; each time it checked where it actually had to go.

Looking at the steps, this is exactly the bin picking chain: segmentation, point cloud matching, transformation into the robot's coordinate system. The role of AI was, once again, to tell reliably which points matter in a noisy, changing environment. The matching and the transformation are classic geometry.

What we took away from this project is that 3D vision and AI are best treated as a capability rather than a single application. Wherever a robot has to reach precisely for a part in an uncertain position, whether in a bin, an assembly fixture or a distorting welding machine, the same questions come up.

Decision aid: classic 3D or AI + 3D?

The chart below is a quick first filter. It does not replace a test with sample parts, but it helps start the project in the right direction.

Decision chart: does your bin picking need AI?

Figure 1: Does your bin picking need AI? A quick decision path (own figure)

The table summarises the same by part characteristics:

Part characteristic	Recommended approach	What to watch
Rigid, matte surface, CAD available	Classic 3D matching	Bin walls, picking the last parts
Shiny, machined metal	AI segmentation + matching	Camera type, lighting, incomplete point cloud
Tangled parts	AI segmentation, special gripping	Separation strategy, retry logic
Touching parts that look like one	AI segmentation + matching	Detecting part boundaries, training on touching samples
Deformable, variable shape	AI-based recognition and grasp points	Variety of training data
Many variants, frequent changeover	AI or hybrid solution	Process for adding a new part number
Small, light, high volume	Often a vibratory feeder or 2D vision	Is 3D worth it at all

Table 1: Part characteristics and recommended recognition approach

Before you ask for a quote: questions worth clarifying

A large share of bin picking projects do not stall on technology, but on expectations that were never spelled out at the start. It is worth answering these questions before the first meeting:

What cycle time is needed, and how much margin do we allow for retries?
How many part numbers run on the cell, and how often do new ones come?
How big is the bin, and does it have to be emptied completely, or can a few parts remain?
What is the surface like: oily, shiny, painted or raw?
Can two or more parts touch in a way that makes them look like one (nested or sitting flush)?
Is there a CAD model, and how closely does it match the manufactured part?
What should happen if the robot cannot pick a part: retry, shake the bin, or alert the operator?
Where and with what accuracy does the part have to be placed, and is regripping needed?

A practical tip: send sample parts in the very first round, ideally in the original bin, the way they arrive in production. A half-day test with real parts tells you more than any datasheet. This is how our feasibility studies start as well.

Key takeaways

Bin picking is not a camera question. The 3D image is only the first step; recognition, gripping and path planning decide together.
Stick with classic matching where you can. For rigid, matte parts with a CAD model it is often the faster and cheaper route.
For difficult parts, AI is what makes the system stable. With shiny, tangled, look-alike, deformable or many-variant parts, AI-based segmentation is the key, typically combined with classic matching.
The same chain works outside the bin. Position correction, compensation of distorting fixtures, handling parts in uncertain positions.
The decision should rest on a test with sample parts, not on a demo video.