Dex-Net 4.0 allows ambidexter robots to choose the best capture

The ability to choose a grip helps two-handed robots lift objects faster than ever

For several years we have been following the progress of the Dex-Net project, which is trying to develop a universal capture for robots, and in the middle of January a new work appeared in the journal Science Robotics, in which scientists from the University of California at Berkeley present Dex-Net 4.0. The most important and interesting news related to this work is that the latest version of Dex-Net has successfully coped with the capture of 95% of previously unknown objects at a speed of 300 pieces per hour, thanks to the added ambidexter robot, which allows it to choose one of the following on the fly two kinds of grips.


So that you can compare, let us say that a person is able to lift such objects twice as fast, from 400 to 600 pieces per hour. And I would say that in the case of a person, you can expect 100% success in capturing - or at least a good approximation to this figure, if you allow the subject to try several times with each of the objects. So we set a very high bar for cars. Part of our success in capturing objects (and the ability to grasp in general) is in the great life experience of working with objects of many shapes, sizes, weights, with friction of various materials and with the possible deformation of objects during capture. Even without realizing this, we are able to build detailed models of objects in our heads, and they help us to easily grab and lift objects that have never been seen before.



But robots do not have access to this experimental model of the world. They rely on learning based on a specific task - and here comes into play Dex-Net. She learns to capture things, training in simulations, using millions of models of three-dimensional objects and a little random physics, in order to better transfer success in the simulation to the real world. Artificial ambiguity allows the system to work with things like sensor noise and small gradual calibration shifts — of course, more realistic results could be obtained by training real robots, but then such restrictions as the need to have many real robots come into effect time to work - and who wants to wait for them?

The uniqueness of Dex-Net 4.0 lies in the fact that the rules for capturing “ambidext” objects produced by it, that is, the robot has two captures, and decides which of them should be used at the moment. However, unlike ambidextral people, this robot has different grips on its hands: a two-finger clamp and a vacuum sucker. On the basis of a preliminary assessment of the quality of capture, Dex-Net chooses which of the captures can more reliably grab an object. This technology makes it possible to snap objects quickly and reliably: ABB YuMi in the above video can capture about 300 objects that it has never seen before, per hour, with an efficiency of 95%. And Dex-Net allows you to connect other types of captures. After additional training (and by adding grips to a robot, you can teach him to work with electrostatic grippers, five-finger fingers, grips like “paws of a gecko” or anything else.

Of course, it is always interesting to study those 5% of cases when the robot failed to take the object, and here are some examples:



The first photo shows “problematic” objects that are especially difficult to lift due to “problematic geometry, transparency, mirror surface and deformability”. Dex-Net copes with such objects only in 63% of cases, although if you allow the system to memorize previous failures and move the object a little, if it is not clear how to grab it, then reliability increases to 80%.



The second photo shows objects with which Dex-Net 4.0 cannot cope at all, “due to reflective properties, such as transparency, which affects the perception of depth, and material properties, such as porosity and pliability (for example, loose packaging), which affects the ability to stick to the surface with a vacuum sucker. " It is worth noting that the two-finger grip does not have any force sensors and tactile sensors, so there is something else to improve in the system.

You can also consider cases in which efficiency that does not reach 100% will be acceptable. There are many realistic ways of handling failures with grips: you can instruct the robot to collect all things from the basket, and send the remainder to a person who will cope with complex objects. Or, perhaps, at some point it will make sense to change the packaging of items so that it becomes easier to take the items that are especially difficult for a robotic capture. In any case, this is more a question of “when”, rather than “if”, and due to how attractive it is to increase supply chain automation, this is the “when” probably will come very, very soon.