The first author of the DexNDM achievement is Liu Xueyi, a Ph.D. student at Tsinghua University's Institute for Interdisciplinary Information Sciences. The corresponding author is Li Yi, an assistant professor at Tsinghua University's Institute for I

When can the day come when robots use dexterous hands to help humans twist screws in factories and cut vegetables at home? To realize this vision, DexNDM, which aims to solve the sim-to-real problem of dexterous manipulation skills, has emerged.

Paper Title: DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics ModelnPaper Link: https://arxiv.org/abs/2510.08556nProject Website: https://meowuu7.github.io/DexNDM/nYouTube Video: https://www.youtube.com/wa

Background - High-Agility Complex Tool Teleoperation

Achieving complex tool teleoperation with high dexterity, such as controlling a robot arm to use a screwdriver or hammer, has been a long-standing core challenge in robotics. Traditional direct mapping teleoperation approaches, which involve direct c

To break through this bottleneck, we propose a semi-autonomous teleoperation paradigm. The core idea is to decompose complex teleoperation tasks into a series of autonomous and stable atomic skills that robots can execute. Operators only need to give

Among many atomic skills, in-hand object rotation is a crucial and challenging basic ability. It is not only a concentrated expression of the dexterity of the dexterous hand, but also a prerequisite for using most tools. However, in-hand rotation inv

To this end, we propose DexNDM, a new method aimed at learning general and stable low-level atomic skills. DexNDM is designed to overcome the limitations of existing work, enabling the dexterous hand to master the skill of stably rotating various obj

Based on the powerful and stable rotating atomic skills provided by DexNDM, we have finally built a highly flexible and robust semi-autonomous teleoperation system. In this system, operators can easily guide the dexterous hand to complete previously

Unparalleled Flexibility

Highlight 1: Full-scene object coverage: from tiny to ultra-long, from simple to complex, all can be precisely controlled

First achieved continuous rotation under extreme challenge

DexNDM breaks through the ceiling of existing intra-hand rotation technology and for the first time realizes continuous and stable rotation of long objects (such as sticks and pens) along the long axis under extremely challenging wrist postures such

Fig. 1: Rotating small objects and long objects

Wide object coverage, perfect for complex geometry

Extensive Object Coverage

DexNDM has made significant breakthroughs in the diversity of operable objects, achieving full coverage from tiny to slender and from simple to complex geometries, which is far broader than any previous work (as shown in Figure 2).

To quantify this advantage, we directly compared it with Visual Dexterity, which previously performed the best in rotating complex geometric objects. The results showed that even with a smaller and more general Leap Hand than the customized D’Claw ma

Furthermore, DexNDM has introduced an unprecedented capability. For the first time, we have demonstrated the ability to manipulate complex geometries with various surface irregularities using generic robot hands such as Allegro and Leap Hand, in chal

Rotating Complex Geometry

Highlight 2: Full-attitude precise control: Any wrist posture and various rotational axes can be moved freely

Besides its excellent versatility in object types, DexNDM is also characterized by its strong adaptability to wrist posture and rotational axis. Regardless of the orientation of the manipulator or the axis along which the task requires the object to

Diverse wrist orientations

【Highlight 3】The highly 'flexible' and robust dexterous hand remote operation system can handle various tools and is competent for long-range assembly tasks

We have taken DexNDM's powerful in-hand rotational capability as an 'atomic skill' to build a teleoperation system with far greater flexibility than traditional solutions. In this system, operators simply control the pose of the robotic arm through t

Furthermore, the super robustness of DexNDM strategy enables teleoperation systems to be competent for long-horizon assembly tasks that require extremely high stability. For example, we successfully completed the complete assembly process of installi

Fig. 5: High dexterity and high robustness dexterous hand teleoperation

Technological Breakthrough

DexNDM's powerful object rotation capability in the real world is achieved through its innovation in the sim-to-real approach (Figure 6).

Specifically, the core of this approach is a joint-level dynamic model, which effectively fits a small amount of real-world collected data, and adjusts the actions of the simulation strategy accordingly, thereby bridging the dynamic deviation between

The author adopted a set of fully automatic data collection strategies to collect diverse real-world interactive data with minimal human intervention. The joint-by-joint dynamic modeling and fully automatic data collection strategy proposed by DexNDM

Based on the joint dynamics model of a dexterous hand in the real world obtained through training, the author trains a residual policy network on the original policy to output a correction term according to the command output of the original policy n

Fig. 6: Overview of Methods

Joint-level Neural Dynamic Model

Unlike the more direct modeling of whole hand and object interaction dynamics, the joint-level neural dynamic model decomposes the complex interaction dynamics at each joint, and predicts the state of each joint independently at the next moment from

The author has verified the three key properties of the joint-level neurodynamic model through theoretical analysis and experiments, namely high expressiveness, high data efficiency, and strong generalization ability. This kind of generalization abil

Automated Data Collection System

Based on four principles: (i) the collected data is related to the transfer distribution of the strategy network, (ii) there is an object load, (iii) the distribution covers a wide range, and (iv) it is easy to expand, the author has built an automat

The implementation method is simple: place the robot hand in a container filled with soft balls. Then the author plays back the actions from the simulation-based strategy in an open-loop manner, which provides a coarse-grained distribution prior (i).

The advantage of joint-level dynamic modeling under biased data distribution

Training Based on Residual Strategy

Conclusion

DexNDM has taken a solid and significant step forward in solving the core problem of robotics, Sim-to-Real. It has solved the problem of learning reliable real-world dynamic models for dexterous manipulation, and achieved unprecedented dexterous mani

It is true that DexNDM has its limitations. However, the author believes that this is just the beginning, and the dexterous hand, as the crown of humanoid robots, is destined for a bright future and will surely shine brilliantly.