Embedding high-resolution touch across robotic hands enables adaptive human-like grasping

F-TAC Hand hardware

F-TAC Hand advances the state of dextrous robotic hands through its comprehensive tactile sensing capabilities while maintaining a full range of motion. The hand achieves human-like tactile coverage, with sensing elements extending across 70% of the palmar surface at a density of 10,000 taxels (the pixels in camera CMOS) cm⁻² (Fig. 1), notably surpassing current commercial solutions such as Shadow Hand, which provides only five-point feedback over less than 20% of its surface¹¹ (comparison with other tactile arrays is available in Supplementary Section 1). This extensive coverage is achieved through an array of vision-based tactile sensors in multiple configurations (see exploded view in Extended Data Fig. 1a and physical dimensions in Supplementary Section 1), featuring specially designed covers that align with the hand’s phalanges and palm to minimize mechanical redundancy while replicating the natural kinematic structure of the human hand (Fig. 2a). Each of the five fingers incorporates three DoFs, contributing to the hand’s total 15-DoF configuration that enables human-like dexterity. A specialized electronic module enables large-scale sensor reading acquisition while minimizing space, weight and cabling requirements (Extended Data Fig. 1b). The hand’s dimensions mirror those of an adult human hand, measuring 194 mm from wrist to middle fingertip (Fig. 2a), and its modular design architecture allows for easy adaptation to different physical dimensions while maintaining functionality.

a, The seamless integration of 17 vision-based sensors in six configurations, maintaining 15 DoFs—three per finger—and adult hand dimensions. Each sensor includes a streamlined camera module for efficient tactile data acquisition in confined space. PCB, printed circuit board. b, F-TAC Hand demonstrates its strength by holding a 2.5-kg dumbbell; each phalanx contributes to a total grasping force of 10.3 N. c, Schematic representation of a finger, with K_n, θ_n and F denoting joint stiffness, rotation angle and cable force, respectively. Offsets in rotation due to cable and joint alignment are also shown. d, Top-down-view comparison of F-TAC Hand and human finger flexion. e, Despite the numerous sensors, F-TAC Hand retains its mobility, as evidenced by a successful Kapandji test²³, where the thumb fingertip sequentially touches specific points on the hand as numbered in the figure. PP, proximal phalanx; MP, middle phalanx; DIP, distal interphalangeal; PIP, proximal interphalangeal; MCP, metacarpophalangeal.

Building upon its extensive tactile sensing coverage, F-TAC Hand also achieves comprehensive motion capabilities that match state-of-the-art dextrous hands^9,10,11. The hand implements full mobility using just five slim cables (Extended Data Fig. 1c) with substantial payload capacity (Fig. 2b). Each cable controls the flexion and extension of a finger (Extended Data Fig. 1d,e), working in concert with stiffness-tuned springs at each joint (Fig. 2c) to replicate the coordinated yet semi-independent movements characteristic of human hands³¹ (Fig. 2d). An additional degree of actuation enables thumb opposition, expanding the hand’s motion versatility (Extended Data Fig. 1e). Detailed fabrication procedures are provided in ‘Tactile sensor fabrication’ and ‘F-TAC Hand fabrication’ in Methods. The hand’s dexterity is demonstrated through two evaluations: the Kapandji test²³, completing all ten designated thumb-to-hand contact points shown in Fig. 2e, and the successful execution of all 33 human grasp types (Fig. 3).

Empowered by its smart design, the workspace of F-TAC Hand enables it to perform all 33 human grasping types, as documented in ref. ²⁴.

The hand’s tactile sensing system utilizes the photometric stereo principle^32,33, converting light intensity variations into surface gradient information (Fig. 4a). Contact surface geometry is reconstructed through a two-stage process. First, an array of encoder–decoder neural networks (Fig. 4b) maps physics-based relationships between surface gradients and intensity variations for each sensor. Next, a Poisson solver generates high-fidelity surface geometries, visualized as normal maps (Fig. 4c). The detailed sensor characteristics are available in Supplementary Section 1.

a–d, Raw tactile sensor readings from the configuration shown in Fig. 1a (a) are processed by neural networks (b) to reconstruct contact site geometries (c), visualized as normal maps. The neural networks, trained on simulated data (d) generated by a physics-based image formation model, enable efficient and precise mapping of extensive raw data to geometric information at the contact interface. e,f, When grasping an object (e), F-TAC Hand captures detailed contact information through its advanced tactileer sensing capabilities (f). g, This rich tactile feedback enables F-TAC Hand to accurately perceive and interpret object characteristics, as demonstrated by its precise estimation of in-hand object poses.

The unprecedented scale of F-TAC Hand’s tactile sensing system required development of efficient calibration solutions. We addressed this through a physics-based image formation model (detailed in Supplementary Sections 2 and 3) that generates synthetic readings of elastomer deformations during contacts (Fig. 4d). This approach enables efficient neural network training (Fig. 4b) and accurate sensor calibration.

The integration of fine-grained tactile sensing with robust motion capabilities enables F-TAC Hand to effectively grasp diverse objects, including challenging cases such as crystal balls (Fig. 4e), while simultaneously capturing detailed contact information (Fig. 4f). These sensory data enable accurate object pose estimation during manipulation (Fig. 4g). Additional demonstrations are provided in Supplementary Video.

Through this combination of dense tactile arrays and advanced motor capabilities, F-TAC Hand achieves unprecedented biomimetic fidelity, advancing both robotic manipulation capabilities and our understanding of human manual dexterity.

Powering F-TAC Hand with human-like diverse grasping

While F-TAC Hand’s high articulation enables sophisticated manipulation, it presents unique challenges in grasp planning. The increased number of DoFs makes traditional mechanical equation-based methods computationally intractable³⁴. Learning-based alternatives³⁵, though avoiding complex analytical solutions, require extensive training data that are both costly to collect and potentially biased by human demonstration preferences—a particular challenge for highly articulated dextrous hands.

We model the robotic grasp generation of rigid objects as sampling hand poses from a Gibbs distribution conditioned on object geometry. Each grasp is associated with an energy value derived from force closure criteria, which evaluates how well the grasp can resist external forces. Lower energy values indicate better grasping capability; see ‘Probabilistic formulation for grasp generation’ in Methods and Supplementary Section 4 for details. Due to the hand’s high number of DoFs and the non-convex nature of the problem³⁶, we sample grasps from random initializations and apply a modified Metropolis-adjusted Langevin algorithm to reduce energy and escape local minima, converging to low-energy, high-quality grasps; see ‘Exploration algorithm for complex energy landscape’ in Methods for details. We validated the approach using a diverse test set of 23 objects, including spheres, cylinders, cuboids and irregular shapes (Fig. 5a). By executing the algorithm from various initializations, F-TAC Hand’s biomimetic kinematics (Extended Data Fig. 2) and varied object geometries result in diverse grasping poses (Supplementary Video).

a, A total of 23 objects, varying in dimensions and geometrical complexity, are chosen to assess the efficacy of the grasp generation method. b–d, The landscape of possible grasps is depicted as a disconnectivity graph, generated using the ADELM algorithm. e, For the pliers, the diversity in grasp strategies mirrors its common usage in daily human activities. f, This diversity in grasping strategies is maintained even when the object possesses complex geometric features. g, Comprehensive categorization of the generated grasps for all 23 objects, on the basis of human grasp types adopted from ref. ²⁴. The results collectively cover all 19 common grasp types, implying the human-like diversity of the generated grasping strategies.

The resulting grasping poses are analysed through the Attraction–Diffusion Energy Landscape Mapping (ADELM) algorithm³⁷ to visualize the complex energy landscape defined in equation (2), as shown in Fig. 5b–d. In this visualization, circles represent local minima (areas of low energy), where each local minimum contains at least one feasible grasp. The circle’s size indicates how many similar grasps exist within that local minimum. The circles are colour-coded according to the majority grasp type on the basis of ref. ²⁴: power, precision and intermediate. The vertical connecting bars between circles represent energy barriers between different local minima, where shorter bars indicate easier transitions between grasping poses, while longer bars signify transitions that are more difficult to achieve. Direct comparisons between generated grasps and human demonstrations (Fig. 5b–d) in the boxes below validate the human-like nature of our solutions. This approach maintains its effectiveness even for challenging cases such as pliers and adversarial objects³⁸ (Fig. 5e,f), with comprehensive energy landscapes presented in Extended Data Fig. 3.

To quantitatively assess the human-like diversity of our approach, we analysed 1,800 generated grasps according to the taxonomy of ref. ²⁴, categorizing them into 19 common grasp types (Supplementary Section 5). The resulting distribution (Fig. 5g) demonstrates comprehensive coverage across the human grasp repertoire, from frequent strategies such as power sphere and precision sphere to specialized configurations such as distal type and palmar grasps.

Further analysis using contact maps³⁹ reveals natural clustering patterns that align with human grasp classifications. By applying dimensionality reduction through principal component analysis and visualization via t-distributed stochastic neighbour embedding (Extended Data Fig. 4), we observe distinct groupings of power and precision grasps, with intermediate grasps appropriately positioned near the boundary defined by a radial-basis-function-kernel support vector classifier. This distribution mirrors human grasp categorization patterns, where intermediate grasps share characteristics of both primary types (computational details in Supplementary Section 6).

The demonstrated ability to generate diverse, human-like hand configurations provides F-TAC Hand with both optimal and near-optimal control strategies. This algorithmic foundation, working in concert with the low-level controller, enables enhanced dexterity and adaptability in real-world manipulation scenarios.

Adaptive behaviours of F-TAC Hand

The integration of advanced tactile sensing with diverse grasping strategies enables F-TAC Hand to implement a closed-loop sensory–motor feedback mechanism, allowing real-time adaptation to environmental changes. The implementation details are illustrated in Extended Data Fig. 7 and described in ‘Context-sensitive motor controls’ in Methods.

We demonstrate F-TAC Hand’s capabilities through multi-object grasping⁵, a critical benchmark for hand dexterity that surpasses the limitations of 1-DoF parallel grippers. This challenging task demands precise contact detection and strategic adjustments to avoid collisions—capabilities that remain elusive for current AI systems⁵. While recent advances^13,25 show promise, managing the stochastic nature of real-world objects, especially those with complex geometries, remains challenging. F-TAC Hand overcomes these limitations through precise contact-point identification (Fig. 6a).

a, F-TAC Hand’s ability to grasp multiple objects simultaneously in a human-like manner. b, Despite execution noise, F-TAC Hand can optimize object transport through multi-object grasping. c, Real-world disturbances significantly increase second-grasp collision rates (inset: collision detection point in red), highlighting the necessity of tactile monitoring. d, Tactile-informed adaptation significantly improves second-grasp success rates. Plots c and d show 60 independent technical replicates (blue dots), each representing a unique object combination tested across 10 trials. Box plots show 25th–75th percentiles (box), median (centre line) and minimum/maximum values (whiskers). Shaded areas (orange in c, green in d) show kernel-density estimations of data distribution. The identical P values (2.1 × 10⁻¹⁷) displayed in the two panels derive from one-sided paired t-tests (t(59) = 11.8) comparing perfect versus real-world execution (c) and with versus without tactile-informed adaptation (d). No adjustments were made for multiple comparisons.

To evaluate real-world performance, we mounted F-TAC Hand on a Kinova Gen3 robotic arm for multi-object transport tasks (Fig. 6b). The goal is to grasp as many objects as possible in one go to maximize transportation efficiency. While optimal strategies exist under ideal conditions (red route in Fig. 6b), real-world variables—such as imperfect robot positioning and object perception—require adaptive motor control. The other coloured routes in Fig. 6b illustrate actual scenarios where F-TAC Hand encountered wrong object positions but leveraged its comprehensive tactile sensing capabilities (Fig. 4g) to assess situations and dynamically switch to alternative strategies that accommodate available space, even if theoretically suboptimal. Additional demonstrations of adaptive behaviours, including responses to finger impairments and ball size adaptation, are shown in Extended Data Fig. 5 and Supplementary Video.

We quantified the impact of tactile sensing through extensive experiments involving 60 object combinations across 600 real-world trials. Initial grasps were programmed from a disembodied AI perspective, using theoretically optimal strategies without considering environmental dynamics. Each combination underwent ten real-world trials, with tactile feedback assessing in-hand object positions and potential collision risks. Collision detection involves two steps: using tactile information to estimate the grasped object’s pose as shown in Fig. 4g, and then checking whether the union of the grasped object geometry and the next target object geometry is null. The observed collision rate in real-world execution (M = 0.465, s.d. = 0.306) differed significantly from theoretical predictions (M = 0.000, s.d. = 0.000), highlighting the substantial gap between simulation and reality, t(59) = 11.8, P = 2.1 × 10⁻¹⁷ (Fig. 6c).

F-TAC Hand’s adaptive capabilities become particularly evident when comparing tactile-informed versus non-tactile-informed control. Upon detecting collision risks, the system rapidly (~100 ms) switches to alternative strategies that might be suboptimal in theory but practical in reality. In scenarios with potential collisions, the non-tactile-informed approach inevitably fails, while the tactile-informed approach maintains productivity through adaptive replanning. The tactile-informed approach achieved perfect adaptation (M = 1.000, s.d. = 0.000) compared with significantly lower success rates without tactile feedback (M = 0.535, s.d. = 0.306), t(59) = 11.8, P = 2.1 × 10⁻¹⁷ (Fig. 6d). Notably, in collision-free scenarios, the two approaches demonstrate comparable execution times, with tactile-informed collision checking adding only ~1 s of processing time, indicating that tactile sensing provides critical robustness with minimal computational overhead during normal operations. The detailed method and the logic chain are available in ‘Context-sensitive motor controls’ in Methods.