Dexterous manipulation · Robot learning · Sim-to-real

DexEyeGraspLearning Dexterous Grasping in Clutter
with FingerEye Observations

Anonymous Authors

Paper coming soon Code coming soon Video BibTeX
Binocular FingerEye cameras on four fingers observe a highlighted target as the robot initializes, searches, and grasps it in a cluttered box.
Seeing from the fingertips. Learning to grasp in clutter.
DexEyeGrasp uses articulated FingerEye viewpoints to guide both finger and wrist motion.

Project video

DexEyeGrasp in three minutes

Our method and real-world experiments, with narration.

3:00 1080p English captionsDownload video

The idea

Abstract

Dexterous grasping in dense clutter requires precise finger and wrist control under severe occlusion, which can hide the target and spatial cues needed for grasping from third-person cameras. We present DexEyeGrasp, a visuomotor learning framework that leverages FingerEye observations to guide both finger and wrist motion. Our framework distills a state-based teacher trained in simulation into a visuomotor student policy that integrates multiple dexterous viewpoints.

To help the policy extract task-relevant spatial information, we introduce self-supervised learning objectives that predict current and action-conditioned future camera–target geometry. These objectives encourage geometric awareness for approaching the target and refining contact-rich interactions despite changing visibility. To further address the visual sim-to-real gap, we learn a residual image encoder from real FingerEye trajectory sequences while keeping the simulation-trained network frozen with the aforementioned objectives. Together, these designs support learning and transferring dexterous grasping in densely cluttered environments.

From simulation to the real world

Learning through dexterous viewpoints

A privileged teacher, a visual student, and a shared geometric learning objective.

Method overview: a PPO state-based teacher supervises a FingerEye visual student through DAgger; current and next-step geometry prediction train visual features and a residual encoder for real-world adaptation.
Overview of DexEyeGrasp. Click any figure to explore it at a larger size.
01

Learn in simulation

A recurrent PPO teacher uses privileged state information to learn coordinated arm–hand control in clutter.

02

See from the hand

Online DAgger distills the teacher into a student that combines FingerEye images, attention across views, and proprioception.

03

Adapt with geometry

FE-Targ objectives predict current and future target geometry, then train a residual visual encoder on real trajectories.

Visual randomization during training

Scene appearance and image augmentations vary during simulation training. Magenta overlays identify the target while preserving the surrounding visual context.

Randomized simulation environments with scene views, raw FingerEye RGB, and augmented target-mask policy inputs.
Randomized environment 1
Randomized environment 2
Randomized environment 3
Randomized environment 4

Synchronized scene and raw FingerEye observations. Each clip plays once at its recorded speed.

Experiments

Experimental platform

The robot, fingertip cameras, and objects used in our real-world experiments.

A closer view of contact

An xArm7 arm and XHand use binocular FingerEye cameras on four fingertips. The visual policy combines these moving viewpoints with proprioception to control wrist and finger motion.

A side-view RGB-D camera supports target pose estimation for adaptation data. A residual image encoder learns from approximately 50 real teleoperated trajectories using geometric supervision.

Real-world control runs at approximately 10 Hz with multi-view target tracking.

The real xArm7 and XHand setup with FingerEye cameras, a side-view camera, and cluttered workspace.
Robot platform
Collection of real-world target objects with diverse shapes, colors, and materials.
Real-world target objects

Real-world experiments

Real-world videos

Targeted grasping in clutter, with observations from the fingertips.

FingerEye observations during grasping

The same grasp, shown alongside six synchronized fingertip views.

Target: orange ballTarget object
Thumb · top rowMiddle · middle rowRing · bottom row

Different target objects

Six grasping trials. Each target image matches the adjacent video.

Normal-speed clips. Play individually or start all six together. No automatic replay.

Increasing clutter

Grasping examples with 15, 30, 40, and 50 objects in the workspace.

Target object for the 15-object scene
15 objects

Clutter 15

Target object for the 30-object scene
30 objects

Clutter 30

Target object for the 40-object scene
40 objects

Clutter 40

Target object for the 50-object scene
50 objects

Clutter 50

BibTeX

Provisional entry for the current anonymous manuscript.

@misc{dexeyegrasp,
  title  = {DexEyeGrasp: Learning Dexterous Grasping in Clutter
            with FingerEye Observations},
  author = {Anonymous Authors},
  note   = {Manuscript},
  url    = {https://dexeyegrasp.github.io/}
}