RGB-D OBJECT PERCEPTION

F2G-Pose:Geometry-Aware Foundation Feature Lifting for Direct RGB-D Category-Level Object Pose Estimation

From one segmented RGB-D observation, directly predict 6D pose, metric size, and camera-frame completed shape.

Jinyu Zhang^1,2,* · Haitao Lin^1,* · Jiashu Hou^1,*
Xiangyang Xue¹ · Yanwei Fu^1,2,†

¹ Fudan University ² Shanghai Innovation Institute
* Equal contribution † Corresponding author

Paper PDF Code Weights Watch video

ROPE scene 000152 with F2G-Pose predictions on a bottle, box, fire extinguisher, and picnic basket

01 / 6D POSE02 / METRIC SIZE03 / COMPLETED SHAPEONE FORWARD PASS

01 — METHOD

From pixels to metric geometry.

F2G-Pose uses the observed RGB-D geometry and dense visual features together. Its two output branches share one learned representation.

Lift visual features to 3D

A frozen RADIO backbone extracts dense image features. Aligned depth and the object mask associate those features with observed 3D points.

Fuse local and global structure

DGCNN builds local geometric tokens; a Mixture-of-Experts Transformer brings in object-level context.

Predict together

Parallel heads produce the 6D pose and metric size, plus a completed point cloud in the camera frame.

Figure 1: RGB-D input is lifted into visual and geometric point features, fused by a MoE Transformer, and decoded into pose, size, and completed shape — Paper Figure 1 · Tap or click to view at full size.

02 — FILM

See the full story in 90 seconds.

Real and synthetic scenes, pose comparisons, three selected shape predictions, and recorded robot cases.

FULL FILM01:30 · Selected qualitative examples

03 — SELECTED EXAMPLES

Look closer.

ROPE / COMPARISON

Ground truth and prediction

Two pose overlays on the same frames from ROPE scene 000300.

ROPE / SCENE 000354

Changing viewpoints

Frame-wise predictions for a donut, helmet, pineapple, rice cooker, and teapot.

ROPE / SCENE 000064

Shapes and scales

Six everyday categories in another continuous ROPE interval.

ROPE / MATERIALS

Glass and a specular dish

Selected transparent red wine glass and specular dish cases.

Direct shape predictions

SOPE / ERASER

Eraser

SOPE / CHINESE CHESS

Chinese chess piece

SOPE / FLUTE

Flute

RECORDED ROBOT SCENES

Pose estimates in the workspace.

Mug and cup, hammer, and transparent bottle cases.

ARX ROBOT / THREE RECORDED CASES

Paper results at a glance.

ROPE · IoU50 AUC27.8F2G-Pose vs. 19.1 GenPose++

SOPE · IoU50 AUC43.7F2G-Pose vs. 31.9 GenPose++

MODEL THROUGHPUT28 FPSPose-only · 23.7 FPS full model

SOPE · SHAPE CD-L15.42Paper metric, ×1000

ROBOT · FULL MODEL68/80Successful placements in the paper trials

AUC values are percentage points. Throughput is model inference on the paper’s hardware and protocol; the film’s 30 FPS playback is separate.

05 — READ & BUILD

Explore the paper and code.

Find the complete method, evaluation details, installation instructions, and pretrained checkpoint.

Read camera-ready PDF Browse source code Download weights