CoordRefer

CoordRefer: Coordinate-Aware 3D Visual Grounding from Multiview Images

Haijie Li1, Jiaxin Zhang2, Dave Zhenyu Chen3,*, Youyu Chen2, Yanmin Wu1, Jian Zhang1,4,†
1School of Electronic and Computer Engineering, Peking University   2Harbin Institute of Technology   3Huawei
4Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology,
Shenzhen Graduate School, Peking University

* Project Leader    † Corresponding Author

Paper

Video

A short introduction to CoordRefer.

The video includes an AAC stereo audio track. Click play and check the speaker icon if your browser starts with volume disabled.

Synchronized 3D Comparison

Drag, zoom, or pan any view—the other two cameras stay synchronized.

scene0084_00

“A round trash can is located under the counter top.”

Baseline IoU0.000CoordRefer IoU0.719

Baseline IoU 0.000

Loading…

CoordRefer IoU 0.719

Loading…

Ground Truth Reference

Loading…

Drag to rotate · Scroll to zoom · Right-drag to pan

Overview

CoordRefer selects a coordinate frame before predicting its coordinate-conditioned 3D box.

CoordRefer overview
Figure 1: Left: Resolving box ambiguity for 3D visual grounding. Existing coordinate-agnostic frameworks jointly infer coordinate frames and 3D boxes, causing ambiguous supervision and producing invalid coordinate–box predictions. Our coordinate-aware framework decouples these two objectives by first selecting a coordinate frame and then grounding objects in the selected coordinate system, yielding a unique box representation and improved performance. Right: Performance gains on 3D visual grounding benchmarks. Our framework achieves improvements of 11–14% across multiple benchmarks over baseline.

Coordinate-Relative Box Ambiguity

Joint frame-and-box prediction creates multiple equivalent targets across coordinate systems.

Coordinate-relative box ambiguity
Figure 2: Coordinate-relative box ambiguity and its resolution through explicit coordinate conditioning. (a) The same physical 3D box has different representations under different coordinate frames. (b) Without coordinate conditioning, multiple equivalent targets may lead to a compromise but invalid solution. (c) Conditioning box regression on the selected coordinate frame provides a unique optimization target.

Abstract

Multiview image-based 3D visual grounding predicts a coordinate frame to define a coordinate system and then regresses a 3D bounding box for localization. However, existing methods jointly optimize coordinate frame selection and box regression, leading to coordinate-relative box ambiguity and degraded grounding performance. This ambiguity arises because the same box admits different numerical representations across coordinate frames, creating multiple optimization targets and yielding invalid compromise predictions. To tackle this challenge, we propose CoordRefer, a coordinate-aware framework that decouples coordinate frame selection from coordinate-conditioned grounding. CoordRefer first selects a reference frame to define the coordinate system and then conditions 3D box prediction on the coordinate system. We perform coordinate-aware supervised fine-tuning to establish coordinate frame selection and coordinate-conditioned box regression, followed by Group Relative Policy Optimization with 3D IoU-based rewards to align both stages with downstream grounding quality. On ScanRefer with Qwen3-VL-2B, CoordRefer achieves gains of +11% Acc.25 and +7% Acc.5 over the coordinate-agnostic baseline, while its geometrically refined variant surpasses methods using explicit 3D inputs.

Method

Coordinate-aware SFT establishes correspondence; IoU-driven GRPO optimizes both stages.

CoordRefer method
Figure 3: Coordinate-aware training strategy. In the SFT stage, the pretrained VLM is jointly trained on decoupled coordinate-frame selection and coordinate-conditioned grounding using pseudo frame labels and ground-truth boxes. In the RL stage, GRPO optimizes box prediction with a direct IoU reward and frame selection with an indirect IoU reward computed by a frozen grounding model. The grounding model is periodically updated with the latest policy weights.

Qualitative Results

Coordinate conditioning produces more accurate and spatially consistent 3D boxes.

CoordRefer qualitative results

Single- and Multi-Object Grounding

The same formulation generalizes across single- and multi-object queries.

Single- and multi-object grounding results

Performance

Strong RGB-only grounding, competitive 3D-refined results, and zero-shot generalization.

RGB-Only 3D Visual Grounding

In-domain comparison across four benchmarks.

MethodScanReferNR3DSR3DMulti3DRefer
Acc@.25Acc@.5Acc@.25Acc@.5Acc@.25Acc@.5F1@.25F1@.5
SPAR31.912.4——————
VG-LLM-4B36.411.825.37.528.79.831.811.2
VG-LLM-8B41.614.931.511.337.214.138.614.8
3D-RFT-4B42.915.9——————
Qwen3-VL-2B†39.716.427.410.631.512.834.414.8
CoordRefer-2B51.123.642.117.242.617.845.420.8
Improvement+11.4+7.2+14.7+6.6+11.1+5.0+11.0+6.0

Explicit 3D Input

In-domain comparison on ScanRefer.

MethodAcc@.25Acc@.5
ScanRefer37.324.3
3D-LLM30.3—
Grounded 3D-LLM47.944.1
LLaVA-3D54.142.4
Video-3D LLM58.151.7
CoordRefer-2B*60.053.4

Zero-Shot Generalization

Train on ScanRefer, evaluate on ARKitSceneRefer.

MethodAcc@.25Acc@.5
VG-LLM-4B3.40.2
VG-LLM-8B3.70.1
Qwen3-VL-2B†6.20.6
CoordRefer-2B7.90.7
Video-3D LLM0.20.0

* 3D refinement   † Coordinate-agnostic Qwen3-VL-2B baseline.

BibTeX

@misc{li2026coordrefercoordinateaware3dvisual,
  title={CoordRefer: Coordinate-Aware 3D Visual Grounding from Multiview Images},
  author={Haijie Li and Jiaxin Zhang and Dave Zhenyu Chen and Youyu Chen and Yanmin Wu and Jian Zhang},
  year={2026},
  eprint={2608.05569},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2608.05569}
}