CoRL 2026 Workshop on Agentic Robotics · Poster
1Independent Researcher 2Georgia Institute of Technology 3The Hong Kong University of Science and Technology
Online ring-on-peg, Geometric Abstraction vs. Pixel Reasoning
Recent multimodal language models can reason directly about robot scenes, but it is unclear whether they need to repeatedly reinterpret raw visual observations at every step of an interaction. We propose shifting routine spatial reasoning from images to text and symbols through a 3D geometric abstraction that is refreshed locally from observations and read directly by the language models. Specifically, our interface represents scene geometry via typed, evidence-aware symbolic references to object parts, grounds them from calibrated RGB-D, and returns compact textual checkpoints with metric residuals after each action. Symbolic task structure can thus be reused across decisions while metric values are rebound to the current scene, with direct visual reasoning invoked only on demand. Across offline policy synthesis and continuous online execution, reasoning over grounded text and symbols improves multi-step manipulation completion while substantially reducing repeated multimodal queries, positioning explicit geometric state as an effective substrate for agentic robot policies.
Calibrated side and wrist RGB-D are back-projected into the world frame to estimate each object's center, orientation, dimensions, vertices, edges and faces, each with a timestamp, using lightweight CPU estimators. Shape may persist, but pose must be refreshed: after an object moves, its pose is re-estimated from a current observation, never copied from a commanded motion.
Policies refer to typed parts like red.bottom_face and blue.top_face and compose them with operators such as surface alignment and support checks. Every value carries its evidence status: observed inferred unavailable. The same program rebinds to new scenes.
After each action the agent receives a compact checkpoint with the updated 3D coordinates, evidence status and metric residual to the goal. If an object is occluded, the residual is marked unavailable rather than computed from a stale estimate. Images stay available on demand.
| Operation | Inputs | Output |
|---|---|---|
reference | object and role | geometric entity with evidence |
relation | predicate and objects | residuals and true/false/unknown |
align | source and target geometry | desired rigid transform |
checkpoint | active object and goal | compact textual state |
A simulated SO-100 arm builds and inserts with simple rigid objects. Each card shows a start and goal configuration. These are constructed renderings, not recorded policy trajectories.
Single-policy acquisition
Continuous online execution
A simulated SO-100 arm with calibrated side and wrist RGB-D cameras. We compare a Pixel Reasoning agent that reasons directly from visual observations with a Geometric Abstraction agent that additionally receives our grounded geometric references and textual checkpoints. Both share the task specification, sensor access, actuator interface and initial skills; neither ever sees simulator object poses or private evaluator state.
The LLM (gpt-6-astra, high reasoning effort) revises an executable Python policy between reset rollouts: an observation-only scaffold (R0), then three revisions (R1–R3), with no LLM calls during execution. Five paired chains per task train on five shared layouts; a revision is kept only if it solves every previously solved layout plus one more. After training freezes, every retained version is tested on 100 shared held-out layouts.
The LLM (gpt-6-astra, medium reasoning effort) orchestrates shared prior skills in one continuing scene until it succeeds or hits a 200 s wall-clock budget. Failed actions are not reset, and physics keeps running while the model thinks, so waiting costs the same budget as acting. Both agents can request images and depth samples on demand. We evaluate 50 paired layouts per task (300 episodes).
Inside an average tower-relocation episode
| Task | Agent | Success | Time (s) | Model (s) | Tokens (k) | Requests | Images |
|---|---|---|---|---|---|---|---|
| Bridge-tower | Pixel Reasoning | 12/50 | 198.0 | 55.5 | 112.5 | 7.88 | 7.50 |
| Geometric Abstraction | 28/50 | 185.3 | 51.8 | 129.5 | 7.32 | 4.28 | |
| Ring-on-peg | Pixel Reasoning | 49/50 | 115.8 | 90.0 | 214.5 | 11.88 | 10.94 |
| Geometric Abstraction | 50/50 | 77.2 | 45.6 | 101.3 | 6.90 | 2.72 | |
| Tower relocation | Pixel Reasoning | 15/50 | 186.4 | 80.3 | 185.9 | 11.58 | 11.34 |
| Geometric Abstraction | 42/50 | 139.2 | 31.7 | 88.3 | 5.36 | 3.00 |
Continuous demo rollouts of each task in the same simulated SO-100 setup, recorded for this page. They show the physical tasks, not the study's LLM agents: single-policy tasks use hand-written controllers from the repository, and the online tasks use scripted controllers. The result tables above report the actual study episodes. Sparse replays of real study episodes are in the code repository.
Single-policy tasks
Online tasks
@inproceedings{liu2026geometric,
title = {Do Agentic Policies Need to Look at Every Step? Geometric Abstraction for Reasoning and Action},
author = {Liu, Yihang and Chen, Yiming and Zheng, Mingzhe},
booktitle = {CoRL 2026 Workshop on Agentic Robotics},
year = {2026}
}