What Objects Enable, Not What They Are: Functional Latent Spaces for Affordance Reasoning

Rohan Siva1,*, Neel P. Bhatt1,2,*, Yunhao Yang1,2,*, Seoyoung Lee1, Nishant Gadde1, Christian Ellis1,2, Alvaro Velasquez2,3, Zhangyang Wang1, Ufuk Topcu1,2
1The University of Texas at Austin 2Neurosymbolic Intelligence 3University of Colorado Boulder
*Equal Contribution
arXiv Code Hugging Face Dataset Hugging Face Model Checkpoint Demo

Full demo video: method overview and real-robot deployment on manipulation and navigation tasks.

TLDR: A4D reasons about what objects enable, not what they are, mapping visual observations into a functional latent space built from affordance axes like movable ↔ fixed, and discovering new affordances when the existing ones are insufficient.

Abstract

Existing robot planning systems rely on appearance-based reasoning, where visual observations are encoded into latent spaces organized around object appearances (e.g., recognizing a "cart" based on how it looks). However, planning requires reasoning about task-relevant functionalities of objects (e.g., whether an object is "movable"), which appearance-based latent spaces do not capture. As a result, existing approaches struggle to generalize to novel robot-object interactions.

We address this limited generalizability through affordance reasoning, enabling planning based on task-relevant object functionalities instead of appearance alone. We introduce A4D, which maps visual observations into a shared functional latent space structured around affordances (e.g., "movable"). By projecting visual observations into this functional latent space and measuring their proximity to affordances, A4D infers functionalities relevant to the observed object.

Furthermore, we introduce an affordance discovery mechanism that expands the latent space to handle unseen scenarios where existing affordances are insufficient. A4D uses proximity in the functional latent space to quantify uncertainty in affordance inference and selectively triggers affordance discovery. We evaluate A4D across several planning tasks involving diverse and unseen affordances. A4D achieves 94% inference accuracy on existing affordances outperforming state-of-the-art approaches by over 15% points, improves new-affordance inference accuracy from 70% to over 90% with fewer than 10% of the original training data, and enables 100x faster inference.

Try A4D

Every result below is the fine-tuned model run on a real image: it projects the image onto each affordance axis and calibrates the projection into a confidence score. Start from a highlight, or pick any object and affordance yourself.

Highlights: confident & decisive
Highlights: low confidence, A4D defers to a VLM
Run it yourself
A4D encodes the image with the fine-tuned CLIP encoder, projects it onto the chosen affordance axis, and calibrates the projection margin into a confidence score. Pick an object and an affordance, then hit Run A4D.
Calibrated confidence:
Similarity to
Similarity to

Framework Overview

A4D maps observations into a latent space built around what objects enable, rather than how they look, and expands that space on its own when it runs into something new.

  • Appearance-based latent spaces capture how an object looks, not what it enables, and whether something is "movable" has little to do with its shape or color.
  • We introduce A4D: visual observations and affordance descriptions are mapped into a shared functional latent space, built from affordance axes like movable ↔ fixed.
  • Projection proximity onto each axis is calibrated into an uncertainty estimate, so A4D knows when it doesn't know.
  • When uncertainty is high or existing affordances fall short, A4D triggers affordance discovery: a VLM proposes and labels new affordances (e.g., "traversable"), which fold back into the latent space through offline learning.
A4D generation and discovery pipeline building up step by step

Demonstrations

Four clips below pair the external view with the on-robot view for each task shown in the video above, followed by the qualitative results.

Manipulation: “place the strawberry in the can.” External view (left) and on-robot view (right).

Manipulation: “stack the cube and orange on the book.” External view (left) and on-robot view (right); the green tag is A4D's live affordance inference.

Navigation: “go to the far end of the hallway.” External view (left) and on-robot view (right). A4D infers the blocking cart is movable and clears the path.

Navigation: a second hallway run. External view (left) and on-robot view (right).

Cross-Scenario Decision Making

The same framework picks the right object, or the right new affordance, whatever the scenario calls for.

Qualitative demonstrations

Cross-Scenario Decision Making. A4D performs affordance-based decision making across diverse scenarios. In the first example, the planner selects the cart as the most movable object for the task. In the second example, existing affordances are insufficient, triggering uncertainty-guided affordance discovery and introducing a new traversable affordance to successfully complete the task.

Cross-Platform Transfer

The same affordance pipeline, unmodified, running on a different robot entirely.

Robot arm deployment

Cross-Platform Transfer. A4D transfers across robot platforms and domains. The same affordance generation, inference, and discovery framework is deployed on a tabletop robot arm, demonstrating that affordance reasoning is not tied to a specific robot platform. By conditioning generation and labeling on platform-specific capabilities, A4D adapts affordance predictions to new robots and tasks without modifying the underlying framework.

Browse the Dataset

A look at the raw, human-verified affordance labels A4D trains and calibrates on: ground truth, not model predictions. These objects are separate from the ones in Try A4D above. Each image carries all ten affordances; pushable and traversable were added later through affordance discovery, so a handful of labels here fill in our best judgment where the discovery pass didn't reach that object.

Ground-truth affordance labels
← → keys

View Full Dataset on Hugging Face

Related Links

Affordance Generation

Open-vocabulary affordance discovery with a VLM.

View Notebook

Classification & Uncertainty

CLIP-based affordance classification with calibrated uncertainty.

View Notebook

OpenCLIP

The vision-language backbone fine-tuned for affordance inference.

Explore Repository

A4D Dataset

552 labeled images across 29 object classes and 10 affordances.

View Dataset

BibTeX

@inproceedings{siva2026objectsenablearefunctional,
            title={What Objects Enable, Not What They Are: Functional Latent Spaces for Affordance Reasoning},
            author={Rohan Siva and Neel P. Bhatt and Yunhao Yang and Seoyoung Lee and Nishant Gadde and Christian Ellis and Alvaro Velasquez and Zhangyang Wang and Ufuk Topcu},
            year={2026},
            booktitle={Proceedings of the Tenth Conference on Robot Learning},
            address={Austin, TX, USA},
            publisher={PMLR},
      }