The Binary-Preference Bottleneck
For years, reinforcement learning from human feedback (RLHF) for robots has borrowed a playbook from large language models: present a human evaluator with two trajectory clips and ask, "Which is better?". That binary signal works reasonably well for chatbots where 'better' is roughly one-dimensional (helpful vs. unhelpful). But in long-horizon manipulation—folding laundry, setting a table, rearranging a cluttered shelf—'better' collapses several competing axes into an ambiguous thumbs-up: speed often fights with carefulness, placement accuracy fights with smoothness, and task completion fights with safety. A single binary label cannot express that trajectory A was faster but B was gentler with the glass.
That observation is the starting point of Freeform Preference Learning (FPL), a paper posted to arXiv on June 30, 2026 (arXiv:2606.32027) by Marcel Torne, Anubha Mahajan, Abhijnya Bhat, and Prof. Chelsea Finn at Stanford's IRIS Lab.
Letting Annotators Speak in Axes
FPL replaces the forced pairwise choice with a freeform, multi-axis annotation interface. Annotators first define their own natural-language axes that matter for the task—examples from the paper include speed of task completion, carefulness / safety around fragile objects, quality of placement, and how smoothly the motion was executed. For each pair of trajectories, they then provide a pairwise preference per axis, rather than one global winner.
Those per-axis preferences train a language-conditioned reward model: given a trajectory and a natural-language axis label (e.g. 'be careful'), the model outputs a scalar reward specific to that axis. The authors then train a reward-conditioned policy using a re-labeling trick reminiscent of goal-conditioned RL—at training time they sample an axis, train the policy to maximize that axis's reward, and expose the axis embedding as an additional input to the policy network.
Results: +38 Points, and Test-Time Steering Without Retraining
The team evaluated FPL across four real-robot long-horizon tasks (including table-setting with fragile dishes and cloth-folding style tasks) and two simulated benchmarks, comparing against:
- Sparse-reward RL (success-only signal)
- Standard binary-preference RLHF (global 'which is better?' labels)
- Multi-objective RL baselines with hand-specified reward terms
FPL outperformed the strongest binary-preference baseline by 38 percentage points in average task success, and matched or beat hand-tuned multi-objective rewards without requiring engineers to write down numeric reward terms. Perhaps more importantly, because the policy is conditioned on an axis embedding, users can steer behavior at test time simply by changing the language prompt—switching from 'complete the task quickly' to 'handle the objects carefully' produces visibly different behaviors without any fine-tuning, retraining, or reward re-weighting. The paper reports emergent compositionality: combining axes (e.g. 'fast and careful') produces policies that trade off the two dimensions even though that specific combination was never labeled.
Dense Progress Signals Without Subtask Labels
A subtle but valuable side-effect: the per-axis reward model functions as a dense, multi-dimensional progress metric across the rollout, without requiring humans to manually segment the task into subtasks or engineer distance-to-goal features. That's meaningful for long-horizon work where hand-designed dense rewards are notoriously brittle.
What This Changes for Physical-Robot RLHF
If FPL's gains hold up beyond the paper's tasks, it shifts the conversation on RLHF for robots in three ways. First, it moves human feedback from a monolithic ranking problem to a multi-dimensional, language-grounded one—much closer to how we actually critique motion. Second, the reward-conditioned policy architecture gives operators a natural-language knob at deployment time, which is operationally useful when the same robot must switch between 'go fast' (warehouse) and 'be gentle' (homecare) modes. Third, by collapsing the work of specifying reward terms into annotation, it lowers the barrier for non-engineers—line leads, clinicians, home users—to shape robot behavior without writing code.
The paper is open-access on arXiv and accompanied by a project site with videos of real-robot rollouts at freeform-pl.github.io/fpl.website. Code and datasets were not yet released at time of posting, but the authors have signaled an open-source release is planned.
Limitations and Open Questions
The authors are careful to flag two open issues the method does not yet solve. First, FPL does not automatically select how strongly to weight each axis at deployment time—users still have to pick which axis (or combination) to optimize, and the paper does not propose a meta-controller for that. Second, handling out-of-distribution behaviors (truly novel motion styles that fall outside what annotators labeled) remains fragile. Those are natural next steps, and the framing—treating human preferences as a language-defined coordinate system rather than a scalar score—gives the community a cleaner interface to build on.
Source: "Freeform Preference Learning for Robotic Manipulation," Torne, Mahajan, Bhat, Finn, arXiv:2606.32027, submitted 30 June 2026.
