Freeform Preference Learning: Teaching Robots with Multi-Axis Human Preferences

Paper: 2606.32027 Authors: Marcel Torne, Anubha Mahajan, Abhijnya Bhat, Chelsea Finn Categories: cs.RO, cs.AI, cs.LG

The Gap

Reward engineering for long-horizon robotics is stuck between two bad options. Sparse success/failure labels give you a single bit at the end of a 30-second trajectory — like grading a cooking competition as only “delicious” or “not delicious.” Binary preference learning (à la RLHF) is slightly better: you compare two trajectories and say “A is better than B.” But this collapses every dimension of quality — speed, safety, precision, elegance — into one vague judgment. When a human says trajectory A is better, do they mean it was faster? Safer? More precise? The signal is ambiguous, and the reward model has no way to disentangle these factors.

Prior approaches tried to fix this with task-specific reward shaping (brittle, requires domain expertise) or hierarchical decomposition (requires explicit subtask boundaries, which are hard to define for dexterous manipulation). What’s missing: a way to capture the multi-dimensional structure of human preferences without hand-crafting reward functions or segmenting tasks.

Problem: Reward signal too coarse or too brittle
    |
    v
Assumption: Humans think in multiple quality dimensions
    |         (speed, safety, care...) simultaneously
    v
Method: Let annotators name the axes, then collect
    |    pairwise preferences along EACH axis
    v
Evidence: 38pp improvement over baselines on 6 tasks
    |      + compositionality + test-time steering
    v
Conclusion: Multi-axis preferences unlock richer
             reward signal AND flexible behavior

The Increment

One sentence: Before FPL, human preferences were a single opaque scalar; after FPL, they become a structured vector of named dimensions that can be independently steered at test time.

Core Mechanism

FPL has three phases. Phase 1: Annotation. Instead of asking “which trajectory is better?”, annotators define preference axes in natural language (e.g., “speed,” “carefulness,” “quality of placement”) and then provide pairwise comparisons along each axis independently. For the same pair of trajectories, an annotator might say: “A is faster, but B is more careful.” This yields structured labels: (trajectory pair, axis label, preference direction).

Phase 2: Reward model training. A language-conditioned reward model takes three inputs — a trajectory, a preference axis label (as text), and a partial trajectory (for dense credit assignment) — and outputs an axis-specific reward score. The model is trained with a pairwise ranking loss on each axis separately. At test time, you can query it with any natural-language axis description, including ones not seen during training.

Phase 3: Policy training. A reward-conditioned policy is trained to maximize a *weighted combination of axis-specific rewards. Crucially, the weights can be changed at deployment without retraining: want more carefulness and less speed? Adjust the weights. This is the steering knob.

Training Data:
[traj_a, traj_b] x [axis: "speed", "carefulness", "precise"]
         |                  |
         v                  v
    pairwise pref      axis label
    per axis            (text)
         |                  |
         +--------+---------+
                  |
                  v
       Language-Conditioned
         Reward Model
            [traj, axis] --> scalar r
                  |
                  v
       Reward-Conditioned Policy
            [obs, r] --> action
                  |
                  v
         Test-time: adjust
         axis weights --> new behavior

The cooking competition metaphor: Imagine you’re judging a cooking competition, but instead of one trophy, there are five: Best Taste, Best Presentation, Best Creativity, Best Technique, Best Time Management. Each judge focuses on one criterion and compares dishes pairwise: “Dish A tastes better than Dish B, but B presents better.” A scoring rubric is built for each criterion — this is the reward model. The chef (policy) learns to cook by receiving feedback across all dimensions. Now, here’s the magic: when the restaurant shifts to fine dining, the manager can turn up the “Presentation” and “Technique” knobs without sending the chef back to school. The chef has internalized all five skills and can rebalance emphasis on the fly. That’s FPL’s test-time steering.

Key Concepts

• Language-conditioned reward model: Think of it as a translator between human words and robot learning signals. You feed it a trajectory and a phrase like “how careful is this?” and it returns a number. The “language-conditioned” part means the same neural network handles arbitrary text descriptions — it doesn’t memorize a fixed set of axes. Under the hood, the text is embedded into a vector, concatenated with trajectory features, and mapped to a scalar. This generalizes because the model learns a shared representation of “what the words mean” and “what good behavior looks like” that can be composed freely.

• Reward-conditioned policy: A standard policy maps observations to actions. A *reward-conditioned policy also takes a reward vector (one scalar per axis, or a weighted combination) as input. Think of it as a thermostat: the desired temperature (target reward) is an input, and the policy is the heating/cooling system that adjusts to reach it. During training, you sample different reward targets so the policy learns to handle any setting. At deployment, changing the target is free — no retraining needed.

• Compositionality beyond the data: The training data contains preferences along individual axes (speed vs. not-speed, safety vs. not-safety). But at test time, you can combine axes in ways never seen: “be fast AND careful.” The reward model generalizes because each axis score is computed independently and then linearly combined. The policy generalizes because it was trained on a *distribution of reward targets, not a fixed one. This is similar to how a bilingual person can produce sentences they’ve never heard by recombining known words and grammar.

Framework Shift

Before (mainstream approach):        After (this paper):

  traj_A, traj_B                      traj_A, traj_B
       |          |                       |
       +----+-----+                  +----+----+----+
            |                        |    |    |    |
            v                        v    v    v    v
     "Which is better?"         "speed" "safe" "care" "precise"
      one ambiguous                  pairwise pref per axis
       comparison                        |
            |                            v
            v                     axis-specific rewards
     single reward r                    |    |    |    |
            |                           +----+----+----+
            v                                 |
     one fixed policy                   weighted combination
                                        (adjustable at test time)
                                              |
                                              v
                                         steerable policy

From one opaque “better/worse” judgment to a structured vector of named, independently steerable quality dimensions — the core shift is treating preference as a multi-axis coordinate system rather than a single scalar.

Expert Assessment

Problem choice: This is a genuine gap, not manufactured. The reward engineering bottleneck is well-recognized in robotics, and the binary preference collapse is a real limitation of RLHF-style methods. The paper sits at a natural convergence point: LLMs for language understanding + reward learning + robot learning. The timing is right.

Method maturity: The core insight — let humans name their preference axes — is simple and elegant. The implementation (language-conditioned reward model + reward-conditioned policy) is competent but not novel in its components; the novelty is the *combination and the annotation protocol. One concern: the method assumes axis independence (each axis scored separately, then linearly combined). Real human preferences may interact non-linearly (speed matters more when safety is already high). The paper doesn’t address this.

Experimental integrity: The 38pp improvement is strong and consistent across tasks. Baselines include sparse reward and binary preference (RLHF-style), which are fair comparisons. Four real-world tasks is a good sign — this isn’t just simulation magic. However, the tasks are all tabletop manipulation with relatively short horizons. The “long-horizon” claim in the abstract deserves more scrutiny: how long is long? Would this scale to minutes-long assembly tasks? Also, the annotation burden is not carefully measured — how many axis-specific comparisons are needed, and how does annotation cost compare to binary preferences?

Writing quality: The paper is well-structured but cuts corners on two fronts. First, the related work section is thin — there’s substantial prior work on multi-objective RL and reward decomposition that deserves deeper engagement. Second, the failure modes are underexplored: what happens when an annotator’s axis definitions are ambiguous or inconsistent? How sensitive is the method to noisy axis labels? A deeper analysis section here would elevate the paper significantly.

Verdict: weak accept — the multi-axis framing is genuinely useful and the results are strong, but the method’s simplicity (linear combination of independent axes) may limit its ceiling, and the experimental scope needs broader validation.

Takeaways

Three concrete ideas to steal:

1. Multi-axis annotation protocol: Even if you don’t use FPL’s reward model, you can collect richer human feedback by asking annotators to specify *which dimension they’re comparing on. This works for any RLHF pipeline — language models, game AI, recommendation systems. The annotation cost is slightly higher, but the signal-to-noise ratio improves dramatically.

2. Language as a policy steering interface: The idea of conditioning a reward model on natural-language axis descriptions means you can define new objectives at test time without re-labeling data or retraining. This is a powerful pattern: train a general-purpose scorer, then steer with text. Transferable to content generation, dialogue systems, any domain where preferences are multi-dimensional.

3. Reward-conditioned policies as adjustable thermostats: Training a policy on a *distribution of reward targets (not a single target) gives you a single model that can be steered post-hoc. This is cheaper than training separate policies per objective. If you’re building any system where users might want to adjust behavior after deployment, this architecture is worth considering.