Teaching Robots to Peel: Learning Fine-Grained Manipulation Through Human Preference

Paper: 2603.03280 Authors: Toru Lin, Shuying Deng, Zhao-Heng Yin, Pieter Abbeel, Jitendra Malik Categories: cs.RO, cs.AI, cs.CV, cs.LG, eess.SY

The Gap

Existing robot manipulation research has conquered tasks with clear success criteria—pick this, place that, insert peg A into hole B. But a vast class of real-world tasks resists this binary framing: food prep, surgery, craftsmanship. These tasks share two nasty properties: (1) they’re contact-rich and force-sensitive (press too hard, you bruise the apple; too soft, the peel stays on), and (2) success is continuous and subjective (what counts as “well peeled” varies by person and context). Prior work either avoids these tasks entirely, or tries to engineer quantitative reward functions—an approach that fails because you can’t capture “good peeling” in a formula without losing what makes it good.

The logical chain:

Problem: Contact-rich tasks with subjective quality
    |
    v
Assumption: Humans can judge quality even if they can't formalize it
    |
    v
Method: Learn robust base policy (imitation) + refine via preference feedback
    |
    v
Evidence: 90%+ success on varied produce, 40% improvement from refinement
    |
    v
Conclusion: Preference learning bridges the gap between objective dynamics
            and subjective task quality

The Increment

One sentence: Before this paper, robots couldn’t learn tasks where “done well” is subjective and contact-sensitive; after, they can bootstrap from demonstrations and align to human quality judgments using preference feedback.

Core Mechanism

The method has two stages. Stage one builds a foundation: collect human demonstrations of peeling while recording force/torque data, then train a policy via behavioral cloning that conditions on both visual observations and haptic feedback. This gives you a policy that can generalize across object variations (different cucumber sizes, apple shapes) because it’s learned the force-sensitive dynamics—when to press, when to ease up.

Stage two refines this foundation using human preferences. Generate trajectory pairs from the base policy, show them to humans who pick the better one (cleaner peel, less waste, smoother motion). Train a reward model on these preferences using Bradley-Terry modeling. Then finetune the policy against this learned reward using PPO, with the reward model providing dense feedback that captures subjective quality.

Stage 1: Imitation Learning
    Human demos --> [Force-aware policy] --> Base behavior
         |                                        |
    (visual + haptic)                    (generalizes to variations)

Stage 2: Preference Refinement
    Base policy --> Trajectory pairs --> Human picks better
                          |                      |
                          v                      v
                    [Reward model] <-- Bradley-Terry learning
                          |
                          v
                    PPO finetuning --> Refined policy

Think of it like learning to cook. Stage one is watching a chef and copying their knife technique—you learn the basic motions and when to apply pressure. But “good” cooking isn’t just correct motions; it’s about texture, presentation, taste. Stage two is like having the chef taste your dishes and tell you which attempts are better. You don’t get a recipe (explicit reward function), but through repeated comparisons, you internalize their standards. The reward model is your developing palate—it learns to predict what the chef will prefer. The policy refinement is you adjusting your technique based on that internalized sense of quality.

Key Concepts

• Force-aware imitation learning: Standard imitation learning trains policies on visual observations—what the robot sees. But contact-rich tasks depend critically on what the robot feels. When you peel a potato, you’re constantly adjusting pressure based on resistance. Force-aware imitation means the policy conditions on force/torque sensor readings alongside vision. During data collection, humans demonstrate while the robot records both modalities. The trained policy then uses haptic feedback as a real-time signal: if resistance suddenly drops (knife broke through skin), ease up; if resistance is high (thick peel), maintain pressure. This is why the base policy generalizes—it’s learned the physics of peeling, not just the visual appearance of a specific cucumber.

• Preference-based reward learning: You can’t write down what “well peeled” means mathematically, but you can look at two peeling attempts and say which is better. Preference learning exploits this asymmetry. Instead of asking humans to score trajectories (hard, inconsistent), you show pairs and collect binary preferences (easier, more reliable). The Bradley-Terry model assumes each trajectory has a latent quality score, and the probability of preferring trajectory A over B follows a logistic function of their score difference. Train a neural network to predict these scores from trajectory features (visual outcomes, force profiles, motion smoothness). Now you have a differentiable reward signal that captures human judgment, which you can optimize against using standard RL.

• Generalization across object variations: The key challenge in manipulation is that every apple, cucumber, or potato is different—different sizes, shapes, skin thickness, firmness. The force-aware policy handles this by learning the relationship between sensed forces and appropriate actions, rather than memorizing specific trajectories. When encountering a new object, the policy uses real-time haptic feedback to adapt: if the peel is thicker than expected, it adjusts pressure accordingly. This is analogous to how humans peel—we don’t memorize exact motions, we feel our way through the task.

Why It Matters

This work opens the door to teaching robots subjective, quality-sensitive tasks that have been out of reach. The implications extend beyond peeling:

• Food preparation: Slicing, dicing, filleting—tasks where “done right” depends on texture, appearance, and human preference
• Medical procedures: Suturing, tissue manipulation—where force sensitivity and subjective quality are critical
• Craftsmanship: Sanding, polishing, assembly—tasks requiring feel and judgment

The two-stage approach is generalizable: use imitation to learn the physics and dynamics, then use preference feedback to align with human quality standards. This pattern could apply to any task where success is continuous and subjective rather than binary and objective.

The Results

The system achieves 90%+ success rates across cucumbers, apples, and potatoes with varying properties. The preference refinement stage improves performance by 40% over the base imitation policy. Crucially, the learned reward model transfers: a model trained on cucumber preferences generalizes to apples, suggesting it’s capturing general principles of “good peeling” rather than object-specific heuristics.

The force-aware policy is essential—ablations show that removing haptic feedback drops success rates dramatically. Vision alone isn’t enough for contact-rich tasks; you need to feel what’s happening.

Critical Analysis

Strengths: The combination of force-aware imitation and preference learning is elegant and practical. The results demonstrate real generalization across object variations. The approach sidesteps the impossible task of engineering reward functions for subjective quality.

Limitations: Preference collection is expensive—humans must watch and compare many trajectory pairs. The method assumes access to force/torque sensors, which not all robots have. The tasks studied (peeling) are relatively structured; it’s unclear how well this scales to more open-ended manipulation.

Open questions: Can we reduce the preference data requirements through better active learning? How does this approach handle tasks with multiple valid strategies (e.g., different peeling techniques)? Can we learn from implicit preferences (watching humans work) rather than explicit comparisons?