SafeGen-LLM: Enhancing Safety Generalization in Task Planning for Robotic Systems

Paper: 2602.24235 Authors: Jialiang Fan, Weizhe Xu, Mengyu Liu, Oleg Sokolsky, Insup Lee, Fangxin Kong Categories: cs.RO, cs.AI

Abstract

Safety-critical task planning for robotic systems faces a fundamental trilemma: classical planners lack scalability, reinforcement learning methods fail to generalize, and base large language models cannot guarantee safety constraints. SafeGen-LLM addresses this challenge through a two-stage post-training framework that enables LLMs to generate safe, constraint-compliant plans while generalizing to novel safety properties across multiple domains. The approach constructs a multi-domain PDDL3 benchmark with explicit safety constraints, then applies supervised fine-tuning on constraint-compliant planning data followed by Group Relative Policy Optimization guided by formal verification-derived reward machines and curriculum learning. Experimental results demonstrate that SafeGen-LLM achieves superior safety generalization and outperforms frontier proprietary models across diverse planning tasks and input formats.

Key Contributions

• Construction of a comprehensive multi-domain PDDL3 benchmark with explicit safety constraints for evaluating safety-aware task planning
• A two-stage post-training framework combining SFT for planning syntax/semantics learning and GRPO for safety alignment
• Integration of fine-grained reward machines derived from formal verification to guide reinforcement learning toward safety-compliant solutions
• Curriculum learning strategy to progressively handle increasingly complex planning tasks
• Demonstration of strong safety generalization to novel constraints across multiple domains and input modalities

Methodology and Technical Framework

The SafeGen-LLM framework operates through a carefully designed two-stage training pipeline. In the first stage, supervised fine-tuning is applied to a base LLM using a curated dataset of constraint-compliant planning examples. This stage teaches the model the syntax and semantics of PDDL3, including how to represent states, actions, preconditions, effects, and crucially, safety constraints.

The second stage employs Group Relative Policy Optimization (GRPO), a reinforcement learning technique that optimizes the model’s policy relative to groups of generated plans. What distinguishes this approach is the use of reward machines—formal structures derived from the safety constraints through automated verification tools. These reward machines provide fine-grained feedback signals that guide the model toward plans that satisfy safety properties while achieving task objectives.

The curriculum learning component progressively increases task complexity, allowing the model to build competence on simpler problems before tackling more challenging scenarios. This staged approach prevents the model from being overwhelmed by complex constraint interactions early in training and improves overall convergence.

Experimental Results and Performance Analysis

SafeGen-LLM was evaluated across multiple planning domains including logistics, manufacturing, autonomous driving, and healthcare robotics. The benchmark includes both seen and unseen safety constraints to test generalization capabilities. Results show that SafeGen-LLM achieves significantly higher safety satisfaction rates compared to base LLMs and even frontier proprietary models like GPT-4 and Claude.

Particularly notable is the model’s ability to generalize to novel safety properties not encountered during training. When presented with new constraint types, SafeGen-LLM maintains high compliance rates, suggesting that the training framework instills a deeper understanding of safety reasoning rather than mere memorization of specific constraints.

The framework also demonstrates robustness across different input formats. Whether receiving formal PDDL specifications or natural language descriptions of planning problems, SafeGen-LLM generates plans that respect safety boundaries while achieving task goals. This flexibility is crucial for real-world deployment where users may not be familiar with formal specification languages.

Performance metrics indicate that the GRPO stage with reward machines is essential for achieving high safety compliance. Ablation studies show that removing either the formal verification-derived rewards or the curriculum learning component results in degraded performance, validating the importance of each architectural choice.

Implications for Safe Robotic Systems

This work has significant implications for deploying LLMs in safety-critical robotic applications. Traditional approaches to safe planning either sacrifice scalability (classical planners) or struggle with generalization (RL methods). SafeGen-LLM demonstrates that LLMs can be effectively aligned with safety requirements through appropriate training frameworks, opening new possibilities for flexible yet safe autonomous systems.

The formal verification integration is particularly important. By grounding the training process in mathematically rigorous safety specifications, the approach provides stronger guarantees than purely data-driven methods. This bridges the gap between the flexibility of learned models and the reliability requirements of safety-critical systems.

The generalization capabilities suggest that SafeGen-LLM could adapt to new operational contexts with minimal retraining. As robotic systems encounter novel environments and constraints, the model’s ability to reason about unfamiliar safety properties becomes invaluable. This could accelerate deployment timelines and reduce the engineering effort required to adapt systems to new scenarios.

Takeaways

1. SafeGen-LLM successfully combines the flexibility of LLMs with formal safety guarantees through a two-stage training framework integrating SFT and GRPO with verification-derived reward machines.

2. The approach achieves strong generalization to novel safety constraints across multiple domains, addressing a critical limitation of both classical planners and pure RL methods.

3. Integration of formal verification into the training loop provides mathematically grounded safety alignment, bridging learned flexibility with reliability requirements.

4. Curriculum learning proves essential for handling complex planning tasks with multiple interacting constraints, enabling progressive skill development.

5. The framework’s robustness across input formats (PDDL and natural language) enhances practical applicability for users with varying technical backgrounds.

6. Experimental results demonstrate superiority over frontier proprietary models, suggesting that specialized training frameworks can outperform general-purpose LLMs in safety-critical domains.