Oversight boosts reliability of language-model robot planning

What the study found: The authors report that augmenting large language model (LLM) planners with symbolic planning oversight improved reliability and repeatability for embodied task execution. They also present a more transparent way to define hard constraints than traditional prompt engineering.
Why the authors say this matters: The study suggests this approach may help preserve the flexibility and generalizability of LLM-based robot planners while reducing problems from hallucinations and unclear prompt engineering.
What the researchers tested: The researchers introduced a planning method that combines LLM planners with symbolic planning oversight and tested it in simulated environments and on a real-world quadruped robot. They compared performance across several embodied tasks, including tasks requiring complex reasoning and interaction with humans in realistic scenarios.
What worked and what didn't: In simulation, the approach outperformed current state-of-the-art methods. On the real robot, it achieved 75% task success, compared with 50% for a pure LLM planner and 14.3% for a symbolic planner across several embodied tasks.
What to keep in mind: The abstract does not describe detailed limitations, and the summary available here is limited to the information provided. The results are reported for simulated environments and one real-world quadruped robot setting.

Key points

• The study reports improved reliability and repeatability when symbolic planning oversight is added to LLM planners.
• The authors say their method provides a clearer way to define hard constraints than traditional prompt engineering.
• In simulation, the approach outperformed current state-of-the-art methods.
• On a real-world quadruped robot, the method achieved 75% task success versus 50% for a pure LLM planner and 14.3% for a symbolic planner.
• The tasks included complex reasoning and interaction with humans in realistic scenarios.