HumanoidGPT · Part of The Humanoid Group

LLMs as Task Planners

Ask a humanoid to tidy a kitchen and something has to turn that request into a sequence of actions the robot can perform. Large language models increasingly do that job. This guide explains how they plan and where they go wrong.

By Arjun Rao · Updated October 2026

From a request to a list of skills

In many LLM-based planners, the language model does not move the robot directly. It sits above a library of skills the robot already has, such as pick up, open, place or go to, and decides which to call and in what order. Given a request like 'I spilled my drink, can you help?', a planner might choose to find a sponge, pick it up, bring it over and wipe the table. The skills themselves run as separate learned or programmed controllers. This division of labour lets the language model contribute everyday knowledge about how chores are usually done, while the robot's own software handles grasping, balance and motion.

Grounding plans in what the robot can do

A language model knows a lot about kitchens in general but nothing about the robot standing in this one. Left unchecked, it may propose steps the robot cannot perform or that make no sense in the current scene. The SayCan project at Google tackled this by scoring each step the model suggested with value functions that estimate how likely the corresponding skill is to succeed from the robot's present state, so the chosen step is both useful and feasible. In the authors' framing, the robot serves as the language model's hands and eyes. Other systems ground plans with scene descriptions from a vision model, or by feeding back the result of each step before choosing the next.

Writing plans as code

Another approach asks the model to write a short program rather than a list of words. In Code as Policies, Google researchers prompted code-writing language models with examples and a set of perception and control functions, and the models composed new robot programs from fresh instructions. Code brings real advantages to planning: it can loop, test conditions, do geometry with maths libraries and turn vague words such as 'a bit faster' into numbers. It is also easier for an engineer to read before it runs. The weakness is shared with any generated code, since it can call functions wrongly or assume something false about the scene, so it needs limits and checks.

Knowing when to ask for help

Language models sound confident even when they are guessing, which is risky when their output moves a machine. If someone says 'put the bowl in the microwave' and two bowls sit on the counter, the planner should ask rather than pick one. KnowNo, from Princeton University and Google DeepMind, uses a statistical method called conformal prediction to measure a planner's uncertainty and request human help when the options are ambiguous, while giving formal guarantees on how often tasks are completed. For buyers, the practical points are how a robot's planner handles unclear instructions, whether it confirms before irreversible steps and whether its plans are limited to actions that have been tested.

Sources and further reading

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances — Robotics at Google (arXiv).
SayCan pairs a language model with skill value functions so a mobile robot only picks steps it can actually carry out.
Code as Policies: Language Model Programs for Embodied Control — Google (arXiv).
Language models write short robot programs from instructions, calling perception and control functions and maths libraries.
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners — Princeton University and Google DeepMind (arXiv; CoRL 2023).
KnowNo uses conformal prediction so an LLM planner asks a person for help when an instruction is ambiguous.

Common questions

Can a general chatbot plan tasks for a humanoid robot?

It can suggest steps, but on its own it knows nothing about the robot's skills, surroundings or limits. Useful systems connect the language model to the robot's skill library, perception and safety checks, and the combination is tested on real tasks before anyone relies on it.

Does an LLM planner need an internet connection?

That depends on where the model runs. Larger models are often hosted remotely, while smaller ones can run on the robot's own computer. Ask the maker what happens to planning, and to the robot's behaviour, if the connection drops halfway through a task.

Robot Foundation Models — Large models trained on broad data, then adapted to control robots across many tasks.
Vision-Language-Action Models — Models that see a scene, read an instruction and output the robot's next movements.
Data & Simulation — Teleoperation, demonstrations and simulated worlds: where robot training data comes from.
Evaluating Humanoid AI — How to judge a humanoid's intelligence on your own tasks, not in a demo video.
LLMs as Task Planners — How large language models break a spoken request into steps a robot can actually carry out.
World Models for Robots — Learned simulators and video prediction that let robots imagine the result of an action before they act.
On-Device or Cloud AI — Where a robot's AI runs, and how latency, privacy, connectivity and power shape that choice.
Open-Source Robot Models — Open robot foundation models and datasets: what gets released, what open really means and how to use them.
AI Regulation for Robots — How the EU AI Act, NIST's AI Risk Management Framework and India's AI guidelines apply to robots.
Latest updates — Robot models are now tested in unseen homes and judged on safety, yet benchmarks still show wide gaps. These figures and dated updates show what progressed, what failed and how to read the claims.
About us — What we cover, how we source every figure, and who publishes this site.

Visit The Humanoid Group

People also search for LLM task planning, large language model robot planning, language model robot actions, SayCan, Code as Policies and robot skill library.