TU Delft Lets Passengers Talk Their Way to a Custom Autonomous Ride

Researchers at Delft University of Technology (TU Delft) in the Netherlands have built a system that uses a large language model to translate natural-language passenger requests — such as "I am running late, go fast" — into real-time adjustments of a self-driving car's control system. The team, led by postdoctoral researcher Diego Martinez-Baselga, posted its preprint on arXiv and will present the work at the IEEE Intelligent Transportation Systems Conference in September.

The research targets a real gap in how autonomous vehicles work today. Self-driving cars balance parameters like speed, acceleration, and turn smoothness through a software component called the motion planner. Engineers tune that planner before the vehicles hit the road, which leaves passengers almost no way to adjust the driving style on the fly. Human preferences, by contrast, shift constantly — depending on whether someone is in a hurry, feeling carsick, or stuck in heavy traffic.

The stakes are significant for the autonomous vehicle industry, where rider acceptance remains a stubborn obstacle. If passengers cannot tell a robotaxi to ease off the accelerator or take turns more gently, the technology stays rigid in ways human-driven cars are not. "The motion-planning problem is not only about reaching a place while avoiding collisions, it's also how you do it," Martinez-Baselga says. "The motivation here is trying to make the way the autonomous car drives adaptable by end users easily, just by talking to the car."

The system does not hand users direct control over driving decisions. Instead, it tunes the parameters of a safety-aware motion-planning algorithm, keeping the vehicle's behavior within safe bounds. It relies on a model predictive-path integral controller the researchers previously developed, which identifies multiple candidate paths to a goal and scores them on criteria including speed, steering angle, and collision probability. It then computes an optimal path from the best-scoring trajectories.

The team combined this controller with OpenAI's GPT-4o-mini. The model receives the user's prompt plus a natural-language description of the driving scenario. In the study, the researchers wrote those scenario descriptions by hand, but Martinez-Baselga says a car's perception system could ultimately supply them directly. The LLM does not tweak the controller's settings outright. It uses the prompt to rate the relative importance of the trajectory-scoring criteria, adjusting each one up or down around a safe baseline the researchers set. If a user says they feel dizzy, the model dials up parameters favoring smooth steering and gentle acceleration.

Before any change takes effect, the model presents the passenger with a plain-language description of the adjustments and asks for confirmation. The passenger can approve the plan or push back with further suggestions. This human-in-the-loop design lets riders catch misinterpretations and handle the inherent subjectivity of instructions like "go faster." If the car's behavior doesn't match expectations, the passenger can simply follow up "as you would do if you were in a taxi or with a friend that is driving," Martinez-Baselga says.

Why keep the LLM one step removed from actual driving decisions? Earlier research explored using LLMs and vision-language models to direct self-driving decision-making directly, but Martinez-Baselga points to two problems: response times too slow for fast-paced driving decisions, and the inability of these models to provide the concrete performance guarantees a deterministic motion planner can.

In tests inside the nuPlan self-driving simulator, using scenarios that involved merging onto a busy highway, the system adjusted the controller across eight different prompts in ways matching user intent. Requests for a more comfortable ride increased smoothness; requests signaling urgency produced higher speeds.

The TU Delft team is not the first to use an LLM to tune a motion planner. Nicolas Baumann, a Ph.D. student at ETH Zurich, published research last year in which an LLM adjusted the parameters of a model racing-car controller, letting users change driving style and give concrete commands like "reverse the car" or "maintain a specific speed." Baumann says the strength of this architecture is containment: even if the model hallucinates, it cannot do anything dangerous. "You get the possibility of language interaction, but you can guarantee that it is going to be within the constraints of this classical controller, so you can bake in safety," he says. The trade-off, he adds, is that setting those constraints demands considerable engineering work.

Provable safety requires going further, according to Matthias Althoff, a professor of cyberphysical systems at the Technical University of Munich. His group built a system in which an LLM suggests driving decisions that a mathematical process then checks against traffic rules and predictions of other road users' behavior before committing. The Delft paper offers no such verification. "As with any LLM, it is not guaranteed that the result is correct," Althoff says. "For that reason, we safeguard the decisions of the LLM in our works."

For now, the Delft results exist only in simulation, with handwritten scenario descriptions standing in for real perception data. Closing that gap — and hardening the safety baseline against formal verification standards — will determine whether talking to your car becomes more than a demo.