Behind the AI Agent Boom, Humans Are Still Making the Phone Calls

When an AI agent offers to negotiate your cable bill by phone, there is a real chance a human contractor will make that call instead of the software. Startups building consumer AI agents — including one led by former Yahoo CEO Marissa Mayer — are quietly routing tasks to people when frontier models from Meta and OpenAI fail to complete them.

Mayer now runs agent startup Dazzle AI. She says today's top-tier AI models are good at surfacing home renovation services for consumers but can stumble badly when the task turns to booking and scheduling, such as arranging a rooftop upgrade.

"We're relying on the frontier models to give us good suggestions and a good candidate list, but we're checking that over to make sure that we're really delivering the type of service that we can stand behind," Mayer said, describing how Dazzle hands certain tasks to human contractors.

The disclosure matters because Meta, OpenAI and others are marketing digital assistants as autonomous tools capable of browsing the web and acting on a user's behalf, from shoe shopping to bill negotiation. The gap between the marketing and the machinery sits at the center of the current agent boom.

Which tasks do AI agents actually fumble?

The failure points are consistent across companies. Meghan Joyce, CEO of digital agent startup Duckbill, said the barriers typically involve tasks that would require an AI agent to make a phone call, schedule a doctor's appointment on someone's behalf, or reserve a couch scrubbing from a local cleaning service.

For anything demanding real-world judgment, Joyce said, humans still beat AI — and companies building consumer-grade digital assistants will need to patch in real people to take over.

That is why Duckbill recently debuted an application programming interface (API), a software connection between products, aimed at companies building personal agents that want access to human contract workers. Companies making agents can use the API to delegate the tasks their agent is likely to botch. Duckbill then translates those tasks into small, standardized units the average adult can complete consistently.

"We work on a model that is not dissimilar from Uber, where we boil tasks down into atomic bite-sized pieces," Joyce said, referencing the ride-hailing company where she previously worked. "And then we track what happens."

The takeaway is simple: the agent layer is partly an interface, with people doing the hard parts behind it.

How badly can a frontier model fail?

Mayer offered a concrete case from a Dazzle AI super user who asked the technology to handle several home renovation projects at once: a roof replacement, a broken window and the purchase of a glass tabletop.

The agent confused window glass with tabletop glass, Mayer said, and ended up "trying to figure out if he could make the tabletop fit in the window."

The mishap was short-lived because Dazzle's technology flagged the task and outsourced it to a person. "Within a few seconds, the human had things sorted out. Whereas an agent would have burned a ridiculous number of tokens," Mayer said. "Or they may never have gotten to a good solution." Tokens are the units of text that language models process, and heavy consumption carries real compute cost.

Shivani Poddar, CEO of personal AI agent startup Wajo AI, is also incorporating human workers — for phone calls and even for CAPTCHA tests, the bot-detection puzzles websites use to keep out automated traffic.

"I think AI plus humans, collaborative AI, is what you need to actually get the job done," Poddar said. She emphasized that people will stop using the technology if agents cannot complete the tasks they are given.

The economics cut both ways: humans fix failures quickly, but they also add labor cost to products marketed as automated.

What about the privacy problem?

Chatbot and agent interactions can be intensely personal, and users have good reason to be concerned about humans stepping in — especially if the company never disclosed that arrangement.

Meta has already faced this question internally. Company employees previously raised privacy concerns about an internal test that used human contractors to handle phone calls on behalf of Meta's Muse AI agent. A Meta spokesperson said in a statement at the time that this kind of testing is "core to the product development process."

Meta currently does not use humans to complete users' Muse requests, according to the company. Meta's Muse and OpenAI's Dots are among the agents that can accomplish a range of web-browsing tasks, but the technology remains early, and it frequently hits the roadblocks Joyce described.

The episode shows how disclosure, not capability alone, will shape user trust in agents that appear fully autonomous.

Who are the humans behind the agent?

The startups building human-in-the-loop systems describe screening processes modeled on gig-economy hiring, though with varying rigor.

Joyce said Duckbill runs "full background checking" on its 1099 contractors — independent gig workers classified for US tax purposes — and compared them to drivers for Uber and DoorDash: "people who live here in the United States and want a source of additional income." Duckbill also uses contract workers from businesses that typically provide contact-center labor to companies.

Mayer said Dazzle conducts background checks and screens for qualities someone might seek in a human personal assistant.

"We generally are screening for people who we think are extremely discreet and have good taste and good judgment, both in terms of how they interact with our users, but just the ability to kind of forget and compartmentalize that information afterwards," Mayer said. "I understand it's a point of privilege, but I've had personal assistants for some time, and I know how I want them, and I know how I guide them."

Notably, that discretion is a promise about contractor behavior, not a technical guarantee — no encryption or data-handounding system is described in either company's account.

Is the human layer temporary or the product?

The startups diverge on whether humans are scaffolding or substance. Mayer frames the current moment as a "transitionary period" until foundation models can perform more involved web tasks and phone calls as efficiently as people. Dazzle, in her telling, treats contractors as a bridge across that gap.

At the same time, Mayer does not present the human layer as an embarrassment. "We're not even trying to hide it, we think the human in the loop is a benefit," Mayer said.

Joyce's Duckbill has gone further, turning the fallback into infrastructure: its API invites other agent builders to make human labor a standard component of their products rather than a stopgap.

The unresolved question is what consumers believe they are buying. An agent sold as autonomous that routes sensitive calls to background-checked strangers is a different product than the one Meta and OpenAI pitch — and neither model capability nor disclosure standards currently settle it. Until foundation models can reliably make phone calls and finish multi-step tasks, the agent market's output will depend on how many people are on the other end of the software.