OpenMobile-2 hits 59.6 on SPA-Bench, topping open smartphone agents

An open-source AI phone agent scored 59.6 points on SPA-Bench on real Android devices — the highest result among open models, according to its creators. The team behind OpenMobile-2 says the project proves open smartphone agents can now rival proprietary systems, and they're releasing everything needed to check that claim: two models, the full training environment, the data, and reproduction instructions.

The work comes from researchers at Nanjing University, the AI company SenseTime, and several other universities. Their core finding challenges a common assumption in agent training: more data alone doesn't make a smartphone agent better. When the team held the training data constant and only increased the number of apps the agent practiced on, the success rate on unseen applications rose from 60.7 to 67.5 percent.

That matters because the field has a concentration problem. AI agents that operate apps on their own learn mainly through practice, but the data they need is mostly locked up at big companies, the researchers write. Open models lag behind as a result. OpenMobile-2 is their attempt to close that gap.

What exactly did the researchers release?

The project ships two agent models: one built on Qwen3.5 with 9 billion parameters, and a larger one built on Qwen3.6 with 27 billion parameters. Both come with the full training environment, the data, and instructions to reproduce everything.

The authors say their model ranks near the top among current smartphone agents. That claim rests on benchmark numbers across three test suites, including one run on physical Android phones rather than emulators.

Why rebuild 110 apps instead of using real ones?

Training on real apps is painful. Logins, real purchases, and no way to reset an app's state make the process expensive and hard to repeat. So the researchers rebuilt popular apps as simulations.

The rebuild covers 35 apps popular mainly in China, including Taobao, JD, DiDi, and Meituan, plus globally used services like Slack and Google Drive. Combined with existing simulations and apps running in an Android emulator, the practice ground spans more than 110 applications.

Human reviewers compared the rebuilt apps to the originals and kept refining until workflows and interfaces matched. Orders, budgets, and travel plans persist across sessions, and data can pass between apps via the clipboard. Every simulation resets to its starting state automatically.

The agent only sees screenshots and has no access to internal app data, mimicking how a person would actually use a phone. From this setup, the team generated about 11,600 example workflows and more than 2,000 practice tasks with automated success checks.

There's an irony worth noting: Google's Gemini 3.1 Pro served as the teacher model that produced the examples. OpenMobile-2 is effectively learning from a closed system.

How much did performance improve?

Both models show clear gains on established benchmarks.

• On AndroidWorld, the larger model jumps from 67.1 to 79.9 points.
• The smaller model goes from 54.3 to 74 on the same test.
• The predecessor OpenMobile-1 scored 64.7 on AndroidWorld.
• On MobileWorld, a harder benchmark covering longer tasks that span multiple apps, the larger model improves from 35.9 to 50.4 points.
• On SPA-Bench, which runs on real Android devices, the larger model climbs from 31.9 to 59.6 points.

The researchers say the SPA-Bench score is the best among open models. The near-doubling on real phones matters most, because simulated benchmarks often overstate how an agent behaves outside the lab.

The diversity finding held consistently. With the same data budget, app variety beat extra examples drawn from a handful of apps. For anyone training phone agents, that's the practical takeaway: spend your budget on breadth, not repetition.

Did reinforcement learning help?

Only marginally, and mainly in environments that resembled the training setup. Reinforcement learning on top of the base training brought limited gains.

On AndroidWorld, the larger model actually scored slightly worse after that phase, landing at 79.9 instead of 80.8 points. The result suggests that once the environment and data are right, additional RL offers little for this class of agent.

What do direct commands change?

An experimental part of the study goes beyond tapping and swiping. More than 50 of the simulated apps offer direct commands like "add to cart," letting the agent skip navigating through menus. The agent freely switches between direct commands and manual navigation depending on what's available.

On its own, this doesn't help much. The models tend to ignore the shortcuts without targeted training. Once trained on them, the agents use the shortcuts well, solving up to 10.3 percentage points more tasks while needing about five fewer steps. The researchers say this ability also transfers to tools the models never encountered during training.

During the later reinforcement learning phase, tool use got a bonus factor whenever a run included at least one direct command — but the reward still depended on whether the task was actually completed. Penalties kicked in when the agent reported a success that wasn't real. The researchers call the whole approach experimental, and these interfaces only exist in the simulated apps for now.

How reliable are the test results?

Some results come from a custom benchmark the team built with 215 tasks, nearly half of which span multiple apps. Task completion isn't judged by a language model. Instead, a program compares the app's stored data before and after each run.

The authors also checked for overlap between training and test tasks and found only minor similarities. That guards against the most common way agent benchmarks get inflated.

Where does this fit in the broader field?

The study lands in an active debate about how agents should interact with software. Microsoft Research had already classified API agents as more reliable and GUI agents as more flexible in a 2025 study, pointing to mobile apps as a typical case where external interfaces don't exist. OpenMobile-2's direct-command experiments sit exactly on that fault line between the two approaches.

Oppo takes a different route with its open agent X-OmniClaw, which runs on real devices and replays previously shown paths in apps like Taobao or Meituan via deeplinks instead of clicking through menus again. Among closed models, Google has built smartphone control directly into Gemini 3.5 Flash.

The release gives academic labs and smaller companies a complete, reproducible stack for training phone agents — an environment, data, models, and evaluation code in one package. If the diversity finding holds up under independent testing, the next round of open agent research will likely compete on environment design rather than raw data collection.