Google Wants to Replace the Prompt With an AI-Powered Mouse Pointer

Google is rebuilding the mouse pointer — a UI element it says has "barely evolved in more than half a century" — into an AI-powered instrument that understands what it is pointing at and why it matters to the user. The experimental system, powered by Gemini, is live today in two products: users can point at parts of a webpage to query Gemini in Chrome, and a "Magic Pointer" is coming soon to Google's new Googlebook laptop experience. A public demo is available in Google AI Studio, where users can edit an image or find places on a map just by pointing and speaking.

The core problem Google is attacking is architectural. Because a typical AI tool lives in its own window, users must drag their context into it — copy text into a chatbot, upload a screenshot, write a detailed prompt. "We want the opposite: intuitive AI that meets users across all the tools they use, without interrupting their flow," the company writes.

Four interaction principles

Google frames the work around four principles it says shift the burden of conveying context and intent from the user to the computer.

Maintain the flow. AI capabilities should work across all apps rather than force users into what Google calls "AI detours" between them. In the prototype, a user could point at a PDF and request a bullet-point summary to paste directly into an email, hover over a table of statistics and ask for a pie chart, or highlight a recipe and request all the ingredients doubled.

Show and tell. Current models demand precise, text-heavy instructions. The AI-enabled pointer instead captures the visual and semantic context around the cursor, letting the computer "see" what matters. "Just point, and the AI knows exactly which word, paragraph, part of an image, or code block the user needs help with," Google says.

Embrace the power of "This" and "That." Humans rarely speak in long, detailed paragraphs to each other; they say "Fix this" or "Move that here" and rely on gestures and shared context to fill the gaps. A system combining context, pointing and speech would let users issue complex requests in natural shorthand, with no prompt engineering.

Turn pixels into actionable entities. For decades, computers only tracked where we point. AI can now understand what we point at, converting pixels into structured entities — places, dates, objects. A photo of a scribbled note becomes an interactive to-do list; a paused frame in a travel video becomes a booking link for the restaurant on screen.

Why it matters

The stakes are control of the interaction layer itself. Since ChatGPT's launch, the dominant AI interface has been the prompt box — a separate destination that pulls users out of their applications. Google's pointer concept inverts that model, embedding Gemini directly into the OS and browser surfaces where people already work. For Google, which has raced to integrate Gemini across Search, Android and Workspace, the pointer is another route to make its models the default layer between users and their screens, ahead of rivals pursuing similar ambient-assistant strategies.

Shipping in Chrome and Googlebook

The principles are already moving into products. Starting today, users can point with their cursor to ask Gemini in Chrome about a specific part of a webpage — select a few products on a page and ask Gemini to compare them, or point to where they want to visualize a new couch in their living room. Magic Pointer will follow in Googlebook, described by Google as letting users "harness Gemini at their fingertips." Google says it will continue testing future concepts across its platforms, including Google Labs' Disco.

"Building technology that adapts to human behavior — rather than forcing users to adapt to it — enables a future where collaborating with AI feels truly intuitive, fluid and seamless," the company writes, adding that these "human-first concepts are being woven into products we use every day."

The open question is precision and trust: an AI that infers intent from a hovered pixel will sometimes guess wrong, and Google has not detailed accuracy benchmarks or privacy handling for the on-screen context the system reads. How quickly users adopt pointing-plus-speech over typed prompts will determine whether the cursor becomes AI's next major interface — or stays a cursor.