We’re reimagining a 50-year-old interface - the mouse pointer - with AI. 🖱️ These experimental demos show how people can intuitively direct Gemini on their screens using motion, speech, and natural shorthand to get things done 🧵
Post
Post
Google DeepMind on X: "We’re reimagining a 50-year-old interface - the mouse pointer - with AI. 🖱️ These experimental demos show how people can intuitively direct Gemini on their screens using motion, speech, and natural shorthand to get things done 🧵"
With an AI-enabled pointer, help is always available where you’re working - without having to detour to additional apps. 📲 Point at a PDF and request bullet points for an email, hover over a table to ask for a pie chart, or highlight a recipe and simply say: "double these
Current models require precise instructions, but our AI-enabled pointer removes that burden. 💡 By "seeing" what’s under your cursor, it instantly understands the specific word, image, or code block you need help with.
In the real world, we don't tend to speak in long paragraphs; we point and say: "fix this" or "move that". 💬 By combining gestures with speech, it lets you use natural shorthand to complete tasks.
For decades, your mouse only tracked where you were pointing. AI helps it understand what you're pointing at. 💭 This means a photo of a scribbled note could turn into an interactive to-do list, or a paused video frame can become a restaurant booking link.
These capabilities are guiding how we think about the next generation of interfaces. As we continue exploring what an AI-enabled mouse pointer would unlock, try our experiments in
@GoogleAIStudio→
Why do you guys always use examples that target the general public but are genuinely useless in practice? Its always the same thing, shopping lists, calendar, travel plans... it is getting boring, man