We introduce Lumine, a generalist agent trained within Genshin Impact that can perceive, reason, and act in real time, completing hours-long missions across 3D open-world environments.
Mondstadt Region: Lumine successfully completes a full five-hour, three-act main storyline playthrough of Mondstadt region. Even though its reasoning ability is trained only on the content of the first act, Lumine autonomously carries out all story progressions.
Liyue Region: In the completely out-of-distribution region of Liyue, Lumine demonstrates impressive generalization, successfully reaching and completing the main storyline in Liyue Harbor, and even locating and visiting the Adeptus dwelling deep within the mountains.
Beyond Genshin: Lumine exhibits exceptional cross-game generalization ability. Without any additional fine-tuning and modification, Lumine directly completes the entire chapter of Herta Space Station in Honkai: Star Rail within 7 hours; similarly, it successfully finishes approximately 100 minutes of main storyline content in Wuthering Waves.
Genshin Impact
Mondstadt Act I (56min)
Genshin Impact
Mondstadt Act II & III (4h43min)
Genshin Impact
Liyue Act I (7h16min)
Honkai: Star Rail
Herta Space Station Chapter (7h18min)
Wuthering Waves
Wuthering Waves Gameplay (1h46min)
An Efficient and Scalable Recipe for Building General-Purpose Agents
Lumine adopts a human-like interaction paradigm that unifies perception, reasoning, and action in an end-to-end manner, powered by a vision-language model. It processes raw pixels at 5 Hz to produce precise 30 Hz keyboardβmouse actions and adaptively invokes reasoning only when necessary.
Lumine makes full use of the collected raw gameplay data, following a curriculum that builds skills incrementally:
- 1731 hours of human gameplay for pre-training to master action primitives;
- 200 hours of instruction following data to ground control in language;
- 15 hours of reasoning data to enable adaptive thinking.
The resulting model can not only autonomously complete hours-long missions but also follow diverse instructions to accomplish a broad spectrum of tasks.
Combat
Benefiting from large-scale pretraining, Lumine has mastered the essential combat skills, dynamically tracking enemies, accurately striking distant targets with a bow, seamlessly switching characters to perform combo attacks, and efficiently locating and opening treasure chests unlocked after combat.
Defeat the enemies ahead and collect the chest
Complete the Domain
Complete the Daily Commission: Defeat all enemies
Boss Fight
Besides regular combat, Lumine also shows a strong understanding of boss mechanics and the ability to respond effectively. It can skillfully evade powerful attacks and employ appropriate strategies to defeat them.
Defeat the Electro Hypostasis
Defeat the Electro Hypostasis
Defeat the Electro Hypostasis
Defeat Stormterror
Defeat Stormterror
Defeat the Anemo Hypostasis
Puzzle
Lumine can handle various challenges and puzzles in the game, which typically require a thorough understanding of game mechanics, strong spatial reasoning skills, and precise low-level control.
Fly along the Wind Current to collect the Anemoculus
After defeating the floating Anemo Slime, open the chest
Break the boulder ahead and open the chest
Open the chest wrapped in thorns ahead
Activate the Elemental Monument using the corresponding element
Complete the Time Trial Challenge ahead: Open the chest within the time limit
NPC Interaction
Lumine exhibits reliable instruction-following ability, consistently interacting with designated NPCs within crowds, laying a solid foundation for accomplishing long-term missions.
Talk to NPC Grace
Talk to NPC Monroe
Talk to NPC Sayid
GUI Manipulation
Beyond open-world exploration, Lumine can also perform efficient GUI operations through human-like relative mouse movements, achieving a unified interaction between 2D interfaces and the 3D world, a capability that is crucial for generalist agents.
Cook Sweet Madame
Teleport using a Teleport Waypoint
Change the character's weapon
In-Context Learning
Meanwhile, Lumine has demonstrated strong in-context learning abilities. When provided with prior task information or more detailed decomposition steps within the instruction, Lumine can successfully complete a range of tasks that it was previously unable to perform.
Climb the stone pillar on the right and, once you reach the top, collect the blue Anemoculus floating in the air on the left
Switch to Kaeya, continuously use his Elemental Skill (E Skill) to freeze the water surface, and collect the Anemoculus floating ahead
Collect the Wind Anemograna to activate a Wind Current, then enter the Wind Barrier to open the chest