Google Deepmind is using Gemini to train agents inside Goat Simulator 3

The researchers claim that SIMA 2 can perform a range of more complex tasks within virtual worlds, learn how to solve some challenges itself, and chat with its users. He can also improve himself by tackling difficult tasks multiple times and learning through trial and error.

“Gaming has been a driving force behind customer research for a long time,” Joe Marino, a research scientist at Google DeepMind, said in a press conference this week. He noted that even a simple action in the game, such as lighting a lantern, can involve multiple steps: “It’s a really complex set of tasks that you need to solve to progress.”

The ultimate goal is to develop next-generation agents that are able to follow instructions and perform open-ended tasks within more complex environments than a web browser. In the long term, Google DeepMind wants to use such agents to drive robots in the real world. Marino claimed that the skills learned by SIMA 2, such as navigating the environment, using tools, and collaborating with humans to solve problems, are essential building blocks for future robotic companions.

Unlike previous work on game-playing agents like AlphaZero, which beat Go grandmaster in 2016, or AlphaStar, which beat 99.8% of rated human competition players in the video game StarCraft 2 in 2019, the idea behind SIMA is to train an agent to play an open-ended game without pre-defined goals. Instead, the agent learns to carry out instructions given to it by people.

Humans control SIMA 2 via text chat, by speaking to it out loud, or by drawing on the game screen. The agent takes the pixels of the video game frame by frame and figures out what actions it needs to take to carry out its tasks.

Like its predecessor, SIMA 2 was trained on footage of humans playing eight commercial video games, including No Man’s Sky and Goat Simulator 3, as well as three virtual worlds created by the company. Learn the agent to match keyboard and mouse inputs with actions.

Researchers claim that SIMA 2, associated with Gemini, is much better at following instructions (asking questions and providing updates as they go) and figuring out how to perform some of the more complex tasks.

Google DeepMind tested the agent inside environments it had never seen before. In one set of experiments, the researchers asked Genie 3, the latest version of the company’s global model, to produce environments from scratch and drop SIMA 2 into them. They found that the client was able to navigate and carry out instructions there.