Elon Musk’s xAI joins race to build ‘world models’ to power video games

Stay informed with free updates

Simply sign up artificial intelligence myFT Digest – delivered straight to your inbox.

Elon Musk’s xAI company seeks to build so-called universal models, joining competitors such as Meta and Google in the race to develop artificial intelligence systems that can navigate and model physical environments.

The San Francisco-based startup hired specialists from Nvidia over the summer to work on its next-generation AI models, which train on video and data from robots to understand the real world.

Universal models can push AI capabilities beyond the large, text-trained language models that underpin popular AI tools like ChatGPT and XAI’s Grok.

Two people familiar with the plans said that the company is building global models with the aim of applying them in games, where they can be used to create interactive 3D environments. One person added that it could be applied to artificial intelligence systems for robots.

xAI has hired Zeeshan Patel and Ethan He, two AI researchers from Nvidia with expertise in global models. Nvidia has pioneered the development of this technology with its Omniverse platform, which creates and runs simulations.

Some technology groups have high expectations of global models, which could unleash uses of AI beyond software and computers in physical products such as humanoid robots. Last month, Nvidia told the Financial Times that the potential market for global supermodels could be roughly the size of the current global economy.

Musk said in a post on the X website that xAI will launch “a wonderful game produced by artificial intelligence before the end of next year,” confirming the goal that the billionaire set last year.

On Tuesday, xAI launched its latest image and video generation model, which it said has “huge upgrades” and is free to use.

Existing video generation models, such as Sora from OpenAI, generate image frames for videos by predicting patterns learned from training data.

Universal models will be a major advance because they will have a causal understanding of physics and how things interact in different environments in real time.

The company is advertising for image and video generation technical staff to join its “holistic team,” which “creates magical AI experiences that go beyond text, enabling content to be understood and generated across different modalities, including images, video and audio.”

Salaries for these jobs range from $180,000 to $440,000. It also has an open position for a “video game tutor,” who will train Grok in video game production and enable “users to explore AI-assisted game design,” for $45 to $100 per hour.

Musk follows other leading AI labs, such as Google and Meta, which are also working on these systems.

However, global modeling remains a major technical challenge. Finding enough data to simulate the real world and training such models has proven difficult and expensive.

Michael Doss, Head of Publishing, Larian Studios, Who develops video games Baldur’s Gate 3said on X this week that AI has been unable to solve the gaming industry’s “big problem,” which is “driving [and] vision”.

He added that the industry does not need “more mathematically produced, psychologically trained play loops.” [but] Rather, they are more expressions of the worlds that people deal with or want to deal with.

xAI, Patel and He did not respond to requests for comment.

Additional reporting from Hannah Murphy in San Francisco