Researchers have developed an AI model called MarioVGG that attempts to recreate Super Mario Bros. This text-to-video AI model could potentially revolutionize video game programming, although it currently has limitations.
MarioVGG: A new approach to game development
The gaming industry is undoubtedly being transformed by AI. We’ve already seen AI-generated video game backgrounds, and some games, like Doom, are being recreated entirely without a game engine using image AI. This not only reimagines game development but also involves players differently, allowing them to modify games through prompts.
A similar approach has now been applied to Super Mario Bros., the classic NES game. Researchers from the tech company Virtuals Protocol are exploring whether an AI model can be used to “create and demonstrate a reliable and controllable video game generator.” They’ve recently published their initial findings in a paper.
Training the AI model
The researchers’ text-to-video AI model, named MarioVGG, was trained on over 737,000 frames to learn the basic gameplay dynamics of Super Mario Bros. To simplify the initial experiment, only two possible actions were used: “run right” and “run right and jump.”
According to Ars Technica, while the result is still full of bugs and far from smooth gameplay, “the results show how even a limited model can derive impressive physical and gameplay dynamics simply by studying some video and input data.” MarioVGG was able to “learn the physics of the game purely from video images in the training data, without explicit, hard-coded rules,” as stated in the research paper. This included behaviors like falling when Mario runs over an edge.
Interestingly, some of the AI model’s hallucinations, such as obstacles for Mario, integrated quite coherently into the game atmosphere. However, MarioVGG also hallucinated visual disturbances like color changes of the game character or passing through obstacles.
Current limitations and future potential
Although the examples published by Virtuals Protocol look good at first glance (albeit blurry due to the low resolution of 64×48 pixels), the AI is still far from becoming a “complete replacement for game development.”
The researchers are aware of this limitation. In their current experiment, using a single RTX 4090, they could only generate six frames in six seconds. They acknowledge this is “neither practical nor user-friendly,” but hope to achieve better results with longer training, larger datasets, and more computational resources.
There’s still significant room for improvement, especially when compared to what Google’s diffusion AI GameNGen can already achieve with Doom. Nevertheless, Ars Technica calls MarioVGG an “entertaining proof of concept, showing that even limited training data and algorithms can create some decent starting models for simple games.”
The extent to which AI can and will replace programmers in the future, who are currently already plagued by large waves of layoffs, remains to be seen. Ideally, AI will remain a tool that offers new possibilities rather than making human creativity obsolete.