What is Nvidia ACE for games?

What is Nvidia ACE and how is it breathing life into software and games? We’ve got everything you need to know about Nvidia’s new AI model announced at Computex in Taiwan.

Nvidia ACE, or Nvidia Avatar Cloud Engine uses generative AI enabling virtual characters to have what Nvidia calls “natural language interactions”. This means that in games, NPCs, or non-playable characters will be capable of natural language, and no longer be constrained by predictable and repetitive dialogue. This is going to change gaming forever. The news was announced by Nvidia’s CEO Jensen Huang during its 2023 Computex Keynote and marks a major turning point for the intersection between games and AI.

So, in the same way that Google’s PaLM 2 is making inroads with AI-generated content, ACE will allow you to have ‘natural’ conversations with an NPC. In a demo provided by Nvidia, we saw a gamer, known as Kai, walk into a ramen shop and have a voice conversation with Jin, the Ramen shop owner. After discussing the high-rate of crime in the city, Jin tells Kai where he might find the person responsible, and there begins Kai’s mission.

When we watched the demo, we found the conversation to be a little stiff and natural, though it’s clear that the potential is there. So, how exactly did Nvidia pull off this interaction?

NVIDIA ACE for Games AI foundation models

The overall effect of an NPC is capable of natural-language conversation is down to three models that cover speech, conversations and character animation. These models are available to be used individually by game developers, or in combination to create the most advanced result. If you’ve watched Nvidia’s demo, you will have seen all three models in action.

Nvidia NeMo

Nvidia NeMo was issued to build and deploy language models. In the case of Nvidia ACE for games, it’s used to add the backstory to a character. It’s also responsible for ensuring that NPCs can’t be vulnerable to “jailbreaking” where they could be tricked into have inappropriate conversations off-topic.

Nvidia Riva

Nvidia Riva is responsible for automatic speech recognition and text-to-speech, allowing you to have live speech conversations with the NPC. To go into a bit more depth, Riva is responsible for converting text-to-speech and speech-to-text so the gamer and NPC can ‘communicate’ with one another.

Nvidia Omniverse Audio2Face

This model enables the facial animation of a character to match the storyline of what’s happening. In layman’s terms, it basically ensures an NPC won’t be smiling or laughing when discussing something upsetting in the game, or being overly serious at more lighthearted moments.

Final word

We’ll probably be hearing a lot more from Nvidia ACE for games in the future. With Nvidia’s foundation models for ACE already in use, there are several game developers we’ve heard that early to take advantage of Nvidia’s generative AI technologies. The first we’ve heard is for S.T.A.L.K.E.R. 2: Heart of Chernobyl, developed by GSC GameWorld. They’re adopting Audio2Face which would certainly help with the unnerving atmosphere of the post-apocalyptic game. We’ve also heard Audio2Face being used by Fallen Leaf for Fort Solis, a thid-person sci-fi thriller that takes place on Mars.