5/23/2018

Smarter NPCs with Natural Language Processing

What happened in Pokemon?

I remember Pokemon on GameBoy. An excellent game with many characters. Like all games (till now - more on that soon) the conversations were somewhat forced. The most creative moment was choosing your name in the beginning. Almost all my interactions with non-player chracters (NPC) involved multiple choice answers at best, and pre-canned conversations at worst.

Screenshot of a simple Yes or No answer to a question in Pokemon for Gameboy

Games have Evolved

A decade after Pokemon was first released on Gameboy, Fallout 3 was one of the most popular games in the world. For all its quality graphics and cool story, the NPC interations remain remarkably unchanged.

Screenshot from Fallout 3 showing multiple choice answers

We have the technology. We can do better

What if you could actually talk to NPCs? Well, now you can!

What is Natural Language Understanding?

the application of computational techniques to the analysis and synthesis of natural language and speech.

NLU, sometimes called Natural Language Processing (NLP), are methods for processing and using natural human speech or text in classical algorithms. Recent technological breakthroughs in Deep Learning have turbocharged NLU, and now you can use it in your own game or application.

Sounds hard...

Not really, check this out.

LUIS - the Language Understanding and Intelligence Service from Microsoft Azure - is super simple to use. You can follow this tutorial and get started in just 10 minutes!

Aside: if you're in Australia, make sure you head to the au.luis.ai domain.

What's LUIS - the quick version

LUIS is a web service, invoked via a REST endpoint, that converts sentences (called 'utterances') like this:

Order me 2 pizzas

into data structures like this:

{
   "query": "Order me 2 pizzas",
   "topScoringIntent": {
       "intent": "FoodOrder",
       "score": 0.9999981
   },
   "intents": [
       {
           "intent": "FoodOrder",
           "score": 0.9999981
       },
       {
           "intent": "None",
           "score": 0.0604290478
       },
       {
           "intent": "Reminder",
           "score": 0.00177723425
       }
   ],
   "entities": [
       {
           "entity": "2",
           "type": "builtin.number",
           "startIndex": 9,
           "endIndex": 9,
           "resolution": {
               "value": "2"
           }
       },
       {
           "entity": "pizzas",
           "type": "FoodType",
           "startIndex": 11,
           "endIndex": 16,
           "score": 0.8928091
       }
   ]
}

If you're making a game or an app, and you can make web requests, then you can have NPCs that respond to natural language!

Now for the good part

Modern games are much more sophisticated than Gameboy Pokemon and Fallout 3. Cutting edge games are now in virtual reality (VR) on platforms like Vive, Oculus, and Windows Mixed Reality. But we're starting to use VR for other interesting applications.

Experience a New Language in VR with V-KAIWA

V-KAIWA are building an amazing tool for students to experience language learning in VR. Beyond building stylised worlds and engaging story lines, V-KAIWA are building customisable conversational NPCs. Let's break that down.

Conversational NPCs

As the name suggests, a conversational NPC is a character with which you can converse. OK, so they aren't going to pass a Turing test, but they can respond to natural language. Ask the hot dog vendor on the street for a hot dog, and you'll get one! Ask a stranger for directions, and they'll help! But if you ask a stranger for the meaning of life - of course you'll get a silly answer (actually that's kind of realistic).

Customisable NPCs

V-KAIWA let's language teachers customise the way their students can converse with NPCs. How does this work in practice? Imagine a language class on recognising descriptions of people and objects. The teacher can describe a person (a woman in a jacket) and name an item (a newspaper) and the students must complete a scenario in the virtual world. The teacher can build precisely that conversation scenario in V-KAIWA's designer tool!

Creating an NPC interaction definition in the V-KAIWA interaction designer

When the student find the newspaper, he can talk with the woman.

Student: Here's the newspaper you asked for. Woman in Jacket: Thanks, I was looking for one of those.

It's almost too easy.

Sugar on top

By using speech-to-text and speech synthesis tools, V-KAIWA make every NPC interaction aural. You can speak directly to NPCs, and they speak back. LUIS means you can speak naturally. Gaze means you speak to the NPC you're looking at.

System design of V-KAIWAs customisable conversational NPCs

I want that!

Well you can have it! There's a great blog written by some of my collegues at Road to MR describing how to use LUIS in your Unity project. They've created a great Unity Package you can use to get started quickly.

What about the designer? It's also open source and available on github

CSE Hacks

This work was a collaboration between Microsoft and V-KAIWA. Big thanks to them for letting me work on their exciting product.