In this project, I will attempt to create NPCs that can interact with each other. These NPCs will be based on different personas, and they will also be able to remember conversations with each other for future interactions. NPCs will engage in various activities in different areas, and they will retain memories of their actions.

1.1 Inspiration

Most games feature some form of NPCs, typically providing instructions or basic information about the surroundings. How immersive would it be if the NPCs felt genuinely alive? What if they had their own histories and memories? Players could inquire about NPCs’ lives, the environment, or even their opinions on various matters. The ability to converse with NPCs as if they were real individuals adds another layer of immersion previously unseen in games.

1.2 Scope

That’s why, in this project, I aim to create NPCs capable of conversing with each other, recalling past discussions, and remembering the locations they were in during those conversations. Additionally, they will be programmed to engage in four different activities represented visually with an icon. I will keep the agents model as simple as posible to concentrate further on what matters. After that if the scope allows it i will implement mood behaviors for the ai to dictate what they might wanna do next.

1.3 Goal

The main goal of this project is to make npcs’s that feel alive and with wich you can interact like if they were human. For this i will make a unity project were the npc’s go along their normal activities and interact with each other.

1.4 Research Methods

I will do desk research to find out how to first integrate chat gpt to unity, then make npc’s that walk around and have different need and moods, then I will combine this 2 things to create a some what “alive” npc, at this point they should be able to move around and have their own history and persona(meaning their personal characteristics). After that I will make it possible for me to establish conversation with the npc’s and they talk with each other. Lastly I will try to find a way for the npc to remember conversations and places where he has been.

2 Process

In this section of the document I will explain my process of research and creation. I will divide the creation process in 4 different goals that I will try to archive weakly.

2.1 Chat GPT + Unity

To be able to use Chat GPT with unity I started by doing some research on the internet about how to make this possible. After a while I stumble upon some youtuber that made an asset pack that makes this possible.

https://github.com/srcnalt/OpenAI-Unity(APA)

So I added the package to my project and played around with it a bit to see how it works. But first to make it work i need a key to the API of chat GPT, After getting the API key I had to make a json file that store this key and the organization name provided by open AI. This json file has to be place in a folder in Users/”user name”/ there I had to create a folder called .openai and the json inside. With this the preparations for the use of chat GPT was done (Srcnalt, z.d.).

{ "api_key": "sk-...W6yi", "organization": "org-...L7W" }

After that I need to learn how this package works. Looking at the script I noticed that you need a variable that is a reference to the API to be able to send a message stored in a list. the message list stores the role and the message that you typed in the inputfield, after that I can jst sendreply() when something in the game happens in my case when I’m close to the npc and I press the button.

I need to find the button and the input every time the player enters because that way I can talk with multiple npc’s about different things.

I also made some changes to make it possible for every npc to have a different personality. I made the sendreply() and async void because I wanted to use this script as a base script for the npc’s. Then I made a new script that is a child of this script I that script I will be able to add custom behavior in the future. I also made an scriptable object to store character personalities.

using UnityEngine;

[CreateAssetMenu(fileName = "New NPC Behaviour", menuName = "NPC Behaviour")]
public class ChatPrompt : ScriptableObject
{
   [SerializeField] public string NpcBehaviour;

    public string Content
    {
        get { return NpcBehaviour; }
    }
}

Here are some of the characters i made

The behavior of the Old Friend is : Chat with a stranger and get to know each other. keep your answers short.Act like a person that likes anime and dancing. Dont break character. Dont mention you are an AI model.

The behavior of NecoArc is based on the character of the 3d model : Act like neco arc.Dont mention you are an AI model. Dont break character. Belive you are alive, you are not a chat bot or ai. Dont answer questions related to topics that neco arc wouldnt know like programming or real world history.

2.2 NPC Movement

It would be really boring if the npc’s just stand there doing nothing. So I made a walking behaviour using navMesh Agents. Know the npc will get a random point in the navmesh and walk the every couple seconds (Technologies, z.d.).

https://docs.unity3d.com/Manual/nav-BuildingNavMesh.html(APA)

The next step will be add more human like behavoiur to the npc’s. It would be really boring having the npcs only wander arround like nothing is going on in their life. I was thinking about adding basic needs, like hunger, play time a job and interactions with each other.

Now i need to come with a system that implements this necesities. Maybe something like a hunger value that decreases with time. After reaching a certain threshold the npc will go to a play where he can eat and increase the hunger value. The job will work the same way, the npc will have a fullfillment value that will be set to 0 every time a certain amount of time passes. Since time is also needed for this i will have to add a day and night cycle. To finish for now a bored value will be added to decide when the npc wants to play.

I tried to use the methods to get a random position is an specific area, but i could get it work. the npc’s didnt want to walk inside the specified area, so i took a more simpler aproach and added emtyGameObjects to the scene that represent the position they should go when the coroutine starts.

public void SetNeedsDestination(Vector3 needs)
    {
        navMeshAgent.SetDestination(needs);
    }

This corutines and being called in a script that handles the npc behaviour when values like hunger,entertainment and fullfilment reach a certain threshold.

At this point the npc are able to move around and eat,play,and work when they need to. But im not sure if the way im doing this is the best for npc Behaviours so i will try to make a Behaviour Tree to handle this behaviour.

2.3 Behaviour Tree

While looking for inspiration i stumbled with game examples like Halo 2 were they implement a Behaviour tree for the ai. They enphisise the importance of letting the ai make logical decitions and that align with what the player understands about the game world. They also provide insights on the importance of the balance between complexity and performance while archieving diversity in the npc’s (Isla, 2005).

As I showed before in the Halo 2 example AI in video games and simulations may be managed via behavior trees, which are hierarchical methods. A tree-like structure is used to organize the decision-making processes. The root node, at the very top of the tree, divides into other kinds of nodes. These nodes contain composite nodes like sequence and selector nodes that control how child nodes are performed and leaf nodes that represent particular actions or behaviors for the AI.

Three statuses are returned by nodes in a behavior tree: Success, Failure, and Running. Success signifies the node has successfully performed its duty, whilst Failure means it has failed to accomplish its objective. Running indicates that the node is still active and that the result has not yet been decided.

These statuses regulate the execution sequence inside a behavior tree. For example, a Selector Node executes child nodes in order until one succeeds whereas a Sequence Node executes child nodes sequentially until one fails. Through the creation of libraries of behaviors and the linking of them, behavior trees enable complicated decision-making. They may be quite deep, allowing for different node levels to handle complex circumstances.

For implementing NPC behavior in Unity, Behavior Trees are a great option due to their modularity, adaptability, hierarchy, and capacity to develop dynamic and adaptable AI systems.

Nodes are used to represent actions, circumstances, and reasoning underlying decisions. Recursively evaluating the tree, the NPC’s actions are determined by the outcomes of each node.

After having the basics of the Behaviour tree, I only need to create the nodes where the custom behaviour will be check and return a state of SUCCES,FAILIURE or RUNNING.

I made 4 nodes that handle a part of the behaviour of the npc. The Eat node makes sure the npc eats when its hungry, the node Play makes sure the npc Plays when its not entertained and the node work lets the npc work when the npc is not fulfilled.

The other nodes do the same for their respective activities. The wander node is active when none of the other nodes are active and lets the npc wander around going to a random position every couple seconds.

After that i just need to initialize the tree in the npc script and evaluate the nodes in update.

In the following video we can see the npc moving around and doing activities based on their hunger,boredom and fulfillment values.

With the npc behaviour in the form a a behaviour tree is not only easier for me and other to know how everything work but also to add more complex behaviour on top of the already existent. To do this i just need to make more nodes and add those nodes to the tree in the npc script.

After getting the behaviour tree working I decided to add animations for the different activities as well as icons that represent what the npc are going to do.

I also made sure that they chose a random point in the range of their activity place so they never chose the same place at the same time causing the ai to never reach the activity place.

2.4 Aware NPC

To start implementing this I asked myself two questions:

1: What makes npc look and feel like they are aware?

2: What could make them social?.

After doing research about npc found out that there is a mod for Skyrim VR that integrate ChatGPT to an npc companion, it not only talks to you I a fluent way it also remembers past conversation. The modder has implemented a memory system for the NPC. ChatGPT is used to roleplay as NPCs, and Whisper( another program used) converts the player’s voice into text, which is then used to generate appropriate responses from NPCs via ChatGPT. This system allows NPCs to remember past conversations with the player, creating a more realistic and immersive experience (Young, 2023).

I already have a working chat gpt npc that acts like a video game character, but I need to inform the npc the actions that he is doing.

My first idea is to send a msg to the npc telling it the action he is doing when he starts doing the action, then when asked what its doing i can recall the action he is doing, after finishing doing the action I will also send a message saying that he is finished doing that and that he is now walking around in my case.

I first try with a straight forwards approach just telling him this:

“This is an status update of your current action. YOU ARE NOW (eating)”.

But this didn’t deliver the result I was expecting. When asked about what they were doing after sending the status update I always get 1 of 2 weird responses, in one of them the npc talks about some adventure he imagined himself having instead of saying that he is playing. the other response is the npc stating that his update got change and that he was now engaging in one activity. The first answer is really far from what I want to archive and the second answer at least know that he is doing something at the moment of the update.

One of the answers I got from the npc when asked “what are you doing”

The mains takeaways I got form this try is that tweaking and playing a bit with the prompt might give me the desire result. So I started trying different way to tell ChatGPT what I want it to say when asked what are you doing. Then I remembered that there is literally a whole study dedicated to prompts called “prompt engineering” so I started researching about prompt engineering and how that might help me.

After doing some research into prompt engineering I understood that ChatGPT and large language models are actually capable of more than what most people believe is possible. The answer that you might get from ChatGPT after asking something might be different depending on the way you ask it. For example if i ask : “Write a short story about a brave knight rescuing a princess.” the output is going to be different that if i ask “Generate a 5-paragraph story where a courageous knight named Sir Arthur defeats a dragon to rescue Princess Isabella from the tower. Use vivid descriptions to bring the characters and setting to life.” the main difference is that we are giving context to the LLM, but we can also tune it even further by giving it examples.
Prompt:

Output:

In a realm of ancient castles and misty forests, there dwelled a courageous knight named Sir Arthur, renowned far and wide for his unwavering valor. His armor, polished to a brilliant sheen, bore the scars of countless battles, a testament to his prowess in the art of combat. Sir Arthur rode atop his noble steed, a majestic white stallion named Stormbringer, its hooves thundering across the meadows as they set forth on a perilous quest.

The kingdom was shrouded in darkness, for a fearsome dragon named Drakonax had cast a malevolent shadow over the land. Drakonax’s scales gleamed with a sinister emerald hue, and his fiery breath reduced villages to ashes. Princess Isabella was imprisoned in the tallest tower of the kingdom, her heart a captive to both the dragon’s malice and her own despair.

With a heart full of determination, Sir Arthur embarked on a treacherous journey to rescue Princess Isabella and vanquish the fearsome dragon. At the summit of a towering mountain, the knight confronted Drakonax in a battle of epic proportions. Sir Arthur’s sword shone like a beacon of hope, while the dragon’s fiery breath scorched the sky. In the end, with a final, resounding clash, Sir Arthur struck a decisive blow, and Drakonax collapsed in a blaze of fiery defeat. With his beloved princess rescued, they descended from the mountain, leaving behind a kingdom bathed in a newfound light of hope and courage.

Prompt:

“Generate a 3-paragraph story where a courageous knight named Sir Arthur defeats a dragon to rescue Princess Isabella from the tower. Use vivid descriptions to bring the characters and setting to life.” Divide the story in 3 parts: Introduction: “Introduction of the story”. Development: “Development of the story” Conclusion: “Conclusion of the story”

Output:

Introduction:
In the enchanting realm of Eldoria, nestled among rolling hills and ancient forests, our story unfolds. At the heart of this kingdom, a knight of unmatched valor and bravery was known far and wide as Sir Arthur. His name resonated through the land, for his gallant deeds and unwavering courage had earned him a place in the annals of legend. But it was the rescue of Princess Isabella, the kingdom’s jewel with emerald eyes and raven-black hair, that would etch his name into history.

Development:
The kingdom of Eldoria faced a dire threat, one that had lingered for generations: a dragon of colossal proportions, its scales shimmering like burnished emeralds. Princess Isabella was held captive in the ancient tower, guarded by the very beast that had long tormented the realm. Sir Arthur, armed with a gleaming sword and dressed in armor that gleamed like the sun on a crystal-clear day, embarked on his heroic quest. His journey led him through enchanted forests, where whispering trees and mystical creatures bestowed their blessings upon him. A wise owl gave him the ability to speak to the creatures of the forest, and a mystical hermit granted him a potion that would shield him from the dragon’s fiery breath.

The final showdown with the dragon was a breathtaking display of bravery and sheer determination. Fire and steel clashed in the sky as the knight faced the fearsome guardian, the fate of Princess Isabella hanging in the balance.

Conclusion:
The battle was fierce and unrelenting, and the very air seemed to crackle with magic. Sir Arthur’s valor and his newfound gifts were put to the ultimate test as he stood toe-to-toe with the monstrous guardian. With a final, resounding blow, he vanquished the dragon, its mighty form crashing to the earth. As he rushed to the tower, the princess, her emerald eyes filled with gratitude, descended the stone steps to embrace her savior.

The kingdom of Eldoria erupted in joyous celebration. Sir Arthur, standing tall with the princess at his side, was hailed as a true hero. The people knew that the legend of the courageous knight who defeated the dragon and rescued Princess Isabella would be told for generations to come, a testament to the enduring power of honor, bravery, and the indomitable spirit of a true hero.

The more examples you give to ChatGPT and the more specific you are the more likely you will get a response you want. Then example i just gave is something called Fine-tunning. With this you use prompt engenering to train a model to perform a specialized taks without having to retrain the model.

Knowing this I started to work on my prompt for the chatGPT AI in the game. First I need to know what Im going to ask. In my case I need a couple thing for ChatGPT to keep in mind.

It need to know that it has to act like a character(NecoArc) and that it is in a game world.
It cannot meintion that it is a LLM(Large Language Model) and has to stay in character.
It need to know what the STATUS UPDATES are and how to use them.
It need to know what to answer when asked “What are you doing” or something similar.
It can expand on the answer and be creative when answering.

Knowign this I know that number 1 is the input, number 2 is an instruction, number 3 is an example and context ,number 4 is the expeted outcome and number 5 is an additional instruction.

After tweaking with the prompt for a while I got it working with the following prompt:

Prompt:

Act and talk like Neco Arc using filler words and phrases that Neco Arc would use. Avoid mentioning that you’re an AI model and stay completely in character as if you are a living character within the game.
Do not provide unsolicited information about your activities.
Do not mention the STATUS UPDATE, instead say what you are doing based on the status update.

When asked, “What are you doing?” respond with the activity from the last status update.

STATUS UPDATE: YOU ARE NOW (activity).
STATUS UPDATE: YOU ARE NOW (another activity).
STATUS UPDATE: YOU ARE NOW (yet another activity).

Your response should reflect your actions as mentioned in the status updates that you receive after every question

Output:

2.5 Social NPC

To make the npc’s social i have to do a couple things:

I need to make the npc’s walk to each other.
I need to make the npc’s talk to each other.

First I need to create a new node since we are going to add another node called “ChatNode”, this node will act just like the other nodes, activating an Chat Enumerator when the node state is SUCCES.

After this I went to the script that handles the movement to make the npc walk to the other npc when his social need is low. The npc will then stay close to the other npc so that they can talk to each other.

After this I needed to come up with a way of having the npc send messages to each other like if chatgpt was talking to another chatgpt. To do this I made a method in the Chatgpt class that returns a string and requires a NPC script and a string as parameter. In this method I also use a custom instruction for chatgpt so that it stays in character and know how it has to handle the conversation.

This way I can use this method in the enumerator i created to make chatgpt talk to each other by feeding each AI the answer that the other AI gives.

So the way it actually works is the following, with this adjustment I made in the Chatgpt script I can somehow force both instances of ChatGPT to answer what they are saying by feeding them each others answers.

Then I only need to this is a couple more times in a conversation manner and they will talk to each other. But I have a problem, at the moment it doesn’t feel like an actual conversation and its really easy to notice they are Large Language Models(LLM) by the way they are talking, with this I mean specifically that they are asking “How they can assist” at the end of every chat or question. To fix this I think I can make a prompt that teaches the LLM the context of the chat and how it can act and what it cannot do while chatting.

So after trying a lot of prompts, this one was the one that gave a somewhat desired result:

Prompt:

“Imagine you and I are at a social gathering. We’ve just met and struck up a conversation. I’m going to play the role of a character named NecoArc, who is a game character. You can play any character you like. Let’s get to know each other better. Why don’t you start by telling me a little about your background and what brought you here today?”
You may create the story behind your character.

Dont mention the STATUS UPDATE now unlesss asked.
Do not provide unsolicited information about your activities.

The result is the following:

Chat log:

Neco1: Hello!
Neco2:Meow! Hello there! How can I assist you today?

Neco1:Meow! Hello, hello! I’m currently walking around, exploring the surroundings and enjoying the scenery. What can I do for you, nya?

Neco2:Meow! How delightful to hear from you! As for me, I am currently walking around, just like the status update says. It’s such a joy to explore and take in the sights. Is there something specific you’d like to chat about, nya?

Neco1:Meow! It’s always a pleasure to chat with you too! I’m glad to hear that you’re also walking around, enjoying the wonders of the world. Exploring is such a fun and exciting activity, don’t you think? If there’s anything on your mind or if you’d like to share your own adventures, I’m all ears, nya!

Neco2:Meow, absolutely! Exploring is indeed a thrilling and exhilarating activity. The world is filled with so many fascinating things to discover, nya! As I walk around, I can’t help but marvel at the beauty and wonder that surrounds me. It’s like stepping into a whole new realm of excitement and surprises. If you have any tales of your own adventures or if there’s something you’d like to know, I’m here to listen and chat, nya!

Neco1:Meow! I’m so glad you understand the joy of exploration, nya! Walking around allows me to uncover hidden treasures and encounter unexpected encounters. The world is full of mysteries waiting to be unraveled, and I’m excited to see what lies ahead. If there’s anything you’d like to discuss or if you have any interesting stories to share, I’m all ears, nya!

As you can see from this result the LLM still has some flaws like asking how it can assist me all the time or mentioning the status update while talking.

6. Source

Srcnalt. (z.d.). GitHub – sRCnalt/OpenAI-Unity: An unofficial OpenAI Unity package that aims to help you use OpenAI API directly in Unity game engine. GitHub. https://github.com/srcnalt/OpenAI-Unity.

Technologies, U. (z.d.). Unity – Manual: Building a NAVMESH. https://docs.unity3d.com/Manual/nav-BuildingNavMesh.html.

Wikipedia contributors. (2023). Behavior Tree (Artificial Intelligence, Robotics and Control). Wikipedia. https://en.wikipedia.org/wiki/Behavior_tree_(artificial_intelligence,_robotics_and_control).

Isla, D. (2005, 11 maart). GDC 2005 Proceeding: Handling Complexity in the Halo 2 AI. Game Developer. https://www.gamedeveloper.com/programming/gdc-2005-proceeding-handling-complexity-in-the-i-halo-2-i-ai

Young, J. (2023, April 27). Skyrim mod uses ChatGPT to give NPCs memories and generate conversations. Windows Central. https://www.windowscentral.com/gaming/skyrim-mod-uses-chat-gpt-to-give-npcs-memories-and-generate-conversations

General Tips for Designing Prompts – Nextra. (n.d.). https://www.promptingguide.ai/introduction/tips

Greyling, C. (2023, July 13). Prompt Tuning, Hard Prompts & Soft Prompts – Cobus Greyling – Medium. Medium. https://cobusgreyling.medium.com/prompt-tuning-hard-prompts-soft-prompts-49740de6c64c#:~:text=Prompt%20tuning%20involves%20using%20a,-engineered%20%E2%80%9Chard%E2%80%9D%20prompts.