# Introduction
## Overview NRN Agents is an AI gaming SDK that enables effortless creation, training, and deployment of intelligent agents for games. The SDK integrates seamlessly with various game genres, empowering studios to craft immersive, intelligent gameplay experiences for players worldwide. Our technology specializes in modelling human player behavior through a process called imitation learning, which sets it apart from generative AI solutions like LLMs or image generation engines. We achieve this by collecting data on player actions in various in-game situations. We have two areas of specialization: * **Imitation Learning:** Agents that play games as humans do—solving tasks and subtasks in a natural, player-like manner. * **Reinforcement Learning:** Agents that are free from the limitations of human capabilities—they learn through a reward mechanism instead of strictly trying to copy humans. NRN Agents currently offers two integration packages: **Full in-game integration** ▸ We help studios create innovative and exciting AI-integrated experiences. NRN Agents' proprietary machine learning infrastructure reduces AI integration costs, making it scalable and profitable for studios to explore AI-enhanced gameplay. **In-game inference and access to NRN’s Trainer Platform** ▸ We provide studios with a variety of tools that help solve common challenges, starting with improving player liquidity, enhancing user experience, and boosting player retention. Below is a non-exhaustive list of game genres we operate in: * Shooter games (top-down, first-person, and third-person) * Fighting games (platform fighting and traditional FGC) * Social casino games * Racing games * Trading Card Games * RPG (solving sub-tasks) * MMO (solving sub-tasks) * MOBA (solving sub-tasks) We provide SDK extensions for games developed using TS/JS, Unity, or Unreal Engine. {% hint style="warning" %} Unreal Engine integration has been deprioritized and will be revisiting it in H2 2025 {% endhint %} ## API Keys Every game studio that we work with will have 2 API keys: * Admin API key * In-game API key The admin API key is responsible for managing the model architectures for the game studio, which is outlined [here](/admin/model-architectures). The in-game API key manages model creation, inference, data collection, and training. The in-game keys are further separated into full-integration access and trainer platform access. # Installation & Setup ## Installation Methods for installing the package differs for each game engine. For games building browser games (using Javascript), you can install the package as follows: ```bash npm install nrn-agents ``` For studios that use Unity or Unreal engine, please directly message an NRN team member to get access to the package. ## Setup In order to start using the SDK, you will need to set your `apiKey` and `gameId` as static attributes on the `AgentFactory` class (in Javascript) or `ModelWrapper` class (in C#). Each `apiKey` only works for a specific `gameId`. {% tabs %} {% tab title="Javascript" %} ```javascript const { AgentFactory } = require('nrn-agents'); AgentFactory.setApiKey("my-api-key"); AgentFactory.setGameId("my-game-id"); ``` {% endtab %} {% tab title="C# - Unity" %} ```csharp using ArcAgents.MachineLearning; ModelWrapper.SetApiKey("my-api-key"); ModelWrapper.SetGameId("your-game-id"); ``` {% endtab %} {% endtabs %} Now that you have your `apiKey` and `gameId` set, every HTTP request you make to the NRN Agents server will automatically use it. ## Additional Files In order for the NRN Agents SDK to know how to communicate with a game, we need to add 2 files. These files will define the [state space](/getting-started/state-space) and [action space](/getting-started/action-space), which we begin to discuss in the [basic integration](/getting-started/basic-integration) section. ### Javascript Structure It is advised that game studios make an `nrn-integration` folder inside their `helpers` folder. Inside this folder, we need to add the files to define the state and action space. ``` . └── src/ └── helpers/ └── nrn-integration/ ├── nrn-state-space.js └── nrn-action-space.js ``` ### Unity Structure It is advised that game studios make an `NrnIntegration` folder inside the`Script` folder. Inside this folder, we need to add the files to define the state and action space. ``` . └── Assets/ └── Scripts/ └── NrnIntegration/ ├── NrnStateSpace.cs └── NrnActionSpace.cs ``` ### Unreal Structure It is advised that game studios make an `NrnIntegration` folder inside the folder where they have all their game code. Inside this folder, we need to add the files to define the state and action space. ``` . └── Source/ └── Game/ └── NrnIntegration/ ├── NrnStateSpace.cpp ├── NrnStateSpace.h ├── NrnActionSpace.cpp └── NrnActionSpace.h ``` # Overview Admin access is responsible for registering and unregistering model architectures. Once we have a model architecture registered, then we are able to create models in-game and begin to use them! The process for defining the model architecture will require some initial joint collaboration from the NRN team and the game studio. Ultimately we need to settle on: * Type of inductive bias * Size and structure of the state space * Size and structure of the action space * Overall size of the model We dive deeper into each of these in the [model architectures](/admin/model-architectures) section. {% hint style="info" %} REMINDER: The API key you'll need to use for registration is the admin API key {% endhint %} # Model Architectures ## Inductive Bias We have 2 models currently in production for NRN Agents: 1. **Feedforward Neural Network:** This model is well-suited for most environments that have a continuous state space. The state space for this model must have the ability to be represented as a vector. 2. **Tabular Agent:** This model is specifically designed for environments in which we can define the state space as a set of discrete scenarios. We have 2 other models in the final stages of testing and will be released to production very soon: 1. Hierarchical Neural Network 2. Convolutional Neural Network {% hint style="info" %} Since most games require agents to perform multiple actions simultaneously, our models are all capable of multi-task learning {% endhint %} ## State Space When registering a model, we need to know the size of the state space. In the case of a feedforward neural network, this is the number of features in the input vector. While for the tabular agent, this is the number of discrete scenarios. When starting the process, the NRN team will work collaboratively with the game studio to figure out which features are important for decision making and design a state space around that. Learn more about state spaces [here](/getting-started/state-space). ## Action Space As part of the initial research, we have to figure out all the actions a player can take in the game and come up with an action space for the agent. The end result will largely depend on how many actions an agent can take simultaneously and whether the actions are discrete or continuous. Learn more about action spaces [here](/getting-started/action-space). ## Model Size The size of the network determines the degrees of freedom that a model has to learn an objective. Generally speaking, the larger the neural network, the higher likelihood it can learn "more". Thus, as the complexity of the games increase, so will the size of the neural networks that we decide to deploy. Part of the research process will be testing various model sizes to find the perfect fit for the game. We want the model to be large enough to behave intelligently in the game, yet small enough to perform inference very fast and not cause frame drop. A member of the NRN team will be advising each game on the appropriate model size. # Register In order to register model architectures for a game on the NRN Agents platform, we will need to define a few parameters: * `modelType`: `"neural-network"` for Feedforward Neural Network or `"simple"` for Tabular Agent. * `architectureId`: The unique identifier you want to use for your model architecture. * `inputDim`: The number of features in the state space (i.e. the input dimensionality). * `actions`: The structure of the action space. Each action "head" is responsible for a portion of the action space. Thus, we have to define the list of actions that each head is able to execute. {% hint style="info" %} Developers can register their models in a no-code environment on our [dashboard](https://arcagents.ai) {% endhint %} ## Javascript Through the Javascript SDK developers can register as follows: ```javascript const { Registry } = require('nrn-agents'); Registry.setApiKey("my-admin-api-key"); const registry = new Registry("my-game-id"); const registrationInputs = { modelType: "simple", architectureId: "my-first-model", inputDim: 7, actions = { direction: ["up", "down", "idle"] } } await Registry.register(registrationInputs) ``` ## Unity For Unity games, we created a component in the library that admins can add to the inspector. To use the registration component, follow these instructions: 1. Select any game object in the scene 2. In the Inspector, click `Add Component` and search for `NrnAgents` then proceed to select `Registration` 3. Fill out all the appropriate parameters and then click the "Register" button Below is a snapshot of what the registration component looks like in the Unity inspector.
## Unreal Not yet implemented. The NRN team will handle registration for the time being. # Unregister In the initial research phase of an NRN integration, it is very likely that we will test an architecture setup that does not work that well. As such, it is advised that game studios unregister their architecture to remove any unused architectures/clutter. To unregister, admins only need to provide the `architectureId` as an input. {% hint style="info" %} Developers can unregister their models in a no-code environment on our [dashboard](https://arcagents.ai) {% endhint %} ## Javascript As mentioned in the register section, there is currently no GUI for registration, so admins will have to execute code as follows to unregister a model architecture: ```javascript const { Registry } = require('nrn-agents'); Registry.setApiKey("my-admin-api-key"); const registry = new Registry("my-game-id"); await registry.unregister("my-first-model) ``` ## Unity To unregister, admins will need to add the "Registration" component as defined [here](/admin/register#unity). At the bottom of the component, admins will be able to enter the `architectureId` of the model architecture that they wish to remove.
## Unreal Not yet implemented. The NRN team will handle unregistering for the time being. {% hint style="danger" %} Unregistering active model architectures may break existing pipelines. Always ensure no current models are running on an architecture that you plan on unregistering. {% endhint %} # Basic Integration As mentioned in the [setup](/overview/installation-and-setup#additional-files) section, game studios will be required to create 2 files in order for the NRN Agent to "understand" the game world, and for the game world to "understand" the NRN Agent output. The NRN team will provide partner game studios with template files for Javascript, Unity, and Unreal.

Red indicates what is understandable for the game, while blue indicates what is understandable for NRN Agent

After instantiating the NRN Agent, the basic flow for integration is as follows: * Convert the game world into an state space that the NRN model can use * **Human Controlled** * Update human inputs * Convert executed inputs to an action that the NRN model can understand * Add the state and action pair to the dataset * NRN **Agent Controlled** * Perform inference to select the action * Convert the selected action to an input that the game can understand * Either send data to the trainer platform or train directly in-game Below we show how to initialize the model and use it at the game manager level: {% tabs %} {% tab title="Javascript" %} ```javascript class Game { // Instantiate agent constructor(agentInputs) { this.agent = AgentFactory.createAgent(...agentInputs); } // Game loop update update() { // Get the state space for the model const state = getStateSpace(this); if (this.human) { // If human is controlling, then update their inputs this.inputs.p1.update(); // Convert the inputs into something the NRN agent can understand const action = getActionOneHot(this.inputs.p1); // Add the observed state and action to the dataset this.agent.collect({ state, action }); } else { // If NRN agent is controlling, then perform inference const agentAction = this.agent.selectAction(state); // Execute the action in-game convertActionToGame(agentAction); } } } ``` Then at the end of the game, you can send the data you collected to the trainer platform as follows: ```javascript await agent.uploadData(); ``` OR If you have the API access to train in-game, then you can directly train within the game loop as follows: ```javascript await agent.train(model.getTrainingData(), trainingConfiguration); ``` {% endtab %} {% tab title="C# - Unity" %} ```csharp public class Game : MonoBehaviour { // Define the agent that will be taking actions public IAgentRL Agent { get; set; } // Instantiate model private async void _loadAgent(modelInputs) { Agent = AgentFactory.CreateAgent("reinforcement", modelInputs); } // Game loop update private async void Update() { // Create the world representation which we will convert to the state space matrix var world = new NrnIntegration.NrnWorld { // All world attributes/objects that are relevant for the AI }; // Get the state space for the model Matrix state = NrnIntegration.NrnStateSpace.GetState(world); if (human) { // Convert the inputs into something the NRN agent can understand Dictionary action = NrnActionSpace.ConvertInputToBools(); // Add the observed state and action to the dataset Agent.Collect({ state, action }); } else { // If NRN agent is controlling, then perform inference Dictionary agentAction = Agent.SelectAction(state); // Execute the action in-game NrnActionSpace.ConvertActions(agentAction); } } } ``` Then at the end of the game, you can send the data you collected to the trainer platform as follows: ```csharp await Agent.UploadData(); ``` OR If you have the API access to train in-game, then you can directly train within the game loop as follows: ```csharp await AgentArc.Train(); ``` {% endtab %} {% endtabs %} {% hint style="info" %} The NRN team will work with each game studio on creating an optimal set of training configurations for their games {% endhint %} # State Space In order for the agents to understand the game world, we need to convert the game state into a format that it can digest - we will call this the model's "state space".
Each model architecture will have different structure for the state space, so it is important to clearly define the features that are important to your game and the method to extract them. Developers can either use the built-in feature engineering module or create their own custom features. ### Feature Engineering Module The NRN agents SDK comes with an ability to automatically extract features from a game world. Developers only need to define a configuration so the SDK knows how to access certain values from the game world. Below we show an example getting the following features: * Raycasts that originate from the player and detect enemies around it * Relative position (distance and angle) to a powerup {% tabs %} {% tab title="Javascript" %} ```javascript import { FeatureEngineering } from "nrn-agents" FeatureEngineering.setStateConfig([ { type: "raycast", keys: { origin: "player", colliders: "enemies", maxDistance: "gameArea.width" }, setup: { numRays: 8 } }, { type: "relativePosition", keys: { entity1: "player", entity2: "items[0].powerup", maxDistance: "gameArea.width" } } ]) const state = FeatureEngineering.getState(world) ``` {% endtab %} {% tab title="C# - Unity" %} Coming Soon {% endtab %} {% endtabs %} The state config is comprised of an array of feature configs of the following format: ```typescript { type: string, // Name of the feature type keys: Record, // Keys used to extract values from the game world setup?: Record // Additional setup parameters } ``` #### Features Available In the configurations below, we use the notation `string -> objectType` to denote that the value for the key must be a string that points to an object of a particular type.
Raycast Configuration ```typescript { type: "raycast", keys: { origin: string -> { x: number, y: number }, colliders: string -> { x: number, y: number, width: number, height: number }[], maxDistance: string -> number }, setup?: { numRays: number } } ``` Returns: Array of Length **numRays** ```javascript [ Ray 1, // (1 / numRays) * 360 degrees Ray 2, // (2 / numRays) * 360 degrees Ray 3, // (3 / numRays) * 360 degrees ... Ray N, // 360 degrees ] ```
Relative Position Configuration ```typescript { type: "relativePosition", keys: { entity1: { x: number, y: number }, entity2: { x: number, y: number }, maxDistance: number } } ``` Returns: Array of Length **3** ```javascript [ Distance // 0 is farthest, 1 is closest Sin(Radians) // Direction indication #1 Cos(Radians) // Direction indication #2 ] ```
Relative Position To Cluster Configuration ```typescript { type: "relativePositionToCluster", keys: { origin: { x: number, y: number }, clusterEntities: { x: number, y: number }[], maxDistance: number } } ``` Returns: Array of Length **3** ```javascript [ Distance // 0 is farthest, 1 is closest Sin(Radians) // Direction indication #1 Cos(Radians) // Direction indication #2 ] ```
One Hot Encoding Configuration ```typescript { type: "onehot", keys: { value: string }, setup: { options: string[] } } ``` Returns: Array of Length **N**, where `N = setup.options.length` ```javascript [ 0 or 1, // 1 if option[0] == value, otherwise 0 0 or 1, // 1 if option[1] == value, otherwise 0 ... 0 or 1, // 1 if option[N-1] == value, otherwise 0 ] ```
Binary Configuration ```typescript { type: "binary", keys: { value: number | string }, setup: { operator: "=" | ">" | "<" | "!=", comparison: number | string } } ``` Returns: Array of Length **1** ```javascript [ 0 or 1, // 1 if criteria is met, otherwise 0 ] ```
Rescale Configuration ```typescript { type: "rescale", keys: { value: number, scaleFactor: number } } ``` Returns: Array of Length **1** ```javascript [ Rescaled value, // Number between 0 and maxPossibleNumber/scaleFactor ] ```
Normalize Configuration ```typescript { type: "normalize", keys: { value: number }, setup: { mean: number, stdev: number } } ``` Returns: Array of Length **1** ```javascript [ Normalized value, // Rescaled number by mean and standard deviation ] ```
### Custom Features If developers want to create features that are currently not offered by the feature engineering module, then they are able to. Below we showcase how to convert a [Pong](https://en.wikipedia.org/wiki/Pong) game world to a state matrix with custom feature engineering: {% tabs %} {% tab title="Javascript" %} ```javascript const bound = (x) => { return Math.max(Math.min(x, 1), -1) } const getStateSpace = (world) => { const paddlePos = { x: world.paddleLeft.x, y: world.paddleLeft.y + world.paddleLeft.height / 2 } const paddleScaling = 1 - world.paddleLeft.height / world.gameArea.height const absolutePaddlePos = (paddlePos.y / world.gameArea.height - 0.5) * 2 / paddleScaling return [[ absolutePaddlePos, (world.ball.y - paddlePos.y) / world.gameArea.height, bound((-world.ball.x / world.gameArea.width + 0.5) * 2), bound(world.ball.dx / 8), bound(world.ball.dy / 8) ]] } const state = getStateSpace(world) ``` {% endtab %} {% tab title="C# - Unity" %} ```csharp using NrnAgents.MathUtils; namespace NrnIntegration { class NrnStateSpace { private static float gameAreaHeight = 8; private static float gameAreaWidth = 16; private static float paddleHeight = 2; private static float PaddleScaling { get; set; } public NrnStateSpace() { PaddleScaling = 1 - (paddleHeight / gameAreaHeight); } public static double bound(double x) => Math.Min(1, Math.Max(-1, x)); public static Matrix GetState(NrnWorld world) { double leftPaddleY = world.leftPaddlePos.y; double absolutePaddlePos = (leftPaddleY / gameAreaHeight) * 2; double yDist = (world.ballPos.y - leftPaddleY) / gameAreaHeight; double xDist = bound((-world.ballPos.x / gameAreaWidth ) * 2); double xVel = bound(world.ballVel.x / 8); double yVel = bound(world.ballVel.y / 8); return CreateStateMatrix(new List { absolutePaddlePos, yDist, xDist, xVel, yVel }); } private static Matrix CreateStateMatrix(List stateList) { Matrix state = new(1, stateList.Count); List> nestedListState = new List> { stateList }; state.FillFromData(Matrix.To2DArray(nestedListState)); return state; } } public class NrnWorld { public Vector3 ballPos { get; set; } public Vector3 ballVel { get; set; } public Vector3 leftPaddlePos { get; set; } public Vector3 rightPaddlePos { get; set; } } } ``` {% endtab %} {% endtabs %} {% hint style="info" %} Developers can also combine the built-in feature engineering functionality with their own custom features. {% endhint %} # Action Space In order for model's actions to convert to something executable in the game world, we need to map it to controller/keyboard inputs - we will call this the model's "action space".
In order to use the actions that an NRN agent recommends, we need to convert it into a format that the game can understand - this will be different for every game. Below we showcase an example of converting the output from an NRN model to a movement vector which can be used in a pong game: {% tabs %} {% tab title="Javascript" %} ```javascript const convertActionToGame = (actions) => { const movementInput = new Vector3() if (actions.up) { movementInput.y = 1 } else if (actions.down) { movementInput.y = -1 } return movementInput } const getActionOneHot = (inputs) => { const action = [0, 0, 0] if (this.pressed["KeyW"]) { action[0] = 1 // Up } else if (this.pressed["KeyS"]) { action[1] = 1 // Down } else { action[2] = 1 // idle } } ``` {% endtab %} {% tab title="C# - Unity" %} ```csharp using UnityEngine; namespace NrnIntegration { class NrnActionSpace { public static Vector3 ConvertActions(Dictionary actions) { Vector3 movementInput = new Vector3(); if (actions["up"]) movementInput = new Vector3(0,1f,0); else if (actions["down"]) movementInput = new Vector3(0,-1f,0); else if (actions["idle"]) movementInput = new Vector3(0,0,0); return movementInput; } public static Dictionary ConvertInputToBools() { Dictionary actions = new Dictionary(); actions["up"] = Input.GetKey(KeyCode.W); actions["down"] = !actions["up"] && Input.GetKey(KeyCode.S); actions["idle"] = !actions["up"] && !actions["down"]; return actions; } } } ``` {% endtab %} {% endtabs %} In the code shown above, we have 2 functions for action conversion: * **Model -> Game:** This is used to execute the recommended action in-game * **Game -> Model:** This is used to take human inputs and turn it into data that the model can use for training # Data Collection Below we show a visual of the flow when a human gamer is playing the game and we are collecting data to be used in training at a later time.
Once we obtain the state and action pair, which can be seen in the [basic integration](/getting-started/basic-integration), we simply call the collect method as follows: {% tabs %} {% tab title="Javascript" %} ```javascript agent.collect({ state, action }); ``` {% endtab %} {% tab title="C# - Unity" %} ```csharp Agent.Model.Collect({ state, action }); ``` {% endtab %} {% endtabs %} By default we collect data every 10 frames, but the optimal frequency will change depending on the game genre and frame rate of the game. In order to change the data collection interval, we can use the following line of code: {% tabs %} {% tab title="Javascript" %} ```javascript agent.setCollectionInterval(15); // Collect every 15 frames ``` {% endtab %} {% tab title="C# - Unity" %} ```csharp Agent.Model.SetCollectionInterval(15); // Collect every 15 frames ``` {% endtab %} {% endtabs %} # Model Initialization ### Demo In order to test out the SDK, users can create a demo agent that can start taking actions in their game! The purpose of the demo agent is to test out the entire flow of feeding in a state, taking actions, collecting data, and executing the actions in-game. {% tabs %} {% tab title="Javascript" %} ```javascript const { AgentFactory } = require('nrn-agents'); const modelData = { config: { modelType: "neural-network", // Type of model architecture inputDim: 5, // Number of features for the state neurons: [12, 6], // Number of neurons in each layer actionOrder: ["up", "down", "idle"] // Order of action outputs } } const agent = AgentFactory.createDemoAgent(modelData) ``` {% endtab %} {% tab title="C# - Unity" %} ```csharp using NrnAgents.Agents; using NrnAgents.MachineLearning; ModelData modelData = new () { Config = new () { ModelType = "neural-network", NFeatures = 5, Neurons = new List () { 12, 6 }, ActionMetadata = MetadataCreation.CreateActionMetadata( new AgentActions () { ActionOrder = new () { "Up", "Down", "Idle" } } ) } }; DemoAgent Agent = AgentFactory.CreateDemoAgent(modelData); ``` {% endtab %} {% endtabs %} {% hint style="info" %} Demo agents do not have the ability to learn {% endhint %} ### Production When initializing a model in-game, we need to use the following inputs: * `architectureId`: A model architecture that has been registered * `userId`: Unique identifier for a specific player * `slotIdx`: Which model to use since each player may have multiple model slots If NRN detects that there is already a model loaded for that user, it will load the trained model. In order to create a new randomly initialized model, developers can either input an unused `slotIdx` when creating the agent or call the `reset` method (currently only available in Javascript). {% tabs %} {% tab title="Javascript" %} ```javascript const { AgentFactory } = require('nrn-agents'); const agent= AgentFactory.createAgent( "my-first-model", // architecture id "user-id", // user id 0, // slot idx (default 0 if not provided) ) await agent.initialize() agent.reset() // To reset the agent ``` {% endtab %} {% tab title="C# - Unity" %} ```csharp using NrnAgents.Agents; IAgentRL Agent = AgentFactory.CreateAgent( "reinforcement", // learning algorithm type "my-first-model", // architecture id "user-id", // user id 0, // slot idx ); ``` {% endtab %} {% endtabs %} # Inference In order to actually use the model in game, we need to perform inference. This means that we give the model a state matrix, and have it output an action. For most use cases, it is appropriate to use the "select action" method. This will create a mapping which can then be used, via the action space conversion mention in [this section](/getting-started/action-space), to execute the action in-game. Alternatively, we allow game studios to access the "raw output", which is the probabilities of taking each of the actions in that state. The NRN Agents SDK comes with a **Probabilistic Agent Wrapper** that handles the distribution, but if game studios want more flexibility in building a custom wrapper, they are able to do this as well. {% tabs %} {% tab title="Javascript" %} ```javascript // Sample from the distribution to select an action const action = agent.selectAction(state); // Get the probability distribution over actions const probabilities = agent.getProbabilities(state); ``` {% endtab %} {% tab title="C# - Unity" %} ```csharp // Sample from the distribution to select an action Dictionary action = Agent.SelectAction(state); // Sample from the distribution to select an action Dictionary probabilities = Agent.GetProbabilities(state); ``` {% endtab %} {% endtabs %} # Training Basic training We have some default that is totally abstracted Potentially have a mapping of hyperparams for each game Link to advanced training for difficult topics in ML Not all users will have access to training natively through the SDK - only those with API access for full-integration. Otherwise training happens through our network of trainers on the trainer platform. Docs can be found here # Saving Show how to save a model and explain the concept of model slots {% tabs %} {% tab title="Javascript" %} ```javascript await model.save() ``` {% endtab %} {% tab title="C#" %} ```csharp await Model.Save(); ``` {% endtab %} {% endtabs %} # Unreal Code Examples Basic Integration {% tabs %} {% tab title="First Tab" %} ```cpp void PlayerController::BeginPlay() { Super::BeginPlay(); World = NewObject(this); ARCActionSpace = NewObject(this); ARCStateSpace = NewObject(this); Model = UARCBPLibrary::LoadModel(ModelConfig, false); } void PlayerController::Tick(float Delta) { Super::Tick(Delta); // Create the world representation which we will convert to the state space matrix World->InitializeWorld( // All world attributes/objects that are relevant for the AI ); // Get the state space for the model TArray State = ARCStateSpace->GetState(*World); if (human) { // Convert the inputs into something the ARC agent can understand TMap Action = ARCActionSpace->ConvertInputToBools(*World); // Add the observed state and action to the dataset Model->Collect(State, Action); } else { // If ARC agent is controlling, then perform inference auto ModelAction = UARCBPLibrary::SelectAction(Model,State); // Execute the action in-game auto Action = ARCActionSpace->ConvertActions(ModelAction); } } ``` Then at the end of the game, you can send the data you collected to the trainer platform as follows: ```csharp Model->UploadData(); ``` OR If you have the API access to train in-game, then you can directly train within the game loop as follows: ```csharp Model->Train(); ``` {% endtab %} {% endtabs %} State Space {% tabs %} {% tab title="First Tab" %} First we will create the header file ```c #pragma once #include "CoreMinimal.h" #include "UObject/NoExportTypes.h" #include "ARCStateSpace.generated.h" class Pawn; UCLASS(Blueprintable) class UARCWorld : public UObject { GENERATED_BODY() public: UARCWorld(const FObjectInitializer& ObjectInitializer); TObjectPtr LeftPaddlePawn; TObjectPtr BallActor; void InitializeWorld(TObjectPtr InLeftPaddlePawn, TObjectPtr InBallActor) { LeftPaddlePawn = InLeftPaddlePawn; BallActor = InBallActor; } }; UCLASS(Blueprintable) class UARCStateSpace : public UObject { GENERATED_BODY() public: UARCStateSpace(const FObjectInitializer& ObjectInitializer); static TArray GetState(const UARCWorld& World); private: double Bound(double x); }; ``` Then we will create the C++ file ```cpp #pragma once #include "ARCStateSpace.h" UARCWorld::UARCWorld(const FObjectInitializer& ObjectInitializer) : Super(ObjectInitializer) { } UARCStateSpace::UARCStateSpace(const FObjectInitializer& ObjectInitializer) : Super(ObjectInitializer) { } TArray UARCStateSpace::GetState(const UARCWorld& World) { const double GameAreaHeight = 8.0; const double GameAreaWidth = 16.0; double LeftPaddleY = World.LeftPaddlePawn->GetActorLocation().Y; double BallY = World.BallActor->GetActorLocation().Y; double BallX = World.BallActor->GetActorLocation().X; FVector BallVel = World.BallActor->GetVelocity(); double AbsolutePaddlePos = (LeftPaddleY / GameAreaHeight) * 2.0; double YDist = (BallY - LeftPaddleY) / GameAreaHeight; double XDist = Bound((-BallX / GameAreaWidth) * 2.0); double XVel = Bound(BallVel.X / 8.0); double YVel = Bound(BallVel.Y / 8.0); return { AbsolutePaddlePos, YDist, XDist, XVel, YVel }; } double UARCStateSpace::Bound(double x) { return FMath::Clamp(x, -1.0, 1.0); } ``` {% endtab %} {% endtabs %} Action space {% tabs %} {% tab title="First Tab" %} First we will create the header file ```c #pragma once #include "CoreMinimal.h" #include "UObject/NoExportTypes.h" #include "ARCStateSpace.h" #include "ARCActionSpace.generated.h" UCLASS(Blueprintable) class UARCActionSpace : public UObject { GENERATED_BODY() public: UARCActionSpace(const FObjectInitializer& ObjectInitializer); TMap ConvertInputToBools(const UARCWorld& World); FVector ConvertActions(TMap Action); }; ``` Then we will create the C++ file ```cpp #pragma once #include "ARCActionSpace.h" UARCActionSpace::UARCActionSpace(const FObjectInitializer& ObjectInitializer) { } FVector UARCActionSpace::ConvertActions(TMap Action) { FVector MovementInput = FVector::ZeroVector; if (actions["up"]) { MovementInput = FVector(0.0f, 1.0f, 0.0f); } else if (actions["down"]) { MovementInput = FVector(0.0f, -1.0f, 0.0f); } return MovementInput; } TMap UARCActionSpace::ConvertInputToBools(const UARCWorld& World) { TMap Action; FVector PaddleVelocity = World.LeftPaddlePawn->GetVelocity(); Action.Add("up", PaddleVelocity.Y > 0.0f); Action.Add("down", PaddleVelocity.Y < 0.0f); Action.Add("idle", PaddleVelocity.Y == 0.0f); return Action; } ``` {% endtab %} {% endtabs %} Data Collection ```cpp Model->Collect(State, Action); ``` ```cpp Model->SetCollectionInterval(15); // Collect every 15 frames ``` Model Initialization ```cpp ModelConfig.ArchitectureId = "my-first-model"; ModelConfig.UserId = "user-id"; ModelConfig.SlotIdx = 0; Model = UARCBPLibrary::CreateModel(ModelConfig); ``` Inference 
auto Action = UARCBPLibrary::SelectAction(Model, State);

// Sample from the distribution to select an action
auto Probabilities = UARCBPLibrary::GetProbabilities(Model, State);
# Catastrophic Forgetting Explain the problem at a high level and how users can combat it with our SDK
Technical Explanation Explain Catastrophic Forgetting
{% tabs %} {% tab title="Javascript" %} ```javascript await model.train(trainingData, { memoryRetention: { direction: 1 } }); ``` {% endtab %} {% tab title="C#" %} ```csharp await Model.Train(trainingData, { memoryRetention: { direction: 1 } }) ``` {% endtab %} {% endtabs %} Some sort of abstraction (registration or we work with them) for grouping # Spurious Correlations Explain the problem at a high level and how users can combat it with our SDK Some sort of abstraction (registration or we work with them) for grouping This is available for Neural Networks only
Technical Explanation Some text to explain spurious correlations
{% tabs %} {% tab title="Javascript" %} ```javascript await model.train(trainingData, { focus: [0, 1, 2] }) ``` {% endtab %} {% tab title="C#" %} ```csharp await Model.Train(trainingData, { focus: [0, 1, 2] }); ``` {% endtab %} {% endtabs %} Some sort of abstraction (registration or we work with them) for grouping # Data Cleaning
Technical Explanation Why is this important?
# Inspector Most extreme case is making a lightweight version of the environment completely toggleable. We have this for every game that integrates onto the trainer platform. You can view it [here](/trainer-platform/overview) Full in-game integrations can also do this, but it's a heavier lift for game studios, so a popular choice is a scenario generator. You can do so with the following code. Start by defining the game world scenarios {% tabs %} {% tab title="Javascript" %} ```javascript const scenarios = [ { gameArea, paddleLeft: { ...paddleLeft, y: 100 }, paddleRight: { ...paddleRight, y: 200 }, ball: { radius: ballRadius, x: 125, y: 300, dx: -4, dy: -4 } }, { gameArea, paddleLeft: { ...paddleLeft, y: 300 }, paddleRight: { ...paddleRight, y: 150 }, ball: { radius: ballRadius, x: 200, y: 100, dx: -4, dy: 4 } }, { gameArea, paddleLeft: { ...paddleLeft, y: 250 }, paddleRight: { ...paddleRight, y: 50 }, ball: { radius: ballRadius, x: 200, y: 200, dx: -4, dy: -4 } }, { gameArea, paddleLeft: { ...paddleLeft, y: 125 }, paddleRight: { ...paddleRight, y: 200 }, ball: { radius: ballRadius, x: 125, y: 300, dx: -4, dy: 4 } } ] ``` {% endtab %} {% tab title="Second Tab" %} {% endtab %} {% endtabs %} Next, use the state space conversion function: {% tabs %} {% tab title="Javascript" %} ```javascript const scenarioStates = scenarios.map((x) => getNeuralNetState(x)[0]) ``` {% endtab %} {% tab title="Second Tab" %} {% endtab %} {% endtabs %} Next you simply call the function to get the probabilities in each situation: {% tabs %} {% tab title="Javascript" %} ```javascript model.getProbabilities(scenarioStates) ``` {% endtab %} {% tab title="C# - Unity" %} {% endtab %} {% endtabs %} # Agent Wrappers {% tabs %} {% tab title="Javascript" %} ```javascript const agentConfig = { numSamples: { direction: 4 }, holdActions: ["up", "down"], // forcedHold: 4 } const model = new Model("nn-v1", "brandinho", 0, agentConfig, false, true) ``` This is the config structure: ```javascript { frameDelay: 0, forcedHold: 0, holdActions: 0, forceHoldActions: 0, numSamples: 1 } ``` {% endtab %} {% tab title="C# - Unity" %} ```csharp private static AgentConfig _getAgentConfig() { List holdActions = new () { "up", "down" }; List forceHoldActions = new (); Dictionary numSamples = new () {{ "direction", 10 }}; return new AgentConfig(15, 4, holdActions, forceHoldActions, numSamples); } private async Task _loadAgent() { try { bool randomlyInitialize = false; ModelWrapper model = new ModelWrapper("nn-v1", "brandinho", 0); await model.InitializeModelData(randomlyInitialize); model.Collector.FrameInterval = 20; AgentArc = new ProbabilisticAgent(model, _getAgentConfig()); ArcModelLoaded = true; } catch (System.Exception e) { Debug.LogError($"Error loading Arc Model: {e}"); } } ``` This is the config definitions ```csharp public class AgentConfig { public int FrameDelay { get; } public int ForcedHold { get; } public List HoldActions { get; } public List ForceHoldActions { get; } public Dictionary NumSamples { get; } public AgentConfig( int frameDelay, int forcedHold, List holdActions, List forceHoldActions, Dictionary numSamples ) { FrameDelay = frameDelay; ForcedHold = forcedHold; HoldActions = holdActions; ForceHoldActions = forceHoldActions; NumSamples = numSamples; } } ``` {% endtab %} {% endtabs %} # Overview # Send Data