State Space

In order for the agents to understand the game world, we need to convert the game state into a format that it can digest - we will call this the model's "state space".

Each model architecture will have different structure for the state space, so it is important to clearly define the features that are important to your game and the method to extract them. Developers can either use the built-in feature engineering module or create their own custom features.

Feature Engineering Module

The NRN agents SDK comes with an ability to automatically extract features from a game world. Developers only need to define a configuration so the SDK knows how to access certain values from the game world. Below we show an example getting the following features:

• Raycasts that originate from the player and detect enemies around it

• Relative position (distance and angle) to a powerup

Javascript

import { FeatureEngineering } from "nrn-agents"

FeatureEngineering.setStateConfig([
  {
    type: "raycast",
    keys: { origin: "player", colliders: "enemies", maxDistance: "gameArea.width" },
    setup: { numRays: 8 }
  },
  {
    type: "relativePosition",
    keys: { entity1: "player", entity2: "items[0].powerup", maxDistance: "gameArea.width" } 
  }
])

const state = FeatureEngineering.getState(world)

C# - Unity

Coming Soon

The state config is comprised of an array of feature configs of the following format:

{
    type: string,                 // Name of the feature type
    keys: Record<string, string>, // Keys used to extract values from the game world
    setup?: Record<string, any>   // Additional setup parameters
}

Features Available

In the configurations below, we use the notation string -> objectType to denote that the value for the key must be a string that points to an object of a particular type.

Raycast

Configuration

{
    type: "raycast",
    keys: {
        origin: string -> { x: number, y: number },
        colliders: string -> { x: number, y: number, width: number, height: number }[],
        maxDistance: string -> number
    },
    setup?: { numRays: number }
}

Returns: Array of Length numRays

[
    Ray 1, // (1 / numRays) * 360 degrees
    Ray 2, // (2 / numRays) * 360 degrees
    Ray 3, // (3 / numRays) * 360 degrees
    ...
    Ray N, // 360 degrees
]

Relative Position

Configuration

{
    type: "relativePosition",
    keys: {
        entity1: { x: number, y: number },
        entity2: { x: number, y: number },
        maxDistance: number
    }
}

Returns: Array of Length 3

[
    Distance // 0 is farthest, 1 is closest
    Sin(Radians) // Direction indication #1
    Cos(Radians) // Direction indication #2
]

Relative Position To Cluster

Configuration

{
    type: "relativePositionToCluster",
    keys: {
        origin: { x: number, y: number },
        clusterEntities: { x: number, y: number }[],
        maxDistance: number
    }
}

Returns: Array of Length 3

[
    Distance // 0 is farthest, 1 is closest
    Sin(Radians) // Direction indication #1
    Cos(Radians) // Direction indication #2
]

One Hot Encoding

Configuration

{
    type: "onehot",
    keys: { value: string },
    setup: { options: string[] } 
}

Returns: Array of Length N, where N = setup.options.length

[
    0 or 1, // 1 if option[0] == value, otherwise 0
    0 or 1, // 1 if option[1] == value, otherwise 0
    ...
    0 or 1, // 1 if option[N-1] == value, otherwise 0
]

Binary

Configuration

{
    type: "binary",
    keys: { value: number | string },
    setup: { 
        operator: "=" | ">" | "<" | "!=",
        comparison: number | string
    } 
}

Returns: Array of Length 1

[
    0 or 1, // 1 if criteria is met, otherwise 0
]

Rescale

Configuration

{
    type: "rescale",
    keys: { 
        value: number,
        scaleFactor: number
    }
}

Returns: Array of Length 1

[
    Rescaled value, // Number between 0 and maxPossibleNumber/scaleFactor
]

Normalize

Configuration

{
    type: "normalize",
    keys: { value: number },
    setup: { 
        mean: number,
        stdev: number
    } 
}

Returns: Array of Length 1

[
    Normalized value, // Rescaled number by mean and standard deviation
]

Custom Features

If developers want to create features that are currently not offered by the feature engineering module, then they are able to. Below we showcase how to convert a Pong game world to a state matrix with custom feature engineering:

Javascript

const bound = (x) => {
    return Math.max(Math.min(x, 1), -1)
}

const getStateSpace = (world) => {
    const paddlePos = {
        x: world.paddleLeft.x,
        y: world.paddleLeft.y + world.paddleLeft.height / 2
    }
    const paddleScaling = 1 - world.paddleLeft.height / world.gameArea.height
    const absolutePaddlePos = (paddlePos.y / world.gameArea.height - 0.5) * 2 / paddleScaling
    
    return [[
        absolutePaddlePos,
        (world.ball.y - paddlePos.y) / world.gameArea.height,
        bound((-world.ball.x / world.gameArea.width + 0.5) * 2),
        bound(world.ball.dx / 8),
        bound(world.ball.dy / 8)
    ]]
}

const state = getStateSpace(world)

C# - Unity

using NrnAgents.MathUtils;

namespace NrnIntegration
{
    class NrnStateSpace
    {
        private static float gameAreaHeight = 8;
        private static float gameAreaWidth = 16;
        private static float paddleHeight = 2;
        private static float PaddleScaling { get; set; }
        
        public NrnStateSpace() 
        {
            PaddleScaling = 1 - (paddleHeight / gameAreaHeight);
        }
        
        public static double bound(double x) => Math.Min(1, Math.Max(-1, x));
        
        public static Matrix GetState(NrnWorld world)
        {
            double leftPaddleY = world.leftPaddlePos.y;
            double absolutePaddlePos = (leftPaddleY / gameAreaHeight) * 2;
            double yDist = (world.ballPos.y - leftPaddleY) / gameAreaHeight;
            double xDist = bound((-world.ballPos.x / gameAreaWidth ) * 2);
            double xVel = bound(world.ballVel.x / 8);
            double yVel = bound(world.ballVel.y / 8);
            return CreateStateMatrix(new List<double> { absolutePaddlePos, yDist, xDist, xVel, yVel });
        }

        private static Matrix CreateStateMatrix(List<double> stateList)
        {
            Matrix state = new(1, stateList.Count);
            List<List<double>> nestedListState = new List<List<double>> { stateList };
            state.FillFromData(Matrix.To2DArray(nestedListState));
            return state;
        }
    }
    
    public class NrnWorld
    {
        public Vector3 ballPos { get; set; }
        public Vector3 ballVel { get; set; }
        public Vector3 leftPaddlePos { get; set; }
        public Vector3 rightPaddlePos { get; set; }
    }  
}

Developers can also combine the built-in feature engineering functionality with their own custom features.