Tell it where to go.
Name a place on de_dust2, and the agent sets off like an assistant answering a prompt. It tells you when it arrives, and if you change your mind halfway, your next message stops it and sends it somewhere else.
An agent you can talk to, inside a 3D world.
Inspired by SIMA 2 from Google DeepMind.
Talk, don't script
Commands are plain English: “rush B”, “go to A site through short”, “take long to A”. A small parser picks out the place for scoring and the minimap, but the policy reads the words themselves.
Words in, keys out
The agent acts through the same controls a player has: forward, strafe, walk, duck, jump, and turning the camera. Ten times a second it decides from the command and what it senses. The goal is never given to it; it has to come from the words.
Interrupt any time
Like a chat assistant, it reports back when it's done, and you can cut in. A new message stops the run where the agent stands and starts the new one from there.
From words to footsteps.
A compact language-conditioned policy in two variants. A frozen T5 reads the instruction; the agent senses the world either through the rendered frame or through its own state. A small transformer combines the two, and a recurrent memory turns them into one keyboard-and-camera action at a time.
Words + senses
The instruction, and either the agent's view at 112 px or its state: position, heading, velocity and range rays.
T5 + DINOv2 or state
Frozen T5-small encodes the text. Frozen DINOv2-small turns a frame into 65 patch tokens, or each state field becomes a token of its own.
Transformer
Two layers at 256 dimensions let the words attend to what the agent senses.
GRU · 512
Carries what the agent has seen and done since the command began.
Action heads
Move, strafe, walk, duck, jump, and yaw and pitch rates, sampled on every decision.
Practice
PPO from scratch in the arena with 2 × 128 agents, rewarded for every unit of walking distance it closes and for arriving. The critic sees the route; the actor only its own inputs and the words.
Grow the map
A curriculum starts with neighbouring regions and adds a region hop each time success passes 80 %, out to the farthest pairs on de_dust2. Routes through a region on the way come next.
Talk
This page. Every message becomes an episode, and a new message interrupts the one under way. The live view is drawn by the Source 2 Viewer renderer with a CS2 player model standing in for the agent.