Sixty days ago I had never made a video game.

Starting from nothing

I started with a question, not a skill set: could AI agents do the hands-on work of building a game while I did the directing?

The game plan was written on 8 August. The camera moved from isometric to over-the-shoulder two days later. The four classes were decided on 13 August, the same day the project went into version control. Everything since has been built by agents working inside the Unreal editor, with me approving, rejecting, and redirecting.

The first real lesson came early. Eleven gameplay nodes had their wires disconnected, and the project still compiled cleanly. It went into the rules as a green compile proves nothing. That sentence is the reason the testing harness exists.

Eight weeks in five frames

What exists now

A third-person, souls-like action RPG in Unreal Engine 5.8, with lock-on combat, four classes, and procedurally generated dungeons.

Combat runs on Unreal's Gameplay Ability System, shared by every class. Enemies think with behaviour trees. The catacombs under the Sunken Sanctum are generated, not hand-placed. Around that sit saving, rest points, a world map, quests, and a bestiary.

From test scripts to a harness that runs itself

It started with runtime assertions: tests that play the game and check what actually happens, stepped frame by frame so they give the same answer every run.

Generation one: a Director and a watchdog

One headless agent session, the Director, ran the work. A boot script started the editor and waited for it to answer. A watchdog checked every two minutes and nudged anything idle. A healing script handled known failures in three tiers (fix it, nudge it, call the human) and escalated after three tries. Locks kept two agents from editing the level at once. Nothing shipped without passing two gates: a technical QA gate that blocks serious bugs, and a playtester gate that judges how it feels. I ran it from my phone, over chat.

Generation two: a state machine with judgement only where it's needed

The rebuild made the Director a deterministic state machine. Language models are used in only two places, planning and the specialist workers. Everything else (locks, gates, merging, rollback, editor recovery) is ordinary, testable code. New behaviour runs in shadow mode first, deciding what it would have done without touching the project.

1. Fact packet A read-only snapshot of the project. Anything unknown is marked unknown, never guessed.
2. Plan The planner splits the task into short-lived specialist lanes.
3. Build Lanes work in the editor under a mutation lock.
4. Gate 1 Does it work?
5. Gate 2 Does it feel right?
6. Merge or roll back The operator is asked only when a human decision is needed.

Self-healing

The rule is nothing ends blocked or halted. A failed task goes back in the queue with a backoff that starts at 30 minutes and doubles up to six hours. Before retrying, a research lane reads the docs and the web for the error. Infrastructure failures are told apart from real task failures. Stale locks are detected and broken. I only get paged on the second, fourth, and eighth failure of the same task.

Self-learning

A learner watches every Director decision and builds statistics: which routes lead where, which locks are probably abandoned, and how often a feature that passes the technical gate also passes the feel gate, area by area. An issue that recurs across three episodes becomes a written lesson that agents get as evidence. One real catch: a lock had been held for 4.1 days, and the learner rated it 85% likely stale. As of 6 October, an autofix loop researches a fix, scores the options, applies the best one, and learns from the result.

The research paper

Bridging the Formalization Gap in Game Mechanics: Agentic Playtesting, Dynamic Context Engineering, and Probabilistic State-Space Evaluation.

The question it asks: can autonomous agents combine deterministic playtesting rules with probabilistic judgement to find emergent mechanics, stress-test a game whose state never stops changing, and estimate how play will feel, all before a human player sees it?

It proposes a three-tier architecture:

• Tier 1, deterministic Scripted state execution: the cheap, exact checks.
• Tier 2, search Tree search and pathfinding to explore what a player could reach.
• Tier 3, semantic Language-model judges scoring pacing, agency, tension, and emergence.

It also argues against loading everything into one giant context window. Short-lived subagents each get only the facts their job needs. The paper names the failure it is trying to avoid the Telemetry Trap: drowning in logs that measure everything except whether the game is any good.

The harness above is the paper's case study. Every number in the next section comes from it.

The data

The harness went live on 3 October with 1,081 passing tests. Over four days it logged 11,785 decisions.

First window0.38

Latest window0

First window0.33

Latest window0.16

Commit 10

Commit 49640

Shadow-mode decisions

18,515

Technical gate, pass / fail

93 / 3

Feel gate, ready / needs tuning

28 / 22

Active lessons learned

31

What building it taught

• A local reasoning model spent its entire 8,192-token budget thinking and returned an empty plan. Raising the cap to 32,768 fixed it.
• A file search was walking a 453 GB project tree at 97% CPU. Pruning it brought that down to 2.6 seconds.
• A screenshot judge that rates feel from pictures scored no better than chance (AUC 0.48). That negative result is in the paper.

To be honest about where it stands: halts went down, but the failure rate has not, yet. The harness keeps going instead of stopping. Making it succeed more often is the next stretch of work.

Teaching the models, not just the harness

Every agent call can be recorded. Those traces are where the training data comes from.

The target behaviour is specific: when a tool call fails, recover instead of trying the same thing again. A dataset builder finds each failure point in the traces. It has the model try six different next moves, and keeps only the ones that are well formed, don't repeat the failure, and actually respond to the error. Those become LoRA fine-tuning data, trained on the same Mac that serves the models.

Repeat-after-failure, first run

18.8% → 0%

Repeat-after-failure, second run

20.5% → 11.4%

Agent traces in the second run

87

Real tool calls behind the loop-breaker

~430,000

Neither adapter has shipped. Both runs got better on familiar tasks and overfit by the first checkpoint, and two out of three failure points yield no usable example. Collecting enough clean data takes weeks of real harness traffic. The pipeline works; the data is still growing.

Meanwhile the harness traffic has already improved the serving layer. A loop-breaker, tuned on hundreds of thousands of real tool calls from the Unreal harnesses, stops agents that repeat the same failing call. It knows the game's long-running editor jobs (navmesh builds, play-in-editor, baking), so legitimate waiting isn't mistaken for a loop.

Local models: WandaVision

Agents that work overnight make a lot of calls. On local models, a night of work costs electricity, not an API bill.

WandaVision is my other project, built alongside this one. It runs Claude Code and every other coding agent on the network against local models on one Apple Silicon Mac. An agent asks for opus, sonnet, or haiku, and gets a local model sized for that job. The cloud is a last resort, not the default. A live dashboard shows speed, warm-up time, and which fine-tuned adapters are loaded.

Machine

Mac Studio, M5 Max, 64 GB

Heavy tier

Qwen3.6 35B mixture-of-experts, 256k context

Middle tier

Qwen3.5 9B, sees images

Fast tier

Qwen3.5 4B, always warm, about 230 tokens a second

Judge

A 4B model that compares images

Image box

A laptop RTX 4080 running ComfyUI

Speed work mattered. Moving the 27B tier to a faster runtime took code generation from 67.7 to 153.5 tokens a second, with byte-identical output.

What the local stack does

• Vision. On a 16-check test of game screenshots and generated-image review, the small 4B model scored 15 of 16, with a median answer in 1.6 seconds.
• Image judging. A pairwise judge asks about each pair in both orders and only counts consistent verdicts. It caught a generated fox with an extra head floating beside it.
• Images and video. Text-to-image, image editing, and short text-to-video clips with sound, all on the local GPU box.
• 3D meshes. An overnight batch renders concept images, I delete the bad ones, and the survivors become 3D meshes through an image-to-3D pipeline (TRELLIS 2), with background removal and mesh clean-up.
• Auto-rigging. Scripts in the editor rig and conform generated characters to the game's skeleton, with a test map that checks the hero still looks like the hero.

See it play

The harness that builds it, the current build in play, the 3D models it generates, and how it all began.