GamePhanes Benchmarks Coding Agents Through Full Game Build Loops 🔗
Evaluates agents on runtime behavior, debugging, and regression resistance in Godot environments
GamePhanes introduces Terminal-Bench, a benchmark that tests coding agents not by file edits or command outputs, but by their ability to build, debug, and repair playable Godot games through terminal workspaces. Agents receive evaluator-controlled probes and must produce executable runtime feedback—importing projects, accepting input, modifying state, and delivering intended behavior.
The system judges engineering work: code changes, debugging efficacy, and resistance to regressions, not just task completion. Built on the Harbor task contract, GamePhanes provides six playable reference projects and plans 20 tasks across gameplay, engine/runtime, UI/interaction, content/systems, and delivery/quality. Each task is a self-contained Harbor directory with versioned calibration runs completed successfully using Oracle 1.0 and NOP 0.0 baselines. Early trials like Kimi K3 show agents can make substantive changes over 30 turns but still fail hidden boundary assertions, revealing gaps in robust, long-horizon reasoning. The benchmark shifts focus from syntactic correctness to functional, interactive outcomes—aligning evaluation with real-world software engineering demands.
The catch: The benchmark remains narrow in scope, targeting only Godot and JavaScript-based agents, with production tasks still unverified and no public leaderboard yet established.
Why this leads today GamePhanes provides a practical, open-source environment for testing and benchmarking AI agents in Godot, filling a gap in tooling for developers integrating intelligent behaviors into games.
- Evaluate coding agents on full-game debugging workflows
- Test agent ability to modify runtime game states
- Measure regression resistance in interactive software repair
Source: GamePhanes/GamePhanes — based on the project README.