What It Looked Like to Watch an AI Play Its Own Game
A scripted run can reach the credits. Giving the AI the next decision uncovered a different kind of problem.

Seeing The Last Signal launch in an emulator was one of the moments that made the project feel real to me. Watching the AI interact with it was the part I hadn’t pictured when I first asked it to make a game.
There was a practical problem behind it. I wanted the AI to find out whether its own game worked all the way through. I also wanted to experience the story without having read it. The AI needed a way to go into a room, talk to someone, make a choice, and see what happened next while keeping the mystery out of its updates to me.
I had something specific in mind when I asked whether the game was playable. Imagine starting from the title screen and making a character. The next thing you do is choose someone to talk to, read what they say, and pick an answer. You might walk away or open your journal. Later you might save, close the game, and come back. The answers about how well it works have to make it through all of those ordinary little actions.
Three ways to get through the game
The project used a few different kinds of play. One test took the game along a planned route so it could check whether the major events worked. Think of this as the game walking a set of directions with a recorder running. In the November 15 run it reached the credits. Twice the test driver couldn’t get lined up on its own and moved the character into position. Those details belong in the result, because lining up at a door is something I’ll have to do with a controller.
Another recording drove the ordinary game from the title screen. It was a list of actions like walking, pressing Cross, and advancing to the next page. The game had to read them and react the way it would on a controller. That route made it to the credits too. It took about 52 minutes at the pace the recording used. That tells me how long the replay lasted. I’m planning to stop and look around when it’s my turn.
Then Claude got a controller channel of its own. It received a description of the actual state of the game, including the dialogue and available replies. It picked a button or movement, waited for the game to respond, and made the next decision. That process covered four stretches of the game. The log records about 1,500 actions and 340 screens of text.
Those stretches included the opening, a companion’s whole personal story, some fights, and the closing portion of the game. Claude could learn something from the latest reply and pick a different one next. It could get into a bad position in a fight and lose. The project kept the inputs so it could go back and take pictures of specific moments. Watching the interaction unfold was what made it feel different to me from seeing a report that simply said the game had passed.
What turned up
One companion was called by the wrong pronoun. One reply assumed the player had already heard something from another branch of the conversation. In a few cases, choosing a line ended a conversation in silence. Those are small failures when you read them in a report. If you’re the person choosing the reply, they change how the whole exchange feels.
The “other branch” problem is a good example. When someone talks as if you were there for something you never heard, you have to stop and work out whether you missed a detail. It could be an intentional mystery, or the game could have given you the wrong line. When an AI can take a particular route through the conversation and report what it actually got, there’s a concrete way to tell the difference. Four abruptly ending replies got responses or notices after that pass.
Another test found a much messier problem. At the game’s normal pace, a companion could get stuck in a doorway while the person leading the group waited for them. The earlier test advanced time in bigger jumps, and the group got through. With ordinary timing, the scene could wait forever. The team fixed the movement and added a check that exercises it at the smaller time steps the game really uses.
Picture a guide walking up to someone and stopping because that person is in the way. The person in the way is also trying to follow a route, and the guide is now standing in it. Leave the game running and it looks as if everyone is still waiting for something to happen. From a player’s seat, the entire story has stopped. That sort of failure is hard to guess from the text of a quest. It takes actually walking the group through the room.
A surprisingly useful losing streak
Claude also lost fights on Normal. Earlier tests had given the simulated party a tidy set of orders and full supplies. The party in the story had whatever it had earned so far. It might have some training it hadn’t spent and no extra equipment from shops. Playing with that party gave the fights more useful stakes.
The game ended up with different rules for how fights are tuned and when characters help themselves to a dressing for an injury. The project checked several kinds of play. Someone might stop giving orders and let a fight unfold, get distracted and give the next order late, or arrive prepared with better gear. Those styles tell you different things. A fight that works once can still leave a group with too few dressings to survive the next two.
There was a similarly ordinary test for saving. The project tried continuing with one damaged save, tried writing with a full memory card in the emulator, and tried saving again after the game refused. For someone playing a campaign at home, a clean response to a full card is part of whether the game feels finished. It’s the kind of thing I wouldn’t have thought to ask for when I wrote the first prompt.
How much it told me
I could see the AI was able to check a lot on its own. The most convincing reports gave me a problem I could picture. A guide and a companion stood waiting. One conversation left the player hanging. A party ran out of supplies after a couple of fights. Those are the details that turned an impressive test result into something I could use to decide what to do next. I could follow them without reading the campaign.
There’s still a part of the experiment I’ll only get by sitting down with the game myself. I want to notice which people I want to talk to again and whether I care enough about the mystery to keep going. I’ll find that out when the game looks and feels ready.
Read the making-of or continue with the retail-game study.