I have been running coding agents on my own projects for long enough that a management question has crept in. How do I know whether an AI employee is doing a good job?
With a human employee you have a lot of cheap signals. You see them at their desk. You hear how they talk about a problem in standup. You notice when they go quiet for three days. None of that is available with an agent. It runs in the background. It comes back with a summary. And the summary is written in the same confident register whether the work is finished, half finished, or quietly wrong.
That last part is the whole problem. The report is not evidence. So I stopped grading the report.
Grade the artifact, not the story
The only thing an agent leaves behind that cannot lie is the state of the repository. Tests pass or they don’t. The build is green or it isn’t. The diff touches three files or thirty.
So my first rule is that every evaluation starts from the codebase, not from the chat. Before I read what the agent says it did, I look at what changed. If the two disagree, the code wins, and the agent gets a lower mark for the gap. A confident summary that overstates the work is worse than a hesitant one that undersells it, because the first one costs me a review I didn’t know I needed.
Make the agent keep the task board
The second signal is the task list. Not mine. Its own.
When I give an agent a piece of work, I ask it to break the work into tasks and keep that list updated as it goes. This does two things. It tells me where the agent thinks it is, which is often different from where it actually is. And it gives me a unit to measure. A task that has been “in progress” for four rounds without a commit is the agent equivalent of an employee who has gone quiet. I go and look.
The board also makes the work fair to judge. Lines of code is a terrible metric for humans and a worse one for models, which will happily generate a thousand lines of boilerplate. Tasks closed against tasks planned is at least measuring the thing I asked for.
Watch the ratio of explanation to change
Here is a signal I did not expect to be useful. Compare how much the agent writes about the work to how much of the work it actually did.
A healthy ratio is small. A short note, a real diff. When the ratio flips, when I get three paragraphs of reasoning and a two-line change, something is usually stuck. Either the task was underspecified and the agent is filling the gap with prose, or it hit a wall and is narrating around it. Either way it needs a human, and the ratio told me before the agent did.
Give one task forty hours
Agents are fast, which tempts you to spread them thin. Ten tasks, ten agents, everything moving at once. What you get is ten things at thirty percent.
I have had better results doing the opposite. Pick one task that matters and make sure it gets the equivalent of a full working week of attention before I judge it. Rerun it. Review it. Send it back with the review. The point is not that the agent needs the time. The point is that I need a long enough window to see whether the work converges or keeps wandering. Convergence is the real performance signal, and you can’t see it in a single run.
The remote employee you never meet
None of this is new management theory. It is how you would evaluate a remote contractor you never see on video: by the artifacts they deliver, by the task board they keep, by whether their reports match reality, and by whether their work gets better when you give feedback.
The only difference is that with a person, you eventually build trust and stop checking. With an agent, I don’t think that day comes. The checking is the job. The good news is that checking is cheap when you check the right things, and the right things are all sitting in the repo.