We recently released our first beta build to our client’s QA. Their Software Development Product Manager knows her failure cases inside out and quickly compiled eight tricky bugs. We wondered how the Agent would handle them.
Gotta start somewhere
We took her bugs as a list of platforms and what she found. We wrote (I told Copilot to write) a skill to generate a bug report for each line of that input file and populate it with a template. I wanted to know
- What was the problem
- What was the platform
- What are the reproduction steps
- Can we write a test for this
- What are the relevant requirements
- What’s the cause
- What’s the fix
Time for actual work
I (had copilot) created an agent to:
- verify the behavior on the reported platform
- write a test if possible to isolate the behavior
- document any applicable requirements
- determine a cause
- apply a fix if determinable
- determine if the bug applies to any unreported platforms
Then I set it free. I had the agent work on one bug report at a time until they were all resolved. It was the quickest I’d ever churned through a list of bugs. And a lot less stress.
Results
Of the eight bugs, Copilot fixed seven. The eighth required more information, as it may have been a misunderstood coincidence with our crash reporting system. The others had unit tests, reasoning, and a fix. The bug report had all the information we asked for, and the tests passed.
