What Does the New AI Model Do for You? Test It on Your Own Work
The useful way to test a new AI model is against your own work, in two parts. First, re-run a few things you already do and compare the output with what you produced before. Second, try two or three things you couldn’t do before or gave up on. Write a one-line standard for each item before you look at the output, and finish with a short note covering what the model holds, what it adds, what you’ll change and what you still check yourself. For a project manager, that note is the benchmark.
A Headline Score Doesn’t Tell You What Changes in Your Week
Claude Opus 5.5 and OpenAI’s GPT-6 Sol and Luna launched on 22 September, and Claude Sonnet 5.5 six days later. Each arrived with benchmark scores.
Every covered the problem on 10 September in “Evals for Everyone”. It noted that Astra and Fable 5 scored 96 and 93 per cent on a test of graduate-level science questions, and that public benchmarks leave it unclear how those scores translate to how well a model helps you do your job. Every’s answer is a personal benchmark, graded against your own standard for what good looks like. Its full method sits behind a paywall, so the kit below is our own design for project managers.
There’s a second reason to look past the score. You probably didn’t choose the model, because your organisation did. What you can act on is what the new one does for your work, and how you’ll use it.
Part One: Re-Run Work You Already Do
Pull three or four pieces of work you were satisfied with, along with the source notes each was written from: a status report, a risk narrative, a stakeholder email. Redact client names and commercially sensitive detail before anything goes near a model, and check your organisation’s policy for the tool you’re testing with.
Write the standard for each item in a line or two, taken from corrections you already make. If you always move the decision you need to the top, that’s a line in the standard.
Then give the new model the same notes and the same instruction, twice, since one run tells you little. Put its output beside the version you sent and describe the difference in words. The entries here and below are illustrative.
Item: Friday status report, permit approval three weeks late
Standard: The slip is named in the first line. The new expected date is stated with who confirmed it. One decision is asked of the sponsor. Under 150 words. No comment on the regulator.
Before: Needed two edits. The decision had to be moved to the top and a sentence about the regulator cut.
New model: Put the decision first unprompted. Commented on the regulator in one run of two.
Finding: One fewer edit. The regulator line stays on the checklist.
Part Two: Test a New AI Model on Work You Gave Up On
Current work only shows whether the model holds its ground. What it adds shows up on tasks you tried once and abandoned, or never tried because checking the output would have taken longer than doing the job. Checking a 40-slide steerco pack against the RAID log for contradictions. Reading a full schedule export and listing every task with no successor. Turning one set of notes into a change request for three audiences.
Pick two or three, and write the standard as what would make the result usable.
Item: Check the steerco pack against the RAID log
Standard: Every mismatch cites a slide and a RAID row. No invented entries. Each one can be verified in under a minute.
Before: Unusable. Citations pointed to rows that didn’t exist.
New model: Eleven mismatches, each with a slide and a row. Nine confirmed, two were wording differences.
Finding: Usable with a check of each citation.
Write Up What the Model Holds, Adds and Changes
Finish with a note of four short parts. Here’s one written from the two items above.
Holds: Status reports are as good as before, with one fewer edit.
Adds: The pack-against-RAID check is now usable.
Changes: Run that check the night before each steerco, starting this month.
Still check: Any comment on a regulator or supplier, and every citation before acting on it.
Now write your own from the items you ran. The note is what you bring to a sponsor or a team asking whether the new model is worth using, because it says what the model does for this project.
Keep the Kit for the Next Release
Save the items, the standards and the note together. When the next model arrives, the test takes an hour, and anything that slipped shows up in the first part. If you’ve built skills, as in AI Skills for Project Managers: Moving Beyond Prompts, give each one a few items of its own and re-run them the same way.
A handful of items and one grader is a small sample. It tells you about your work, and it won’t settle a procurement decision on its own.
Which task did you give up on with the last model that you’d put in front of the new one first?
Frequently Asked Questions
How do I test a new AI model for project management work? Re-run a few pieces of your own past work through it and compare the output with what you produced before, then try two or three tasks that weren’t usable before. Write the standard for each item before you look at the output.
What should I test beyond the tasks I already use AI for? Tasks you abandoned or never tried because the output wasn’t worth checking, such as comparing a whole steerco pack against the RAID log. Those show what a new model adds.
Do I need to score the outputs? A score is optional. A written comparison against a standard you set in advance tells you what to change in how you work.
Which task did you give up on with the last model that you’d put in front of the new one first?
Yes, AI helped me to write this :)
Unsubscribe
