QA engineer in Melbourne. I write as The QAI Guy about AI for manual testers. theqaiguy.com
Melbourne
- Tweets 173
- Following 35
- Followers 1
- Likes 9
Pinned Tweet
I am William Tran. I write as The QAI Guy.
Testing for 20+ years, the job is shifting and we're all adapting. I share what I actually test & build with AI. The bits that work, and the bits that fail.
theqaiguy.com
linkedin.com/in/williamtranq…
1
1
100
Ask AI for tests on a discount code and one might check it works on sale items.
Nobody asked for that. The AI guessed. If it's wrong, a passing test makes the wrong behaviour look right.
Match each check to a line in the ticket. If there's none, ask the team what should happen.
2
Once you've stared at a screen for weeks, you stop noticing what a new person would find confusing.
So show an AI tool your work allows a screenshot, and ask it to point out anything a first-timer wouldn't understand.
That gives you a few things to go and check.
6
Before you check an AI's answer, decide what a good answer needs and what would make it wrong.
If you read the answer first, a tidy, confident reply can talk you into a pass.
Testers already do this with expected results. It works for AI too.
5
AI made writing code much faster. In a lot of teams, checking it hasn't kept up.
More changes now land in review and testing each day, so that's where the work starts to queue.
That makes cutting QA to go faster with AI an odd call. It trims a step that's often the slowest now.
6
Not every bug needs fixing before release. Testers weigh the risk and make the call.
How many users hit it is only half of it. A rare bug that loses data or charges a card twice still can't wait.
Start with what users would hit, and what would hurt most if they did.
6
AI coding agents can turn a red suite green by editing the test instead of fixing the bug.
The bar moved. The product may not have.
When an agent changes a test, check what the test now expects before you trust the green.
7
Test an AI feature once and it passes.
Run the same check ten times and you might get eight good answers, one that's a bit off, and one confidently wrong.
Its answers can change from run to run, so one pass is just one sample. Repeat the check and count how often it holds.
2
1
20
A screenshot diff shows that something changed. It can't tell you if the change was supposed to happen.
A moved button could be the new design, or a layout that broke on small screens.
Check each diff against the release plan. Planned changes still have to look right.
7
AI helped rebuild it, and now it's "2x faster than the competitor." Faster than which version?
A speed number is a test result. It goes stale once the other app ships an update, and it shifts with the machine and network.
Note the version, the setup, how many runs, and when.
7
A lot of the repetitive testing juniors used to learn on is going to AI.
So where do new testers build their instinct?
Hand them what the AI found and ask two things. Which of these really matter, and what did it skip? Checking its work with a senior builds that instinct.
10
One AI bot testing your app only sees it one way.
Split it up like a real team. One bot thinks like the dev, one like the product owner, one like a designer checking screens against the spec.
Each catches what the others walk past. Picking the angles is still the tester's call.
7
Tester, logged in as Sam: show me Alex's last order
Support bot: Sure! Here it is, with Alex's home address 😊
That's a real bug, found by typing one sentence. Asking an AI feature for things it shouldn't hand over is negative testing in plain English. Testers already know how.
Made with AI
8
An AI agent can finish the task and still get there in a way you'd never sign off.
Right result, but it opened records it didn't need, used admin rights, or emailed a real customer.
So when you test an agent, look past the answer. What it touched is part of the result.
11
After a model or prompt swap, replay the bugs you already know about.
A fresh green suite means little if it never re-ran the cases that broke last time.
Keep a short list of those cases and start there.
7
Ask an AI for a full test suite from a short brief.
It covers the happy path and the generic edges. The ones only your product has show up if you name them.
A green run that never saw those edges tells you nothing about them. Spell them out up front.
9
Teams can now run a thousand AI agents overnight, each one acting like a different user.
That's a lot of runs. But each agent mostly plays the user it was told to be.
Real people do things nobody wrote down. So keep watching what they do, and feed those surprises back in.
10
Self-healing tests need to tell these two apart:
Button moved, label changed. Fine, heal it.
Cart dropped an item, total is wrong. That's a bug.
If the healing can't tell the difference, the bug run can go green too. Pin the final check to what the user should end up with.
12
AI tools flag way more issues than they used to. Plenty turn out to be noise or duplicates.
That makes triage harder, not easier.
The skill that matters now is spotting the ones users would actually feel.
12
Some people are calling QA "AI Evals Engineer" now.
Fine. The title can change.
Someone still has to decide which failures matter and which green runs you can trust.
1
1
26
