QA / Testing work being done by a team of clankers is an amazing time saver.
I’m doing something similar with LCOS. A small team of bots running nearly round-the-clock tests on both individual components and the system as a whole. Code reviews, scripted tests, UI-driven tests… the works.
Bots find the issues, a human evaluates the issues and decides what to fix (or not, with guidance), the bots fix the issues… rinse, repeat.
Not only is the velocity of testing & development astronomically fast… but the quality is exceptionally high.
This process doesn’t remove the need for human, hands-on testing… but the benefits are tremendous.
Big Omarchy QA News:
The Headline:
* Omarchy QA will start running against nightly every day. Still some huge hurdles to figure out, but that is the fun stuff.
The Meat:
Omarchy QA is going slower than I hoped for, but its getting into place. This is my second fully running agent test harness where they decide what is good.
* A nice running dashboard with cancellable states
* Jobs, suites, servers: have all the possible states and transitions you would expect from build servers. This part was the most surprising as it is really in depth how much effort you have to spend thinking about every possibility
* suite states, live updating website that ... runs fast? Yes I did something crazy... I made a cache, i know, actual crime...
** I even have an idea to make this load instant for every single user, and make updating cache contents LIGHTNING fast...
*** Fun side note. WTF ARE YOU DOING EVERYONE, WEBSITES ARE FUN TO MAKE FAST AND YOU ALL ARE MAKING HORESHIT SITES... STOP.
* ability to follow live running jobs and cancel them
The sad part is I have had no time to get the actual custom harness running fast, which is ok, but its hard. Now, starting today, I am going to be hooking up nightly builds and actually running this against production!
Will think about how to make this a public site, but for now, its private :(