The Effect of AI in Testing and Test Automation

It can also act as the most confident co-worker you ever had and reply with information that is not correct. It will always respond, but not always correctly. It helps to understand the code well enough to determine whether the response was correct.

It helps people writing code to spend less time on Stack Overflow to find the answers they need. Claude, ChatGPT and Google Gemini are all good for coding-related tasks. The code might not work on the first try, but it’s often a good starting point.

What’s the Effect on Testing?

AI can help a lot in writing tests, but an important thing to understand is that testing is much more than writing code. You’ll need to understand and clarify the requirements with the stakeholders. You’ll need to understand the risks with the feature and what could break. Once you have gathered this knowledge, you need to plan the tests to cover the requirements, where each requirement may have multiple tests.

At this point, you are almost ready to write the test code. Before doing that, you’ll have to use and explore the feature to understand it better. This can involve investigating the elements on a webpage or the structure of API requests or responses. You may also run into apparent problems at this point before writing any automation code.

How much coding is required for each test depends on how significant the change is and whether it’s an entirely new feature or an addition to an existing feature. Sometimes, the only thing needed is to update an assertion. In other cases, you may need to mock some endpoints, set up test data and write something completely new.

Writing Code is Easier

Once we write the code, AI will be helpful in most cases. You can ask about the syntax, how to create random numbers, write selectors and many other useful things related to the test case. I often ask about the approach for a specific thing, especially if I’m unfamiliar with the subject.
This could be about configuring tests when writing Playwright tests in Java or connecting to a database in Kotlin. Or how to configure Playwright tests when using Python, as JS/TS users are spoiled by having an actual configuration file.

To summarise, I recommend checking the approach first and only then going into the details of a specific test case or a task.

Programming Language Requirements

AI can be very helpful when you know what you need to do but are unsure how to do it in a specific language. If you’ve done API testing in one language, the test cases and expectations will be very similar in another language. If you have previously written API tests in Python and need them in TypeScript or Ruby, it will be easy to figure them out in the new language.

Access to AI will make the transferrable testing skills more critical and allow you to apply the same knowledge with different implementation details. This will also require changes in recruitment, as many roles are advertised requiring experience in a specific language or tool. Especially for permanent roles, the new employee should be able to figure them out quickly.

Making Mistakes is Also Easy

As mentioned earlier, the AI will always have a response, whether it knows about the topic or not, so you must double-check it. One way of doing that is by running the tests and, as always, checking that the test also fails when expected.

Depending on the project and the developers, using AI can increase the risk of adding poor-quality code to the application when used irresponsibly. This may increase the need for testing.

Do we Need Less QA and Testing?

There seems to be a recent tendency to hire fewer QA people, although I don’t have stats to prove it. There are plenty of jokes about testers testing funny edge cases. If AI is making writing application code more efficient, does that leave time for the developer to do the same? Are they also going to do more exploratory testing? In theory, maybe, in practice, these are two different roles.

There’s a different mindset where one wants to see a feature working, and the other wants to break it. If anything, there’s a need for more testing now, as developers can do more things in the same amount of time, and testing should not be a bottleneck in the process. If the devs are cranking out twice the number of stories, you’ll need more resources to test them or write test automation for the stories during the sprint.

Summary

This post contains some thoughts I have about using AI for testing and implementing automated tests. Feel free to ping me on LinkedIn if you want to discuss further.