There are silly things that trip up e2e tests like a cookie pop up or network failures and whatnot. An AI can plow through these in a way that a purely coded test can’t.
Those types of transient issues aren’t something that you would want to fail a test for given it still would let the human get the job done if it happened in the field.
This seems like the most useful part of adding AI to e2e tests. The world is not deterministic, which AI handles well.
I predict an all out war over deterministic vs non-deterministic testing, or at least a new buzzword for fuzzy testing. Product people understand that a cookie banner "shouldn't" prevent the test from passing, but an engineer would entirely disagree (see the rest of the convos below).
Engineers struggle with non-deterministic output. It removes the control and "truth" that engineering is founded upon. It's going to take a lot of work (or again, a toung-in-cheek buzzword like "chaos testing") to get engineers to accept the non-deterministic behavior.
Those types of transient issues aren’t something that you would want to fail a test for given it still would let the human get the job done if it happened in the field.
This seems like the most useful part of adding AI to e2e tests. The world is not deterministic, which AI handles well.
Uber takes this approach here: https://www.uber.com/blog/generative-ai-for-high-quality-mob...