Hacker News new | past | comments | ask | show | jobs | submit login

Let’s get to brass tacks... does it still have any possibility of flaky tests? If so, I’m out.



One can still write tests that are flaky - some of the challenges that Selenium et al. experience are still present:

1. Selectors still need to be specific enough `.button` is great until you add another button.

2. Selectors can be _too_ specific in weird ways. Recently, I encountered a bug where reformatting my markup broke a test that detected if an element was present by checking for its contained text. Turns out the newlines that I inserted between the opening and closing tags were significant for the test. Thankfully, it's all JS, so one can appropriate work around.

3. For SPAs, it's harder to convey that a page/view load has completed. So, if I click a button and the entire viewport updates (potentially downloading a JS bundle in tandem), I either need to insert timeouts of arbitrary lengths, or explicitly call out to the the provided network API hooks. This can be annoying because now I not only need to worry about what I _can_ see when I navigate through my app, but also what I can't.

It's far better than what has come before, but it's not a silver bullet.


It depends what is the root cause of your "flakiness." Sometimes it's due to the browser not painting your elements in time because of various layout thrashing other times it could be because of data you're waiting on before you can render a table.

If your flakiness is due to items like layout thrashing, or triggering other HTML re-rendering like painting or compositing, IME you'll have issues with any test runner until you fix the problems of poor browser performance and layout thrashing

If your tests are flaky due to backend calls being inconsistent or taking a long time, Cypress allows you the ability to intercept network requests [1] and return "fixtures" of what you want the data to be. The benefit of this is you can go directly to any route for your web app then just stub the data you need immediately and test against that. The benefit of this approach if you can easily test negative scenarios and how your application behaves. Negative scenarios like failing response (500 error) or having a 200 response with missing data fields. This is useful when testing something like pagination because you can make the "3rd page" fail when you trigger that API call by forcing it to give you a 500 error (or something else). The downside of this approach is that you are no longer using real services that your app relies on, but you gain some confidence on testing multiple scenarios in an easy format rather than setting up proper chaos engineering scenarios/experiments.

Other testing libraries do allow you to intercept requests as well (I know playwright does this), but one of the selling points of cypress is that you have access to the same browser APIs that you have use when doing web development.

[1] https://docs.cypress.io/guides/guides/network-requests.html#...


when we used Cypress, we found the framework itself to be flaky.

it has been months, and I can't remember the actual error we got intermittently, but I do remember it was something buried inside cypress (I think it was related to taking screenshots?) that caused our tests to just randomly fail sometimes.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: