Cypress vs WebDriverIO

There has been a lot of press commenting on the benefits of Cypress vs WebDriverIO. At the end of 2019, we launched our Vitaq AI Test Automation tool at the Selenium Conference in London. This was our first proper foray into the world of web and mobile app testing. For several years we had steadily built up a good run-rate business in the embedded software test automation space with our Vitaq tool.

Where we came from

Our biggest claim to fame was that between 30-40 million cars, out of the 85 million cars coming off global production lines each year, were tested by our tool. Vitaq had become the leading AutoSAR (the operating system software in automotive control systems) test automation solution. We also had many electronic systems companies using Vitaq for a variety of applications. From testing video broadcast codecs to aerospace control systems to wireless network protocol stacks.

Our small self-funded company had made a name for itself in the mission and safety critical testing space. Our unique approach was finding software defects that other approaches missed. We had many tens of companies and many hundreds of users but we just could not get it to grow to hundreds of companies and thousands of users.

Each time we applied the tool to a different application problem, it turned into a custom software engineering task. Then we discovered the mobile and web application testing market thanks to King the developers of Candy Crush. They were making $4.2 Million a day at the time. So failure was not an option. They wanted to apply Vitaq to a very complex application development project.

One ring to rule them all

That was our Eureka moment! We discovered the beauty of WebDriver. Exactly what was lacking in the embedded electronics systems market. A world-wide web consortium standard interface to provide a platform and language-neutral wire protocol. It has become the industry standard way for out-of-process programs to remotely instruct the behavior of web browsers. Better still, Appium WebDriver takes this global standard and extends it for mobile applications. In the darkest depths of our non-scaling business despair we had stumbled across the one ring to rule them all (my precious;).

Late to the party

Over the last several years there has been an explosion of Test Automation companies coming to the web and mobile app market. Are we too late to the party? What we had noticed is the vast majority of tools automate the running of manually created tests. Vitaq AI is focused on auto-generating the end-to-end functional tests themselves.

The only other tool that we see doing something similar to us today is Eggplant DAI.  One of our lead technical people left us to join them as we started to engage the web and mobile test automation space. With the tens of millions of dollars invested by their Private Equity backers, they got into the market 10x quicker than us. But to get payback for their corporate backers, they have to charge 10x more than us. Following their acquisition by Keysight last year (a huge US corporate) they do seem to be losing a lot of people including their original CTO, VP engineering and the person that left us to join them!

Standing on the shoulders of open source giants

To really deliver on the opportunity we knew we needed to stand-out from the crowd. So just supporting Selenium and Appium would not be enough. We would need to provide a complete solution to both QA and Dev testers. Our initial teaching customers informed us on this, following our launch at the Selenium Conference in London at the end of 2019. They asked us to stand on the shoulders of a one of the “next-generation” test automation frameworks. The big question was which one should we build on first and the choice was between Cypress vs WebDriverIO.

Cypress has over twice the number of daily downloads and has a great on-boarding experience. But it cannot support native mobile apps and because it runs in the browser using a “pre-primed” test lag, it’s architecture has fundamental limitations. It cannot support multiple tabs for example. With the imminent release of Selenium 4.0, the advantages of Cypress are matched by the brand new Chrome DevTools Protocol API. This enables Selenium tests to not only mock server requests and intercept responses, but to fully leverage all of the possibilities of Chrome DevTools. This is a game-changing feature for Selenium WebDriver!

To sync or async? Cypress vs WebDriverIO, That is the question!

The “biggy” for us in making the decision between Cypress vs WebDriverIO is that Cypress does not support sync mode. In JavaScript asynchronous operations are handled via async/await. Multiple social media polls show that new users find the JavaScript concept of promises one of the most difficult to get their head around. It also makes writing tests very verbose, as almost every operation is asynchronous. It is very easy to inadvertently miss nested awaits, which can be very difficult to debug. With @wdio/sync you can avoid awaiting for command calls so your test code is much easier to write and debug. 

Conclusion

Don’t get us wrong, we love Cypress but had to make a decision on where to apply our limited development resources first. Taking into account the architectural limitations of Cypress for web testing and  the lack of native mobile app testing. Then pitch all that against the superb progress that webDriverIO 7.0, Selenium 4.0 and Appium have made. The webDriverIO decision was easy to make. We have found working with the excellent team at webDriverIO an absolute joy and completed the additions to the framework we needed a few weeks ago. https://webdriver.io/blog/2021/03/23/grouping-specs/

Leave a Reply