author avatar
By Dan Reale Senior Quality Assurance Engineer

*Views, thoughts, and opinions expressed in this post belong solely to the author, and not necessarily to SemanticBits.

Wouldn’t it be awesome if you could automatically and accurately enter your timesheet everyday? I know it would save me at least a minute or two, and since I always forget, it would spare me a reprimanding email from my supervisor. If you are wondering, yes, I always forget to enter my timesheet, even though Binja, our timesheet software, automatically reminds me twice a day. So, how can we accomplish automating our timesheet entries?

This question doesn’t stray far from how we can approach automatically testing an application. Our goal is to write automated UI tests that evaluate the functionality of our app in an end-to-end fashion, the same way a user would use it. If we do this correctly, we should be able to find any issues and identify any bugs before deployments to production.  It also increases our confidence that our software is working 100%. Now, let’s take a closer look into how we can do this with Cypress, a rival to Selenium, the most mature UI automation tool out there.

What is Cypress?

Cypress is a tool that helps developers and QA folks test their application. With a very easy set up, just run npm install cypress --save-dev, and Cypress is installed and ready to use. If you’re not a developer, you might be asking, “What the heck did he just write?” npm install cypress --save-dev? What does that mean? Well, you don’t need to be a seasoned developer, but you do need a basic understanding of JavaScript, which is the only language it currently supports. Under the hood, Cypress uses Mocha, Chai, and Sinon, which are three of the most popular tools for JavaScript. The best part is you don’t have to maintain the dependencies of these tools. Cypress maintains everything for you. This allows you to do what you’re supposed to as QA engineer and developers, to focus on writing test cases to evaluate the application, and not have to worry about creating and fixing a homegrown framework.

How does Cypress differ from Selenium?

Selenium is a powerful tool and in my opinion, much more mature when it comes to UI testing. Cypress is still in its infancy, although relatively mature at the current time of this writing. Cypress runs as part of your application directly in the browser, whereas selenium works over a network. This is a completely different approach, which provides the following advantages that I find to outweigh the disadvantages of this tool.

  • Unit testing
  • Integration testing
  • Stub/Mocks

These three features are (to me) the the reason why Cypress is the tool that does complete end-to-end testing. What truly sets Cypress apart from Selenium is that it allows us to monitor the application for any API calls that are happening. For instance, I just submitted my entry for today’s timesheet. When I did that, there was an API call behind the scenes that posted my entry to the server and validated that the entry saved. With Cypress, we can test adding an item to our timesheet for the day, but we can also test to see that the API call to save our timesheet is successful or not based on our expectations. We expect that when we submit our timesheet, we will see a POST request. We also expect a specific response. To test this fully end-to-end, we use Cypress. If we are developers and don’t want to test against real data, we can use fixtures to set our expected responses. We can’t do all of this with Selenium.

But wait. You don’t trust the tool actually passes the tests that it says it passes? Cypress has an awesome UI that you can use to debug tests, see snapshots of each action that you perform in your test, see valuable, readable error messages, and also save videos of your test runs. Yes, you can watch your tests run through. You can even save them to the cloud. Selenium can handle screenshots but that’s about it. Debugging tests is a pain, you can’t record videos, and you usually have no idea what the errors mean (unless you are a seasoned developer).

Cypress automatically waits for the page to load, and never has stale elements, which two of the biggest issues QA engineer and developers experience when using Selenium. I found this statement on their website hard to believe, but after testing the web interface, it really does hold up to its claims. Not once have I had an issue waiting for a page to load or dealt with stale elements in the DOM within our application. During the following Binja example, I ran into a snag with waiting, but since this is not the application that we are focused on, I will give Cypress a pass.

***The following video was created by running Cypress via command line. It ran a headless Chrome browser and saved the test run.***

Limitations

While Cypress does a really nice job of giving developers and QA engineer things that they really want in an automation tool, it does have some limitations. It doesn’t support cross browser testing. Sorry, all you Internet Explorer fanatics,this tool is not for you. It only supports Chrome at the moment, and Firefox, if used with Docker. It can also run tests in Electron, and headless in Chrome. The only supported language is JavaScript. Selenium offers a wide variety of language implementations, including JavaScript, Java, Ruby, C#, PHP, Python, just to name a few. This might shy some people away from the tool if they aren’t familiar with JavaScript or don’t care to learn it. It also does not support handling tab switching. For us this is not a huge issue, but for other teams or applications, this could be a deal breaker.

Challenges

Some challenges I have come across while testing the web interface are mainly around downloading and uploading files to our application. Currently, I am working on a way to fully test some of these features, but must say it is not very intuitive. I plan to create some external scripts that will be run during the Cypress test to check for files on the system after the download happened. This actually is another advantage of Cypress. I can just run cy.exec(‘npm run checkForDownloadedFile’). This runs a script called checkForDownloadedFile that I declare in my package.json. I plan to use similar functions for cleaning up data in the database.

Choosing Cypress

We eventually chose to use Cypress for all the reasons I list above but how we came to using it was almost happenstance. I joined SemanticBits about three months ago, knowing that I needed to standup test automation for my team. Naturally, I thought to myself, “Ok, I am going to use Selenium. Let me just see what languages the development team uses for the application, learn that language, and start building the framework.” I started by doing some API automation as the UI was secondary. However, our QA team had a revelation that UI testing was deemed more important than API testing.

So, I shifted gears and started looking into building a framework. I built the barebones of a Selenium solution in JavaScript but, being that I am not a JavaScript expert, I was a little weary that what I built might not hold up and be scalable. One of the tech leads mentioned that they used Cypress for unit testing and I should check it out. I realized early on that the tool really favored developers and unit testing, but that it could do more than that and actually perform end-to-end testing, while doing unit testing and integration testing at the same time.

To me, this was an ultimate tool. I had some exposure to JavaScript and tools like Mocha and Chai. In fact, I am building our API tests using those tools. So, in that respect, Cypress was an easy choice. Cypress also can be run in Jenkins and Docker, which are other tools our team uses. So, that was another positive. I could tell that after doing a POC, the time it would take me to get a test built and ready to use would be far less than if we used Selenium.

The only negative for me was that Cypress didn’t test all browsers. This was something that was brought up as a major concern. Currently 50% of our users are using chrome, and 25% are using some versions of Internet Explorer. The other 25% are a combination of Firefox, Safari, and Opera. Our thinking was that if our tool can cover half of our users, then that is far better than anything we had before in terms of automation, because there was no automation. Right now, we are all aboard on Cypress.