How our QA team uses automated testing to deliver stable version releases
Blackthorn Team
February 25, 2022
February 25, 2022
Quality assurance (QA) is a way of reducing bugs in products and avoiding problems when delivering solutions or services to customers. With frequent updates comes frequent testing, and we’re no stranger to the concept. Unlike other solutions who update their apps quarterly or sometimes bi-annually, Blackthorn releases monthly updates to our Events and Payments AppExchange solutions. As our apps grow, so does the need for efficiencies. Testing our new features for bugs or errors is an ongoing process, and automation greatly increases efficiency.
Why automated testing? Manual testing is time-consuming and regression based (existing functionality) bugs can be missed while manually testing. This benefits by avoiding hours on testing small errors that a system can easily catch.
These tests usually don’t take long to make and target actual use cases! We target how end-users use the system, and how our own team would use the apps we provide.
The automated system our team chose to use is Uilicious. We explored other options before deciding what route to go. The first was Selenium with a Salesforce web driver, and the second was Provar. Selenium is an older tool used for testing, so we weren’t certain on capabilities moving forward. On top of that, Selenium isn’t user-friendly for anyone to pick it up and run with it. Provar was codeless and seemed very easy for anyone to pick up and use, we strongly considered this option. However, due to the lack of communication and the price tag (a steep $23,000 starting – for one user), we decided to pass.
Uilicious is derived from Selenium and customized heavily by their development team to be simple and easy to use. It is a happy medium between code and codeless automation and has a free version. We opted for the “Studio Basic,” and our package is $90 per month. The customer support is very helpful and quite responsive for our time zone, being based in Singapore while we are in the United States. The documentation they provide is excellent and covers plenty of scenarios many people would want to test. Some undocumented features are being worked on, and if you ask for them, they will gladly provide you with the resources that are developed so far.
Glad you asked! In our ecosystem, we push updates every month. Given the size of our team, we update automation as frequently as possible, but sometimes releases get pushed back. This automation is built in terms of scalability. Manually testing one or two features isn’t too much of a feat, however, testing entire packages after a release, hotfix, etc. is time consuming to test manually. Automation can be intimidating when diving in, but investing the initial time pays off ten fold in the long run.
Blackthorn manages automated testing in two different ways.
Uilicious allows for end-user testing. If a flag is thrown for an error, we can investigate further and fix the issue. This allows us to test many scenarios that our end users would do. This can include any…
Uilicious isn’t all sunshine and rainbows, though. Every good piece of software always has the desire for more. The below list of items we have struggled with as a Salesforce app company testing in Salesforce:
Below is an image of the start to creating a new test; they give a simple setup guide here, with plenty more documentation listed here.
Part of the way through, it starts looking something like this…
By the end of testing, the lines start to add up quickly. This is the total from one of our tests that end up being ~1200 lines long. This test is currently a work in progress, testing everything from start to about 80% of full testing.This script is the start of simply going to the Salesforce sign up page, creating an account, and doing our fresh install testing. A brand-new Salesforce developer account is made each time we release a package to run through testing. This allows our team to work on other tasks, while testing the package at the same time.
Uilicious took some time to implement for our use case but was well worth it. We also use it for checking if any of our features are down, this includes DocumentLink and downloading the documents from the web page, PayLink, and our Events app. If the system detects a page that can’t be rendered, download links are working, etc. it flags the test, takes screenshots of what happened, and automatically emails our team to let us know there are issues with one of our services.
We test both our Payments and Events app in Uilicious – below are a couple of video examples.
The first, Payments. In this example, we test our custom Visualforce Component, Virtual Terminal. This was exceptionally important for us to constantly be testing, as it’s a very powerful feature that many of our customers use. For example, a test would look like this.
Secondly, Events. For an example, this test…
At the end of the test, the system checks the ERS and Sales Document to ensure the customer was recorded and charged appropriately.
The second form of automation we use is a tool called Cypress. On top of Cypress, we use a behavior-driven development framework called Cucumber.
Sounds a lot like Uilicious, huh? Here’s the difference – Cypress is a very short path for the development team to raise confidence on every release by simply ensuring that our app works consistently. On the other hand, Uilicious allows us to have a fully integrated end to end testing (between Salesforce, and the UI we build for our web application), which we cannot do with our cypress tests. Uilicious tests also act as independent validation to work on the platform side and provide a filler for tests we cannot perform with Cypress such as testing on Internet Explorer and other browsers not supported by Cypress.