How we Replaced Four Manual Testers with one Library

Introduction

When developing an application consisting of multiple modules that interact with each other, the entire final design of the product can be decided in a flash. From overlapping texts to weird-looking pictures, where you have 10px wide on the left and 30px wide on the other.

Such an application can lead not only to a loss of customer confidence in the application, but also to a greater number of errors in the order.

We may even encounter the disappearance of the "submit order" button or similar missteps leading to a drop in sales and damage to the client's reputation. This must of course be prevented...

Project Objective

We have been tasked with providing long-term solutions to the issues with visual testing of web applications that customers encounter. At the same time, we should have developed a universal library that could be used independently of the project. As a result, we should have been able to visually test any project fully automatically and in a few moments.

Starting Position

The customer developed a web application for its resellers consisting of several modules. He held 4 manual testers, each having repeated about 100-200 same tests after a release, measuring the motion of elements by pixels. To do this, they used a virtual ruler and compared whether the position of the elements agreed with the documentation. This tedious and boring job often took them all of a week - that is, half the sprint time!

Setting Expectations

At the very beginning, we had to define what we expect from the library, what we can do by the deadline and what has the highest priority. The following points emerged from the analysis of tests, applications and the work of manual testers:

1. The library can detect if there has been any movement on the page based on pre-created screens
2. The library will be fully compatible with Robot Framework (due to customer's wishes), but ideally it can work independently of it
3. The library must be able not only to recognize differences in images, but also to ignore certain areas that can change dynamically (for example, ads or various sliders on the page)
4. The library will be able to produce screenshots in various resolutions and also cut-outs from them
5. Great emphasis must be placed not only on the speed of development, but also, if necessary, on modifications from the application developers themselves

Next Steps

Thanks to the analysis described above, we could afford to choose Python as the main language, which offered us several advantages: it is  simple, clear, fast to develop and, in addition, the customer's developers were already able to work in it. Therefore, they should be able to adjust the library for the future needs of the project themselves without anyone's help.

The development itself was then a matter of days. A method for comparing two images has been programmed as the first and most important function. The basic principle is to use OpenCV library to convert the original screen of a web application and the current screen to a matrix of one and one zeroes, which are then compared and return the value. In the following few days, we implemented functions for creating screens (including viewports from the image according to input parameters) and for ignoring areas according to inserted arguments.

At the very end of the development, there was then communication directly with RF - not only error messages had to be resolved, but also inserting images with highlighted error locations. This was followed by the solution to our most pressing problem - the installation of the library itself, which contains a lot of supporting libraries.

In the end, creating a pip directory and sourcing code to GitHub turned out to be the best approach.. As a result, we ensured fast distribution of the library among developers, including updating the library. Now we can only pip install WatchUI and then enter the Library WatchUI in the robot to use it.

What we Worked with

As already mentioned, Python was chosen as the main language of the library for identifying differences. The following libraries have been gradually added to it:

• OpenCV is a free and open cross-platform image manipulation library
• Pandas - A software library written for the Python programming language for data manipulation and analysis
• Pillow - Is a library for opening, manipulating and storing many different image file formats.
• NumPy is a basic library for scientists and analysts who work with Python. It defines a type for an n-dimensional homogeneous array (most often numbers) and an API for working with such an array.
• scikit-image is an open-source image processing library for the Python programming language
• scikit-learn is a free machine learning software library for the Python programming language.
• Imutils - a library for basic image processing functions
• Robot Framework - test framework written in Python
• Selenium - a library for using web browsers
• Robot Framework-seleniumLibrary - a library for robotframework and selenium

What we "Fought”

During the development, we encountered two obstacles. The first one concerned the design of the library itself, where despite the first analysis and deployment for the tests, we did not realize that it was necessary to program the possibility of setting up some features, such as the ability to choose in which folder the resulting images will be saved, or that it will be necessary to insert images with highlighted differences into the report itself created through the Robot framework.

The other issue, unlike the first one, was not complicated in terms of theory (how to design a solution that would be user-friendly), but in terms of time. We needed to implement the simplest possible creation of the pip package, documentation and the ability to update the code independently of the contributor.

Since we were using GitHub throughout the entire development, it has become quite logical that we will also use his GitHub actions function. That was a big challenge, as it was our first encounter with this tool. Nevertheless, after a few days we managed to have a fully functional CI, where "merge" occurs by agreeing to change the code with the repository owner. This will run the tests first, and if they work well, a pip package will be created and published on pypi.org. From there, other library users can easily download it.

What we Got

To the great excitement of all, we have managed to create an infinitely distributable library, which is freely available and currently used not only in the Czech Republic, but also in France. Our library has made our customer especially pleased. It allowed him to move the testers, who spent half a sprint each month by controlling the design, to other tests. As part of the project, we have saved both time and money.