Automation as a part of a test strategy is a solid business case in most, if not all, traditional development teams, regardless of development method, environment, or application nature. For a low-code platform such as Mendix, this is even more applicable. Due to the high speed of delivery, regression test sets tend to rapidly grow out of hand. When you develop in Mendix, it is nearly impossible to test at the same speed you are developing at- without the help of a test automation tool the only remaining option would be to drastically cut the coverage of your regression test set.
With the use of automation tooling also comes the need for a new test (automation) strategy.
Of course, the idea of changing your test/automation strategy to fit new, higher speed development methods is not new. Long before low-code app development became as big as it is today, development teams went through the same struggle when moving from Waterfall to Agile development. This struggle resulted in The Test Pyramid, a true dogma in the software test community. There are several versions of this pyramid but generally, they recognize three layers of automated testing: a base of unit-level testing, a mid-section of system (service, API, component) level testing, and a tip of UI-level testing. Above it all is a ‘cloud’ that represents the non-automated (mainly exploratory) testing. The goal is to push your tests as far ‘down’, towards unit testing, as possible and to keep the number of UI-level tests to a minimum. The reasoning behind this is that while manual testing is laborious and automated UI-level tests are notoriously fragile and expensive to create, run, and maintain, unit tests are the opposite: stable and cheap.
While the Test Pyramid may be considered the ultimate goal, some teams are “stuck” in the Test Ice Cream Cone situation where automated tests are scarce at the unit level. Increasing numbers can be found at the system level and most are created at the UI-level. This "ice cream cone" is then topped off with a big chunk of ice cream- also known as manually executed UI-level tests. It’s easy to see why an Ice Cream Cone is undesirable and why aiming for a Pyramid would make sense. However, the latter might not be that straightforward for Mendix applications.
When we look at unit testing in Mendix, it quickly becomes clear that it deviates from traditional environments. Mendix makes use of many out-of-the-box building blocks that have already been developed, and importantly, tested for you. This alone reduces the need to create unit tests, but there is more. The power of unit tests in traditional environments is that it allows you to create very small tests for very small units, such as classes or methods. In comparison, the smallest units that you test in Mendix are often much larger than that, resulting in less stable, more expensive tests. Furthermore, although the Unit Testing module is a valuable addition to any Mendix project, platform limitations reduce its level of freedom and the benefits of unit testing as compared to traditional environments.
On the other end of the spectrum, there is UI-level testing. This can be automated with many different tools. In most cases, this comes down to either writing tests yourself in code (e.g., Java(Script), C#, Python) that drives Selenium Webdriver. An alternate option would be making use of a framework that drives Selenium Webdriver as well but adds a logical layer that takes care of the technical difficulties. The former leads to test cases that are hard to understand or maintain for nearly anyone who’s not a Selenium specialist. The latter often limits the freedom that Selenium offers but makes it easy to use for those who are not very tech-savvy. Moreover, both types of tools have a problem in common: maintainability. Small changes in the application’s UI can break your test cases, which will then require time-consuming debugging- this is exactly why UI-level testing is usually best avoided.
However, Mendix, together with CLEVR, offers a tool that solves these problems. Mendix Application Test Suite (ATS) enables you to quickly create readable test scripts that almost anyone can create and maintain. This can be done by hand or with its recorder, while still offering the extensive possibilities of Selenium when needed. Additionally, ATS offers Mendix version compatibility. This means that even if the implementation of a widget changes under the hood from one Mendix version to another, ATS is still able to recognize it in your application. Consequently, existing test cases addressing these widgets do not break down upon upgrading Mendix, saving you from spending a lot of time on test case maintenance. Therefore, as compared to other UI-level test automation tools, test cases in ATS are more stable as well as cheaper.
All in all, it’s neither pyramids nor ice cream cones when it comes to test automation in Mendix. The principles of the Test Pyramid remain because aiming for automated tests that are as stable and cheap as you can make them is still good practice. However, given the properties of the platform and the available tools, it is possible to create stable and cheap UI-level tests- tests that can be made in close collaboration with the business, very much in line with the philosophy of Mendix itself.
Get in touch
