How making our app accessible broke our app tests

You may have experienced a situation when someone told you about a piece of code that didn’t work, even when it sounded so simple. That was exactly my case with the relatively new, accessible slider buttons in our apps!

Almost a year before I started working at willhaben, our iOS app development team started an initiative to make our app more accessible, especially to people who are using screen readers. In doing this, they replaced all the slider buttons we use, such as during registration for our terms and conditions, with a new version that can be read by the iOS screen reader. However, our iOS tests with this new button were all broken by this change, and no one was able to find a solution.

The piece of code that broke the tests was just checking if the button was active or not. We used it everywhere to correctly check and set the slider buttons.

As I had some experience in prior jobs with appium, I was tasked with fixing this, and we assumed it would be a piece of cake. Unfortunately, the solution wasn’t as easy as we’d assumed.

The first step I took was to start the appium inspector and check what was going on with the button.

Finding the button on the page was easy. Unfortunately, when I saw what data appium could get from button’s “value” attribute, I realized it was different than before.

Of course, the solution was just going into the code and grabbing the value of the slider button for our checks, start it, and then it should all be fine. Right? Right…

When running the code, it again crashed at the same point as before, but this time it told me that the button had no value. Remember the appium inspector, which clearly shows us that this button has a value of either “on” or “off”? Apparently, our test automation framework couldn’t find this. Our team invested quite some time in researching the problem and found one loosely related stackoverflow question with no meaningful comments or solutions. The devs had no fix for it, as this was apparently a problem coming from somewhere deep within our code.

After that initial throwback and some brainstorming, we tried to use other means for actually getting the value, like XPath and css selectors, instead of the accessibilityID. Again, nothing worked; either it told us the value was empty, or it didn’t find it at all.

All of this led us to abandon the issue and just accept the broken tests while waiting for some miracle to occur.

Then, I figured out how to do it!

After some time thinking about our problem every once in a while, it hit me: “If I can actually see the button being switched, why not let appium also see it, rather than checking for a value?” I planned this to be an intermediate solution, while waiting for someone else to come up with a more permanent one, or waiting for it to be fixed by our iOS app developers (more on that at the end of this blog post).

I wrote a function for recognizing the state of the slider button based on an image. It grabs a screenshot of the element and then converts it to a BufferedImage:

File file = element.getScreenshotAs(OutputType.FILE);
BufferedImage img = ImageIO.read(file);

The second step was checking how many blue pixels the image contains, which was a bit tricky. First, I wanted to transform it into a simpler image with just 8-bit color information, but I ended up finding that I could also count these pixels directly:

int bluePixels = 0;
for (int y = 0; y < img.getHeight(); y++) {
    for (int x = 0; x < img.getWidth(); x++) {
        int redColor = (img.getRGB(x, y) >> 16) & 0xff;
        int greenColor = (img.getRGB(x, y) >> 8) & 0xff;
        int blueColor = img.getRGB(x, y) & 0xff;
        if ( blueColor > 220 && 4*greenColor < 3*blueColor 
                && greenColor > 150 && 10*redColor < greenColor ) {
            bluePixels++;
        }
    }
}

So, I defined a variable for counting the blue pixels and went through the image pixel by pixel. There are two loops to go through, the vertical and horizontal axes of the image. Each pixel is split into the red, green, and blue parts of the color spectrum. I needed to check how much of it fell into blue. For a better understanding, this is what the screenshot with just the blue pixels looks like:

In the end, the function returns “true” or “false” based on how many pixels are blue. If more than a third of them are blue, we say that the button is active right now.

return bluePixels > (img.getHeight() * img.getWidth()) / 3;

The solution we used ended up being pretty easy to implement, and very stable, despite being planned as a temporary workaround. While I was working on the fix, the app team was trying to implement the value in a way that appium could read, but to no avail. All of their “easy fixes” somehow turned out to be incompatible with our test automation framework. Other ideas that popped up would cause a major rewrite of the button’s functionality, to avoid breaking accessibility features, and we had no time for that.

So, as it works out most of the time while coding, it turned out that our “short-term” fix for this problem, and the easiest fix, was the best after all.