It goes through many iterations but also incorporates a ton of customer feedback while it iterates. I will be talking about software development in this post with examples from my career and video games.
What makes a high quality video game? let us compare cyberpunk 2077 with Hades and the different development approaches.
The Jimquisition YouTube channel awarded Hades game of the year for 2020:
They credit the many years the game spent in beta and incorporating customer feedback along the journey as main reasons why Hades as turned into the awesome game it is.
There was never any forced “crunch” time (think overtime and sleeping under your desk) and the game has been well received by the community at large.
Where as cyberpunk 2077 has been the biggest hyped up flop of a game to be released in 2020. There’s an outstanding class action lawsuit, many reports of bugs and Sony pulled the game from the ps4 store.
Yahtzee only got their review copy unlocked 1 day before release, that’s not a lot of time to incorporate any user feedback:
Big Bang releases are bad
Hades had a team of roughly 20 people, they decided to do early access in 2018 on only one platform (the Epic Game Store) before porting to other platforms. It took them about 3 years of development. Source: Wikipedia.
Cyberpunk 2077 was announced in 2012, the team started at 50 people and grew to 500. It was released on all platforms at once and subsequently pulled from the ps4 store. Source: Wikipedia.
With all of the money, time and resources behind cyberpunk 2077 you think they’d able to release a solid game. However big bang releases and expensive projects don’t always get released as planned.
Pick one platform
If you are looking to develop a mobile app or website, pick one platform to learn and iterate on. Don’t start with an Android app + iOS app + Web app only to learn you built the wrong product. Depending on your situation you might pick one platform over the other.
When I was working at Tyro and we were working towards our banking licence, we built an iOS app first for managing your business bank account. Android came after we got our licence and a web banking experience wasn’t even on the tables by the time I left.
My current team released version 1.0 of our app in March in 2020, we’ve since done monthly releases and incorporated customer feedback into each release. This has been a great way of starting with a basic app and fleshing out the features to be the best superannuation app on the market (look I’m biased).
Keep an eye on the app store
Tyro is still doing pretty well on the app store, especially compared to something like the myGovID app which has a similar number of ratings.
Delivering quality products
You don’t need a giant team or lots of $ to build high quality products. If you start small and incorporate feedback it’ll be money better spent than a big bang approach.
A few days ago I posted on LinkedIn that I have hardly written any test automation code in the last year:
And I had a few people ask, “how do I test during a code review?”. So I thought I would dive deeper into what an average day for me looks like.
My Role; Software Engineer in Test
Officially my role is a Senior Software Engineer in Test on a mobile app team, it’s a little like this role description but with a mobile app focus instead of a C#/web focus. So why am I not writing test automation code? It’s there in the job description.
I’ll start my day by checking our scrum board for our current sprint and figuring out what my focus is for the day. What tasks have been assigned to me and what needs some focused testing:
I use git to help me with large code reviews. The developers will create a pull request with their new feature. I’ll check that code out on a local branch and use Xcode/Android studio to test that code on emulators/simulators. I’ll often use this git command to do so:
I’ll explore the new feature and pass the code review. I’ll chat with the developer if there were any issues observed. I might even run the tests they’ve created as part of that pull request.
I don’t do this for every code review, if it’s a small change I’ll approve it and once it’s merged into the main dev branch I’ll update my local to test those changes. Then I’ll test on emulators/simulators or I’ll wait for our continuous integration (CI) testing pipeline to produce an app file and test on devices.
I have a similar set up for Xcode, we are in the process of dropping support for iOS 11 because we’d also like to update Xcode.
I also sometimes checkout backend code but my work machine isn’t set up to be able to build our back end locally. I’ll usually use a test environment to test API’s via Postman or cURL.
Helping with unit testing
I’m more likely to help the developers I work with come up with unit testing strategy. E.G. we were working on a feature recently to add up transactions over the last financial year and we have over 100 different transactions types.
We came up with an approach, if we give every transaction it’s type as it’s value we can more easily check the sums invoved. E.G if one category to calculate was:
we could say we’d expect the unit test result to be $4 (2+3-1). We created the mock JSON that went into the unit tests and it really helped the developer figure out how to test this more easily.
Supporting the App
I helped my team build out a mobile analytics dashboard where we could monitor user engagement and any potential issues that needed more investigation. Recently I’ve be checking it nearly every day and helping my team add more sections because we’ve got an SSL certificate issue at the moment.
We release our app monthly and we’ve got 7 versions that our customers can be on. We have a force update API that the mobile app will hit before it starts the log in process. If a user is on an unsupported version they can’t log in and are directed to the play store to update.
BUT this API uses HTTPS and our production SSL certificate expired recently. We haven’t built certificate white listing yet so it means instead of the old app seeing a forced update, they see a generic error message instead because the SSL certificate has expired. Oops, our bad.
We were able to get 98% of our regular app users onto the latest build with the latest SSL certificate but people who log in infrequently would not have seen the forced update. We are monitoring for an increase in generic error messages and hoping that no one gives us a poor review on the app stores.
Most of my code changes involve adding mock JSON data to our mobile app mocking framework. We can test our mobile app independently of any test environment. It’s a really cool feature that I’m proud of our team for building it.
We have a bunch of profiles mocked into the test build of our app. E.G if you wanted to see the difference between a pensioner vs investor you can change the profile in the test build.
Our whole business has access to this test build to help the company support the app (not every staff member has an account to use the app with). Sometimes these mocks need updating. As a team we’ve discussed a programatic way to keep these profiles up to date but it’s a little hard and hasn’t been prioritised.
If we are working on a new feature and I’ve recently been testing the API, I’ll often add the API mock test data into our mobile repositories ahead of the mobile dev work to make that process easier.
Recently I’ve been branching out and helping the design team with competitor analyst and research for new features we are working on. For example, how do other mobile apps handle two factor authentication?
Running a bug bash
every few sprints I’ll book the team in for a bug bash. It’s dev tools down and bug hunting goggles on. I’ll organise some snacks and a confluence page of test data and information about what has changed recently to help people focus on what to test. We award our best bug bug hunter for their efforts and add bugs found to the backlog. You can read more about running a bug bash here. We try to keep it fun and social.
So yes, I haven’t written much test automation code and this is ok. The developers I work with are great at writing their own unit and integration tests. Does this make my work any less valuable? No.
I enjoy engaging with customers and analytics more than building out test automation code. If you are interested in learning the technical skills that I use on a day to day basis, I have a technical testers guide here.
Other benefits with building with accessibility in mind are known as the Curb-Cut effect. When people first introduced curb cuts (i.e. sloped ramps) at intersections for wheel chair riders they discovered that many other people benefited too; i.e. people pushing strollers/shopping trolleys, people with deliveries and people with walkers all used these new curb cuts to help navigate their urban environments.
Over 4 million people in Australia live with some form of disability. That’s 1 in 5 people (1). Over my lifetime I’ve experienced a number of health issues that have impacted my abilities temporarily. I might not be living with an officially diagnosed disability today but I frequently use features such as the closed captions on YouTube and Audio Books when I feel like changing up how I consume information.
It’s easy to get overwhelmed when thinking about accessibility testing, there’s many different levels of accommodations to consider (7). Getting started with the following is a good step in the right direction;
Vision Impairment (Can people use screen readers interact with your content?)
Hearing Impairment (does any linked video content have closed captions?)
Mental health/Intellectual/Autism spectrum (Does your content overload people’s processing ability? Is it easy to read and follow?)
It’s no wonder people often put this type of testing in the too-hard basket. However if you start with testing with vision impairments in mind you should be able to get a good return on investment towards a more inclusive experience.
Screen Reader technology
Most vision impaired users will use some sort of screen reader technology to browse the web. Even I use this technology when I feel like listening to a news article online instead of reading. 2.3% of the US population have some sort of vision impairment (2). This statistic is likely to grow with an aging population. The growth of voice interfaces will also grow the use case for this technology. I prefer using iOS’s VoiceOver technology but there are many other tools out there; Windows comes with Narrator and Android has TalkBack, there’s also the paid software JAWS.
Using VoiceOver with mobile apps
On my iPhone, I can go into Setting > Accessibility > VoiceOver to enable the screen reader:
I enable accessibility shortcuts on my mobile device to make it easier to use. When I press the side button 3 times, I can easily switch accessibility features on and off.
When I test using VoiceOver;
First I’d navigate to the app/site I’d like to test
Then enable VoiceOver using my shortcut
Finally flick/scroll two fingers up the screen
This enables the screen reader to read everything on the screen from top to bottom and I can test if the flow makes sense. There are plenty of other guides out there for using VoiceOver (12). I suggest switching on a screen reader on your mobile device and getting familiar with the technology. As a test, can you figure out how to take a photo on your phone using screen reader technology?
Large font accomodations
Jacking up the font size or using a dyslexia font plugin in on chrome is a great way to find ways your UI may break. Watch out for views that don’t scroll and text that doesn’t line wrap.
Alternative text for images
Screen readers will read out the alternative text for images for vision impaired users who can’t see your images. Alt text is an optional field for HTML image tags. For example, using Campaign Monitor’s Email Builder I have included an alt text tag with my image to convey the message of the image for screen readers. Here is the HTML tag for the image;
<img src="darth_vader.jpg" alt="Meme; Come to the dark side ... we have cookies">
This is what this email looks like with images;
I used Firefox’s web developer tools to replace images with their alt text to see this next view of the same email;
Check you have alternative text for your images and that it makes sense
Check that you have blank strings or no alt text options for decorative images so that screen reader don’t try to read the image or the file name
Don’t include words like “image of” in your alt text because screen readers will announce it’s an image and then announce the alt text. You will get a double whammy “image. image of blah” from your screen reader, this is super annoying
Use tools that display the alt text or replace images with their alt text to help test this
Test your images for colour blindness
About 4.5% of the British population experience some form of colour blindness (3). To help test this I have a grey scale shortcut button set up on my iPhone. On a side note, setting up grey scale for your phone helps make it less addictive too (17).
Hopefully you now realize that testing for accessibility in your apps/websites isn’t that hard. If a screen reader works with your content, you have alternative text fall back options for images and your images are still readable using grey scale you will be most of the way there in providing an awesome, inclusive experience with your online content.
a strategy doesn’t have to be a big giant document. It starts as an idea in your head and you have to get other people on board as part of that strategy. So you need to share some knowledge in some format to help share your idea. This blog is about how I’d go about developing a new test strategy in a new team.
History of the term
First let’s take some time to understand this term; strategy. Historically the word strategy is associated with war and battle:
Strategy is to help you win or achieve some goal. Many people talk about their tactics when they are thinking of their strategy. Tactics are your how. They aren’t your whole strategy.
A tactic is a conceptual action or short series of actions with the aim of achieving of a short-term goal. This action can be implemented as one or more specific tasks.
This book helped me understand the term, “Strategy” in a visual and fun way.
According to this book a strategy has 4 parts:
A distinct, measurable goal
A sequence of actions or tactics
Start with a purpose
If I was dumped into a new team tomorrow and asked to develop a test strategy, I’d start by interviewing/surveying a few people. Depending on the size of the team and who I was working with it could be an online survey or a casual chat over a coffee. I’d ask something along the following lines:
What does quality mean to you?
What are common problems in the testing process here?
If you could fix just one thing about our quality, what would it be?
Now different people are going to answer this differently. Developers might say test code coverage, easily maintainable code and easy deployments make a high quality product. Your project manager might say happy customers. Testers might say less bugs found in the test phase.
Develop a goal
Once I’ve surveyed enough people (5 people is a good enough number for most user research interviews), I’ll work on constructing a goal. it might be;
improve our continuous integration build times
increase our test coverage
reduce the amount of negative customer feedback
Make sure it is measurable. You could use SMART or OKR goal formats.
Develop a plan
Now what are some things I or the team could do to achieve our goals? We could create tasks during our sprint to help us work towards our goal. Once you’ve achieved something you survey those original interviewers to see if the perceived quality has actually improved.
Measure the improvements in quality of your product. For my team we are tracking the average app store ratings, crash rates and engagement with in app features to see if they are actually useful. https://bughuntersam.com/metrics-and-quality/
Risks and Gaps
A Test Strategy could also have a section about risks or gaps in this approach. For example things like performance testing and security testing might not be included. Having a brief explainer why these aren’t part of your strategy can be useful for explaining the context and scope.
if you are working on improving the UI Test automation coverage you can use this visual risk based framework to help focus on where to start and what to automate first and measure progress against it as part of your strategy.
I’m more comfortable with the term marketing strategy over test strategy because it’s easier to measure your impact and easier to come up with concrete goals. Software testing isn’t as tangible as many other parts of the business process and can be hard to measure.
Can your strategy be summarised by this comic:
What resources have helped you understand test strategies? I’d love to check them out.
The superannuation and investment mobile app I’ve been working on over the last year has finally been released. It’s been on the app store for just over a month now* and this blog is about how we are using metrics to help keep tabs on the quality of our app.
The average app store rating is one useful metric to keep track of. We are aiming to keep it above 4 stars and we are also monitoring the feedback raised for future feature enhancement ideas. I did an analysis of the average app store reviews of other superannuation apps here to get a baseline of what the industry average is. If we are better than the industry average, we have a good app.
Analytics in mobile apps
We are using Adobe Analytics for tracking page views and interactions for our web and mobile app. On previous mobile app teams I’ve used mParticle and mixpanel. The framework here doesn’t matter, I’ve found adobe workspace to be a great tool for insights, once you know how to use it. Also Adobe has tons of online web tutorials for building out your own dashboards.
App versions over time
Here’s our app usage over time broken down by app version:
We have version 1.1 on the app store and released 1.0 nearly 2 months ago. We did an internal beta release with version 0.5.0. If anyone on the old versions tries to log in they’ll see a forced update view.
Crashes are a fact of life with any mobile app team, there are so many different variables that go into app crashes. However keeping track of them and aiming for low rates is a good thing to measure.
With version 1.1 we improved our crash rates on android from 2.77% to 0.11%. You can use a UI exerciser that is called monkey from the command line in your android emulator to try and find more crashes too. With the following command I can send a 1000 random UI events to the emulator:
I can also keep on eye on how many error messages are seen. The spike in the android app error messages was me throwing the chaos monkey at out production build for a bit. However when there is both a spike in android and iOS, I know I can ask, “was there something wrong with our backend that day?”
Test Vs Prod – page views
If every page has one event being tracked, we can compare our upcoming release candidate against production; say we see that 75 page views were triggered on the test build and we compare this to the 100 page views we can see in production. We can then say we’ve tested 75% of the app and haven’t seen any issues so far.
There’s no need to aim for 100% coverage, our unit tests do cover every screen but because they run on the internal CI network those events are never sent to adobe. We have over 500 unit/UI tests on both android and iOS (not that number of tests is a good metric, it’s an awful one by the way).
But if you’ve tested the main flows through your app and that’s gotten you 50% or 75% coverage you are now approaching diminishing returns. What’s the chances in finding a new bug? Or a new bug that someone cares about?
You could spend that extra hour or two getting to 90-95% but you could also be doing more useful stuff with your time. You should read my risk based framework if you are interested in finding out more.
If you are working on a new feature or flow of your app, you can measure how many people actually complete the task. E.g. first time log in, how many people actually log in successfully? How many people lock their accounts? If you are trying to improve this process you can track to see if the rates improve or decline.
You could also measure satisfaction after a task is completed and ask for feedback, a quick out of 5 score along the lines of, “did this help you? was it easy to achieve?”. You can put a feedback section somewhere in your app.
The tip of the iceberg
These metrics and insights I’ve shared with you are just a small subset of everything we are tracking. And is a small part of our overall test strategy. Adobe has been useful for digging down into mobile device’s and operating systems breakdowns too. There’s many ways you can cut the data to provide useful information.
What metrics have you found useful for your team and getting a gauge on quality? What metrics didn’t work as well as you had hoped?
This is not financial advice and the views expressed in this blog are my own. They are not reflective of my employers views
Bugasura is an android app and a chrome extension. it helps with keeping track of exploratory testing sessions and comes with screenshot annotation and jira integration.
Here are a couple of screenshots of the android app in action, being used for an exploratory session on our test app.
First I selected the testing session:
While I’m testing I see this Bugasura overlay which I can tap to take a screenshot and write up a bug report on the spot:
Here’s their reporting a bug flow:
And here’s a testing report after I finished my exploratory testing where I can push straight to Jira if I want:
Here’s the sample report link (caveat, the screenshots attached to the bug are now public information on the internet, so there’s a privacy concern right there), but OMG, the exploratory session recorded the whole flow too (so a developer could see exactly what I did to find that bug).
Here’s that bug report in chrome paused at screen 13 out of 18:
Some caveats I’ve found so far; the test report is public (not private by default), you wouldn’t want to include screenshots of private or confidential information.
Bugasura only works on android/chrome. There isn’t an ios version but I guess with some remote device access running through Chrome it could work? We use Gigafox’s Mobile Device Cloud at work to access a central server of mobile devices and I imagine Bugasura could work with it.
Also I think they may have misspelt Elisabeth’s name in her quote.
This blog post reflects my opinions only and do not reflect the views held by my employer.
My team is going through a beta release for our mobile app to get early feedback. We’ve noticed that our android app is struggling compared to iOS. It seems that having an extra hurdle with signing up for the android beta program impacts installations. Naturally we’d expect the android engagement to lag a little behind iOS based on the Aussie mobile usage market analysis but there is still a significant drop.
We have 428 iOS installs, and 99 android installs. That’s a 19% Android installation rate. We have roughly 75-80% successful registrations and return log ins once people actually figure out how to install the app.
Google Groups vs Test Flight
We are using google groups to manage the distribution of the android beta app and because it’s harder to use than test flight for iOS we’ve gotten less installations. It’s fascinating how an extra hurdle in the sign up process can impact installations.
Our android numbers appear to be higher than usual here but I think it’s to do with the timeframe I’m collecting these numbers over. We’ve had a few people install the android app before we officially started the beta release and I think they’ve been counted in this statistics.
I like to use mind maps to help me test. Mind maps are a visual way of brainstorming different ideas. On a previous mobile team I printed and laminated this mind map to bring along to every planning session to help remind me to ask questions like, “What about accessibility? Automation? Security? or Performance?”:
As I go through exploratory testing (or pair testing), I’ll tick things off as I go and take session notes. Often this will involve having conversations with people, sometimes bugs are raised. Here is a quick mind map I’ll use for log in testing:
Heuristics for testing
This mind map approach can be combined with a heuristic test strategy approach or a nemonic test approach. Heuristics is a rule of thumb that helps you solve problems, they often have gaps because no mental model is perfect.
SFDPOT is a common nemonic that was developed by James Bach; who also developed Rapid Software Testing; a context driven methodology. James Developed his RST course with Michael Bolton.
We truly live in a global and inter connected society. But have you tested your app using a Right to Left (RTL) language such as Arabic? This blog post is a reflection on some of the design considerations to keep in mind when accomodating this.
Why does this matter?
Arabic is one of the top 5 spoken languages in the world with around 3 hundred million speakers and it is the third most spoken language in Australia. Even if you only release apps for the Australian market someone out there will have Arabic set as their default device language. It’s ok if you haven’t translated your app, but you should check that these people can still use it.
How do I test this?
Enable developer options and select “Force RTL layout direction”. On My Samsung S10 this is what my screen and options look like after enabling this option:
In Xcode you can change the build target language to a Pseudo RTL language to see how your app renders in this way without having to change the language on your device.
You don’t actually need to render your key pads in Right To Left, in fact it’s actually more jarring to render numbers in a RTL arrangement because ATM’s and phone pads are left to right in Arabic. Most Arab’s are use to globalised number pads. Samsung has an in-depth article on when RTL should be applied.
When I have RTL rendering set on my android phone, the log in pin screen and phone call functionality is in LTR. However some of my banking apps render their pin pads in RTL.
Common RTL Issues
I was pleasantry surprised to find out how many of my apps weren’t broken when I switched to RTL rendering. Facebook, twitter and email still look very good. Some apps (like my calculator) do not make sense to render RTL and they remain LTR:
Bug One: Overlapping labels
You will have to watch out for when labels overlap like in the domain app here:
Bug Two: Visuals doesn’t match written language
And when your text is rendered RTL but the visual cue is still LTR like in the shade bar for representing countries visitors to my blog in this wordpress statistics view:
Bug Three: Menu’s that animate from the side
In the app I’m helping build, the side menu renders pretty funkily in RTL mode, I can’t show you a screenshot of this behaviour but it’s probably the quirkiest RTL bug I’ve seen. If you find an app with bad side menu behaviour in RTL please share your screenshots with me. I’ve also seen a pin login screen where the icons where flipped but the button presses weren’t.
Bug Four: Icon’s aren’t flipped
Often icon’s have a direction associated with them like the walking person when you get google maps directions. Sometimes it can look a little odd when they aren’t flipped correctly (as if they are walking backwards).
Have you seen these bugs before?
Please let me know your thoughts or experiences in supporting RTL languages. I’d love to hear your stories.