Category: Software Testing

Categories
Critical Thinking Marketing Mobile Testing Software Testing

That elusive Test Strategy

I recently was asked about recommendations for learning about test strategies. Here are my sample strategies:

a strategy doesn’t have to be a big giant document. It starts as an idea in your head and you have to get other people on board as part of that strategy. So you need to share some knowledge in some format to help share your idea. This blog is about how I’d go about developing a new test strategy in a new team.

History of the term

First let’s take some time to understand this term; strategy. Historically the word strategy is associated with war and battle:

Quote: Strategy without tactics is the slowest route to victory. Tactics without Strategy is the noise before the defeat - Sun Tzu
https://www.pinterest.com.au/pin/287878601154737781/

Strategy is to help you win or achieve some goal. Many people talk about their tactics when they are thinking of their strategy. Tactics are your how. They aren’t your whole strategy.

A tactic is a conceptual action or short series of actions with the aim of achieving of a short-term goal. This action can be implemented as one or more specific tasks.

https://en.wikipedia.org/wiki/Tactic_(method)

Book: I have a strategy (No you don’t)

This book helped me understand the term, “Strategy” in a visual and fun way.

https://www.amazon.com.au/Have-Strategy-You-Dont-Illustrated/dp/1118484207

According to this book a strategy has 4 parts:

  • A purpose
  • A distinct, measurable goal
  • A plan
  • A sequence of actions or tactics

Start with a purpose

If I was dumped into a new team tomorrow and asked to develop a test strategy, I’d start by interviewing/surveying a few people. Depending on the size of the team and who I was working with it could be an online survey or a casual chat over a coffee. I’d ask something along the following lines:

  • What does quality mean to you?
  • What are common problems in the testing process here?
  • If you could fix just one thing about our quality, what would it be?

Now different people are going to answer this differently. Developers might say test code coverage, easily maintainable code and easy deployments make a high quality product. Your project manager might say happy customers. Testers might say less bugs found in the test phase.

Develop a goal

Once I’ve surveyed enough people (5 people is a good enough number for most user research interviews), I’ll work on constructing a goal. it might be;

  • improve our continuous integration build times
  • increase our test coverage
  • reduce the amount of negative customer feedback

Make sure it is measurable. You could use SMART or OKR goal formats.

https://www.toolshero.com/personal-development/smart-goals/
https://blog.weekdone.com/introduction-okr-objectives-key-results/

Develop a plan

Now what are some things I or the team could do to achieve our goals? We could create tasks during our sprint to help us work towards our goal. Once you’ve achieved something you survey those original interviewers to see if the perceived quality has actually improved.

Measure your progress

Measure the improvements in quality of your product. For my team we are tracking the average app store ratings, crash rates and engagement with in app features to see if they are actually useful. https://bughuntersam.com/metrics-and-quality/

Risks and Gaps

A Test Strategy could also have a section about risks or gaps in this approach. For example things like performance testing and security testing might not be included. Having a brief explainer why these aren’t part of your strategy can be useful for explaining the context and scope.

UI Automation Visual Risk Framework

if you are working on improving the UI Test automation coverage you can use this visual risk based framework to help focus on where to start and what to automate first and measure progress against it as part of your strategy.

https://bughuntersam.com/visual-risk-ui-automation-framework/

Conclusion

I’m more comfortable with the term marketing strategy over test strategy because it’s easier to measure your impact and easier to come up with concrete goals. Software testing isn’t as tangible as many other parts of the business process and can be hard to measure.

Can your strategy be summarised by this comic:

test all the things
automate all/some things

What resources have helped you understand test strategies? I’d love to check them out.

Categories
Job hunting mental health Software Testing Technology Weight Loss

Discrimination in the Workplace

There are many forms of discrimination you could face when you are looking for work or in the workplace. Whether it’s based on;

  • Gender
  • Appearance
  • Family
  • Religion
  • Sexual Orientation
  • Age

Many of these forms of discrimination are illegal in many countries. Hiring Managers/Recruiters who work in Human Resources departments (HR) know how to protect their companies from potential lawsuits, so they won’t be directly discriminatory to your face. However you still might face indirect discrimination.

This blog post is a deep dive into the the subtleties of discrimination so you can be more aware of it during the job hunting process. Topics covered include:

  • Women in Tech
  • Fatness and Bias
  • Mental Health
  • Local Experience

Women in Tech

Women still face discrimination in the tech industry. Trans folk even more so. I read a story about a female to male transition for a tech support worker. When he transitioned he visited an old client to help fix a computer. The client thought it was a new person and complained about how the women who use to help fix their computer didn’t know anything about tech. It was the exact same person.

Here’s another story about how a Husband came to understand the discrimination his Wife and Business Partner faces on a daily basis.

In my early career I avoided using my full name (Samantha Connelly) on my CV because of the gender association. If you look at my old 2014 CV, I have S. Connelly as my name.

Now that I have a reputation in the industry, I can’t hide my gender. I’m a little gender queer in my representation and could easily go by Sam (he/him) but I don’t suffer from gender dysphoria and don’t have a strong desire to change. I will remain female because I’m comfortable in my own skin (CIS). Also men’s fashion isn’t as fun as women’s fashion.

Fatness and Bias

I use to be 127kg (that’s 280 pounds for my US readers and 20 stone for my UK readers). I’m now around 83kg, at 160cm tall this still puts me in the Obese category. I had always grown up being the fat kid. I had weight loss surgery (gastric sleeve) in 2016.

People tend to think fat people are lazy and unmotivated. Back when I was fatter and saw someone else who was even larger, I use to thing, “well at least I’m not that fat”, but I’ve seen my mindset change first hand. I now have the same knee jerk, “ew, gross” reaction as everyone else.

Fatness itself is generally not cause for legal discrimination (unless it’s classified as a disability), however overweight people are less likely to be promoted to leadership positions because they are seen as less competent.

When was the last time you saw a fat leader in a tech company? That weight loss surgery has probably already paid for itself based on my increased in potential earning capacity and more leadership opportunities.

Mental Health

This time last year, I thought I was going to move to Newcastle to join a start up in a head of engineering role. After the offer had been made, someone on the board did “further research” into my history, and getting fired from Campaign Monitor came up.

They thought I had been fired because of my history of mental health impacting my performance. At the time I was recovering from a broken ankle and I had presented to the whole company about my struggles with depression.

However, the reason why I was let go was a mismatch of skills and expectations. It was an experimental Quality Coach role, through hiring me they discovered they actually wanted someone to help grow the test automation framework for the C# backend. This wasn’t my strength and we broke up on good terms.

I even presented at a conference of how I tried a quality coach role and failed at it. I put together this presentation with the help of my old boss from Campaign Monitor. I was super excited for this role back at the end of 2017.

Local Experience

Through my career coaching sessions and leading Sydney Testers over the last 4 years, I’ve spoken to many people who are looking for their first job here in Australia. They often get rejected for not having any “local experience”. I view this as a form of discrimination. It’s an excuse to not consider you as a candidate.

I can’t blame hiring managers for taking this mental shortcut. When you are dealing with 100’s of applicants and you want to get the list down to 4 to interview, you take many shortcuts to get there.

It does mean people often struggle to land that first job here. I’ve told people to invest in their online profile and networking to overcome this barrier.

Summary

I have no idea how other people over come other forms of discrimination (like agism and racism). But this blog is full of stories of things I’ve tried or heard that can help people put their best foot forward during the job hunting process.

If you’re an older disabled fat black mother working in tech good luck out there because society isn’t on your side.

What’s worked for you? Or did something backfire?

Categories
Craft Beer Critical Thinking Finances Job hunting Marketing mental health mindfulness Software Testing Technology

Buddha in Testing: Chapter 5

At the end of Buddha in Testing, Pradeep asks the reader to co-author the next chapter with him. So this blog post is my attempt at writing part of Chapter 5 of this book:

What is the chaos that surrounds you in testing?

Write now, during the pandemic a lot of people have been made redundant and are struggling to find work. I’m lucky enough that my day job isn’t all that chaotic, which is a good thing. The mobile app I’m working on is doing pretty well. I wouldn’t want to be dealing with a stressful work load on top of everything else.

What is my contribution?

I put together a software testers career cheatsheet to help anyone whose struggling to find work right now. After having career coaching sessions with a bunch of people, a few themes came to light. I got the inspiration to do a video series on those points. I found out it makes for great marketing content.

What situations have put you out of calmness?

Last weekend I recorded 7 career tip videos in one weekend. I was burnt out by Monday and a blubbery, teary mess. I couldn’t focus on work and took the day off to mentally recharge. I told twitter I was out of spoons.

How did you bring peace?

Walking around the city, listening to podcasts and shopping in second hand clothes stores was how I recharged. I even had a beer in a sports bar at lunch and watched some cricket (England vs West Indies) :

What answers are you searching for?

Satisfaction in life. I’m over software testing. I’m starting a graduate diploma in financial advice next week because I have an idea to disrupt the retirement funds industry here in Australia. Making retirement funds easier is something I can get behind.

How will you recognise the peace?

I enjoy adding value to other people. It’s a huge driver to most of what I do. I miss the constant interaction with people from my shop assistant days. If money/labour wasn’t a drawback I’d prefer to work in a supermarket over most of the testing roles I’ve had. With my history of depression, I don’t think I’d ever achieve peace but I can be more content with life.

I’m now outta steam

I could continue answering the questions but I think I’m going to leave it there. How would you answer some of these questions?

Categories
Critical Thinking Job hunting Marketing Software Testing Technology

Technical tips for Software Testers

My software testing career tips on Youtube is going well. So well in fact that I need to break out and collect the 4 part mini series on technical skills into it’s own blog post.

Tip 6: learn command line

Nothing will impress you colleagues more than your technical prowess with the command line, even if all your doing is checking your email. Here’s all of the references in the video:

tip 7: trending tools

You should focus on learning tools and technology that help you collaborate with the developers on your team. Here’s all of the references in the video:

Tip 8: The Technical profile

Having a GitHub profile is key to establishing your tech credibility. Here’s all of the references in the video:

Tip 9: Manual vs Automation

I avoid these terms in my profile like the plague and as an Industry we should drop these terms. Here’s all of the references in the video:

What are your tips for testers when it comes to improving their technical skills?

Categories
Conferences Marketing Software Testing Technology

My Social Media Marketing Strategy

On Sunday I’ll be Interviewing Manoj Kumar who’s a Principal Consultant at ThoughtWorks & a Selenium Conf Organiser. And I was chatting to Manoj about my marketing strategy as a tester. I’ve been fascinated about marketing (more than learning about test automation tools) and this blog post is a reflection on that strategy.

Marketing is a numbers game

In marketing, click through rates, conversion rates and eyeballs are king. You can either increase your eyeballs/views or increase the conversion rate.

E.G. You are promoting a free event, if you send an email to 100 people and 5 people sign up, your conversion rate is 5% for that campaign. It’s alot easier to increase the number of emails sent than the conversions/click through rates.

You could send that same email to 100 more people or send follow up emails to the people who opened the email but didn’t register to increase conversions. Or maybe the subject line didn’t get people’s attention?

Anyway, there’s tons of testing and iterating that can be applied to a marketing campaign.

What is your goal?

Have a think of what you’d like to achieve with your marketing campagin before starting out. I’ve structured my LinkedIn to get more views of my blog. Nearly everything I do on LinkedIn is to increase the web traffic to my blog.

My goal is to get more views/traffic on my blog

You might want to increase ticket sales for a conference, increase sign up rates for an event, etc. My secondry goal is to grow my number of followers on YouTube/Twitch.

How will you measure your goal?

I use analytics on wordpress to measure sources of web traffic. First let me tell you about my template messages and then I’ll explain how I use analytics to measure it.

Here are my LinkedIn template messages

New Connection

Hi {Insert_First_Name},

Thanks for connecting. What are some of the challenges facing you these days?

You might enjoy reading my blog on metrics and quality: https://bughuntersam.com/metrics-and-quality/ 

Is there anything I can help you with?

Regards,

Sam

They Ask for Work/Career Advice

Unfortunately it’s hard for me to refer someone who I’ve never worked with personally. I also don’t know what type of team would suit you.

Here’s my 6 step software testers career cheatsheet: https://bughuntersam.com/my-testing-career-cheatsheet/

Networking is king. I’ve gotten my last 3 job offers from networking, blogging and speaking at Meetup events.

Specific Campaign

I’ve sent the following template message to over 100 people who have QA Engineer in their title, are based in India and I’m connected to (I have up to 240 connections that meet this criteria):

Hi {Insert_First_Name},

How are you? Are you aware of the Selenium Conference in India? https://seleniumconf.in/

I’ll be interviewing Manoj Kumar, one of the organisers at 7am IST this Sunday https://www.meetup.com/Sydney-Testers/events/271598548/

If that’s too early for you the interview will be up on youtube 24 hours after the live event.

Regards,

Sam

I’ve had over 30 people respond to that last message directly over night, that’s a 30% response rate of people who’ve atleast clicked on the auto generated response message, “Thanks”.

Analytics

Using wordpress; Jetpack paid services, I can track views, where they come from and what people look at:

Yesterday I had 286 Views from 141 Visitors which is my second best performing day over the last month for web traffic.

Yesterday I had 286 Views from 141 Visitors which is my second best performing day over the last month for web traffic.

Out of those 286 Views, 11 were on my Metrics and Quality blog, so they were probably from my New_Connections template message.

173 Views were from India (Normally most of my traffic comes from Australia) and 16 of those views came from LinkedIn. This would indicate that my blog was shared outside of my marketing efforts.

Here’s a more regular day

As a comparison, here’s a more regular web traffic day for my blog:

Newer published blogs are the most viewed (especially if I’ve shared them), Most of my traffic comes from LinkedIn and most of my traffic is from Australia, because that’s my biggest area of influence.

I can also keep an eye on my engagement in LinkedIn too, this helps me to understand if the content I’m sharing is resonating with people:

Is this useful?

Would you try/test anything from this blog post? What are your views on marketing?

Categories
Critical Thinking Software Testing Technology

Code Smells and Boolean Logic

Have you ever come across a boolean in code (i.e. a TRUE/FALSE) variable and had to stop and question if TRUE meant enabled or disabled? I call this a code smell.

A code smell is a surface indication that usually corresponds to a deeper problem in the system.

https://martinfowler.com/bliki/CodeSmell.html

Naming variables is hard

One of the hardest things in software engineering is naming variables. They need to be easy to understand, short (ish) yet descriptive. And working in international teams, everyone has a different understanding of language. It’s a nigh impossible task if you ask me.

Kill Switches

Say there’s a kill switch feature that a business or developer can enable to block a client from hitting a backend via an API. Just in case there’s peak demand or something in the system is struggling. Or maybe you are concerned there’s a widespread Denial of Service Attack (DoS) hitting your system and you want to keep your customer data safe.

What should this kill switch be named? And what state corresponds to the switching off of web traffic?

The enabled state should correspond with the no traffic state. When this kill switch is enabled, the business has gone in and switched it on, effectively switching off traffic. By default this kill switch is disabled and web traffic is normal. You might want to call this kill switch something that relates to the API it switches off, e.g. WebLoginKillSwitch if the kill switch prevents people from login to your web.

Feature Flags

A feature flag is a software development technique used to enable or disable functionality remotely without deploying code.

https://launchdarkly.com/blog/what-are-feature-flags/

Say you are working on a new feature but you are operating in a continuous integration and deployment environment. Once your code is merged in it could be deployed to customers within minutes. But your feature isn’t ready for customers just yet. You can wrap your feature behind a feature flag and enable it for your team so you can test in production even before your customers see it.

By default this feature is disabled for most of your users. But you could also set up a % rollout for the feature too. Maybe 5% of your users see the new feature before doing a general release.

When naming a feature flag you don’t need to include the word enabled or disabled in the variable. If you are experimenting with a new way of web login using apple ID you might call this feature flag webLoginWithAppleID and have it disabled by default.

Read the art of readable code

If you are interested in learning more about readable code, I recommend reading the Art of Readable Code:

Do you have any code smells related to naming of variables? How about test code smells?

Categories
Job hunting Software Testing

My Testing Career Cheatsheet

Here’s a PDF of useful links, tips and resources that have helped me in my career so far:

Here’s the YouTube playlist of all the videos in this series:
https://www.youtube.com/playlist?list=PLXzkSAwwpH0m7Llq_HHBsC8KwFI5P1xWp

1 – Start with Why

2 – Building your profile

3 – Networking is king

4 – Interview tips

5 – Technical tester?

6 – Other resources

Why did I create this?

Yesterday I put an announcement on LinkedIn, if you had been made redundant recently you can book me in for a FREE 30 minute 1 on 1 career coaching session for the month of July.

You can still book me in via this calendly link. But my usual rates of $50 AUD for a 30 minute session will apply.

I plan on sharing this PDF with everyone I chat to as a post chat follow up take away.

What resources have helped you in your career? What other video’s on testing would you like to see?

Bonus material

Categories
Agile Design Finances Mobile Testing Software Testing Technology

Metrics and Quality

The superannuation and investment mobile app I’ve been working on over the last year has finally been released. It’s been on the app store for just over a month now* and this blog is about how we are using metrics to help keep tabs on the quality of our app.

*You can download the app via google play or the apple app store, you can also create an account here.

Average app store rating

The average app store rating is one useful metric to keep track of. We are aiming to keep it above 4 stars and we are also monitoring the feedback raised for future feature enhancement ideas. I did an analysis of the average app store reviews of other superannuation apps here to get a baseline of what the industry average is. If we are better than the industry average, we have a good app.

Analytics in mobile apps

We are using Adobe Analytics for tracking page views and interactions for our web and mobile app. On previous mobile app teams I’ve used mParticle and mixpanel. The framework here doesn’t matter, I’ve found adobe workspace to be a great tool for insights, once you know how to use it. Also Adobe has tons of online web tutorials for building out your own dashboards.

App versions over time

Here’s our app usage over time broken down by app version:

We have version 1.1 on the app store and released 1.0 nearly 2 months ago. We did an internal beta release with version 0.5.0. If anyone on the old versions tries to log in they’ll see a forced update view.

Crash Rates

Crashes are a fact of life with any mobile app team, there are so many different variables that go into app crashes. However keeping track of them and aiming for low rates is a good thing to measure.

With version 1.1 we improved our crash rates on android from 2.77% to 0.11%. You can use a UI exerciser that is called monkey from the command line in your android emulator to try and find more crashes too. With the following command I can send a 1000 random UI events to the emulator:

adb shell monkey -p {mobile_app_package_name} -v 1000

Crashes in App Centre

We can dive a bit deeper into crashes in app centre (a Microsoft based platform that integrates with our team city continuous integration pipeline for managing all of our test builds).

When exploring stack traces you want to look for lines that reference your app (instead of getting lost in all of the framework code), look for lines that start with your apps package name.

App Centre gives reports based on device and operating system break down:

With analytics set up, you can even dig into an individual report and get the page views that happened before that crash occurred.

What’s a good crash rate?

That depends on the context of your app, ideally zero is the best but perfect software is a myth we keep trying to obtain. As long as it’s trending downwards you are making progress towards improving it. Here’s a good follow up blog post if you are interested in reading more.

Error Tracking

I can also keep on eye on how many error messages are seen. The spike in the android app error messages was me throwing the chaos monkey at out production build for a bit. However when there is both a spike in android and iOS, I know I can ask, “was there something wrong with our backend that day?”

Test Vs Prod – page views

If every page has one event being tracked, we can compare our upcoming release candidate against production; say we see that 75 page views were triggered on the test build and we compare this to the 100 page views we can see in production. We can then say we’ve tested 75% of the app and haven’t seen any issues so far.

This is great for measuring the effectiveness of bug bashes/exploratory testing sessions. If you want an answer to, “how much testing did you/the team do?”.

Hang on, why 75%?

There’s no need to aim for 100% coverage, our unit tests do cover every screen but because they run on the internal CI network those events are never sent to adobe. We have over 500 unit/UI tests on both android and iOS (not that number of tests is a good metric, it’s an awful one by the way).

But if you’ve tested the main flows through your app and that’s gotten you 50% or 75% coverage you are now approaching diminishing returns. What’s the chances in finding a new bug? Or a new bug that someone cares about?

You could spend that extra hour or two getting to 90-95% but you could also be doing more useful stuff with your time. You should read my risk based framework if you are interested in finding out more.

Measuring Usability

If you are working on a new feature or flow of your app, you can measure how many people actually complete the task. E.g. first time log in, how many people actually log in successfully? How many people lock their accounts? If you are trying to improve this process you can track to see if the rates improve or decline.

You could also measure satisfaction after a task is completed and ask for feedback, a quick out of 5 score along the lines of, “did this help you? was it easy to achieve?”. You can put a feedback section somewhere in your app.

The tip of the iceberg

These metrics and insights I’ve shared with you are just a small subset of everything we are tracking. And is a small part of our overall test strategy. Adobe has been useful for digging down into mobile device’s and operating systems breakdowns too. There’s many ways you can cut the data to provide useful information.

What metrics have you found useful for your team and getting a gauge on quality? What metrics didn’t work as well as you had hoped?

This is not financial advice and the views expressed in this blog are my own. They are not reflective of my employers views

Categories
Mobile Testing Software Testing

Bugasura and exploratory Mobile Testing

A few weeks back, Pradeep Soundararajan; founder of Moolya testing and I were having a conversation on twitter on test strategies for mobile apps. He suggested trying Bugasura for running a bag bash.

Bugasura is an android app and a chrome extension. it helps with keeping track of exploratory testing sessions and comes with screenshot annotation and jira integration. 

Here are a couple of screenshots of the android app in action, being used for an exploratory session on our test app.

Bugasura Flow

First I selected the testing session:

While I’m testing I see this Bugasura overlay which I can tap to take a screenshot and write up a bug report on the spot:

Here’s their reporting a bug flow:

And here’s a testing report after I finished my exploratory testing where I can push straight to Jira if I want:

Here’s the sample report link (caveat, the screenshots attached to the bug are now public information on the internet, so there’s a privacy concern right there), but OMG, the exploratory session recorded the whole flow too (so a developer could see exactly what I did to find that bug).

Flow Capture

Here’s that bug report in chrome paused at screen 13 out of 18:

On a side note, I love having these Jerry Weinberg (who wrote perfect software and other illusions about testing) and Elisabeth Hendrickson (who wrote Explore It!: Reduce Risk and Increase Confidence with Exploratory Testing) quotes sprinkled through out the app.

Some caveats I’ve found so far; the test report is public (not private by default), you wouldn’t want to include screenshots of private or confidential information.

Bugasura only works on android/chrome. There isn’t an ios version but I guess with some remote device access running through Chrome it could work? We use Gigafox’s Mobile Device Cloud at work to access a central server of mobile devices and I imagine Bugasura could work with it.

Also I think they may have misspelt Elisabeth’s name in her quote.

This blog post reflects my opinions only and do not reflect the views held by my employer.

Categories
Mobile Testing

Beta releases for mobile apps

My team is going through a beta release for our mobile app to get early feedback. We’ve noticed that our android app is struggling compared to iOS. It seems that having an extra hurdle with signing up for the android beta program impacts installations. Naturally we’d expect the android engagement to lag a little behind iOS based on the Aussie mobile usage market analysis but there is still a significant drop.

We have 428 iOS installs, and 99 android installs. That’s a 19% Android installation rate. We have roughly 75-80% successful registrations and return log ins once people actually figure out how to install the app.

Google Groups vs Test Flight

We are using google groups to manage the distribution of the android beta app and because it’s harder to use than test flight for iOS we’ve gotten less installations. It’s fascinating how an extra hurdle in the sign up process can impact installations.

Return Logins

Our android numbers appear to be higher than usual here but I think it’s to do with the timeframe I’m collecting these numbers over. We’ve had a few people install the android app before we officially started the beta release and I think they’ve been counted in this statistics.

Tools

We are using adobe analytics, test flight and using a closed test track via google groups to help us manage the beta app releases and to gauge engagement metrics. You can read more about our general mobile app test strategy here if you like.

What were your experiences going through a beta release process? What worked well? How did you improve the process?