Why bother testing?

As a tester, I don’t practice selling my craft very often. This blog post is an attempt to put together my thoughts when asked the question, “why bother testing?”

Let’s talk about bugs

When was the last time you used some buggy software? How did it make you feel? Did it cause you to swear at your computer in frustration? We are surrounded by software full of bugs, some bugs are minor but some cause us to pull out our hair in frustration. Some bugs when discovered cause nightmare headlines to spread like wild fire. Buggy low quality software is hard to sell, maintain and keep people using. Your customers won’t rave about your products to their friends if they think it’s shit.

Your customers won't rave about your products to their friends if they think it's shit

Testing doesn’t improve quality

Testing itself doesn’t improve the quality of the product, but it can help highlight issues that could impact the perception of quality. Bugs will always exist in software, it’s impossible to test for every possible scenario every time before releasing to production (especially in a world of continuous integration/deployment) but if you knew about some the bugs in you products before your customers find them then you’d be in a better place to make informed decisions about what to release and when.

Testing is active discovery

Everyone does testing on some level, most of the time we are unaware of what we are doing until we discover a frustrating situation. Testing is a skill where you practice looking for quirks in software. Testers are constantly experimenting and observing the product and are well practiced in talking through their thought processes.

There are generally 2 activities people say they are doing when they talk about testing;

1 – Verify the product works as intended

2 – Actively go hunting for bugs

Activity 1 is usually where people talk about automation, when your product can be codified as “working as intended” you might be in a place to build some automation checking to help facilitate faster feedback. However you could build all of the automation checking in the world into your products but people can still think it’s shit. You users don’t care that your unit tests are less than 0.01% flakey.

Activity 2 can be easily practiced, it’s not like anyone knows where the bugs are in software until they are discovered. You can build things into your products that make it easier for people to report the bugs they find or you could do a chaos monkey approach to find out where your product crashes unexpectedly.

There are many other activities involved with testing but I want to keep this high level.

People say testing is risk mitigation

This is not true, testers may use their own internal risk radars to help guide their testing efforts but testing itself doesn’t mitigate any risks. I like the analogy; testing is like an x-ray, an x-ray on it’s own can’t tell you how healthy you are or prevent issues from happening but it’s a tool that can give you a snapshot in time so your doctor can make informed decisions about some element of your health.

Testing can help people answer questions like, “are we comfortable shipping this code to customers?”. Bugs are a fact of life but we can’t fix what we don’t know. Testing helps us discover these potential issues.

Running a Bug Bash

For any solo tester out there I recommend leading a regular bug bash/mob testing activity. It’s an activity you can run at the end of a sprint/feature dev cycle. You invite the team, get some snacks/beverages together and get everyone testing for around an hour.

Setup

You might want to make sure you have good device/browser coverage by ensuring everyone is set up with data/devices before hand. You might also want to prepare some help docs to help people get started. I like to have a mindmap on coverage ideas prepared before hand so people have a visual indicator of what they are testing. I might also create a bunch of test accounts before hand and distribute them to the team.

I use this whiteboard for running bugbashes at Insight Timer; regression on the left, what’s changed recently in the middle and “other” things to consider on the right

Light Bug reporting

You should encourage light weight bug reporting;

  • maybe people write bugs on sticky notes
  • add bugs to a shared spreadsheet
  • or raise Jira bugs directly from Slack using a /jirio create bug shortcut

The focus should be on finding bugs, not getting caught up in how to report them. You can always clarify issues further after the bug bash if you need more information.

What to test in a bug bash

I have a 3 step heuristic for deciding what to test (think; guide or rule of thumb):

  1. what’s changed recently? Change introduces new risk. what features have been built? What code has been refactored? What libraries have been updated? Focus the team to test these areas first
  2. Regression testing. Have a checklist of at most a dozen scenarios of core functionality (maybe around registration, payments and the main stuff people do with your product). Ask people to consider these cases while doing 1), maybe ask people to put there name next to a case when they test it so you can get an idea of coverage
  3. What other testing should/could we do? Sometimes it might make sense to do a pen test bug bash session or a performance testing bash or a cross browser test bash but it doesn’t have to happen every time

Test in Production

If you can; test in production. There’s no environment quite like it.

Put on your ruby red slippers and repeat after me, "there's no place like prod"
Toto, I’ve a feeling we’re not in Kansas anymore – Wizard of Oz

But if you can’t test in production, make sure your test environment is stable, your data is set up and you’ve given your team the “Do Not Touch except for bug bash testing” request.

Post Bug Bash

Make sure to thank people for their time, count up the bugs and give kudos to your best bug hunter. Testing/Quality is a team responsibility after all. Send out a thank you email with results and award trophies. I’ve handed out this trophy of a rhino bug in resin on a trophy base to our number one bug hunter before:

rhino bug in resin on a trophy base
Bug Bash Trophy

What is software testing?

A quick google of ‘What is software testing’ will give you 75,300,000 results but not much of an answer. On the surface level, we can surmise that it is about testing software but in reality; it’s more than that.

Software testing is an activity that everyone does on some level. E.g. when you interact with a product like Facebook you are helping to test it. You have certain expectations of how the scrolling behaves and what type of content comes up in your feed. You are probably a part of several on going experiments that Facebook conducts and you’re scrolling habits are also helping shape their performance testing approach.

Q. Hang on, if everyone does it, why do we need dedicated software testers?

A. Justifying the value I add to a project can be challenging sometimes but let me try to go into more detail. There are some very successful companies out there that don’t hire dedicated testers but their context and software development practices can help mitigate some of the risks associated with their products.

Software testing is about reducing the risks associated with software development. Software development is, at its core, a business venture. The software is created to solve an issue that the end user is willing to pay money for. Each element within the development lifecycle is working towards providing a marketable solution to that original problem. Software testing serves to highlight any risks that might impact on the value of the product.

The software development cycle is a continuous iteration of identifying the customer’s problems, developing the solution, ensuring the solution works, and selling the solution. Software testing ensures that our apps, websites and software does what it’s supposed to do whilst not doing what it shouldn’t. It is an integral part of any successful software project.

Planning

Good testing can be completed with very little “formal” planning procedures if a skilled tester is involved. You can see an example test report generated from 1 day of testing from James Bach for testing a medical device.

Often the waterfall model of software development is associated with exhaustive requirement gathering and testing planning even before the software is designed. On the contrary, the agile manifesto focuses on providing working software over comprehensive documentation. The whole Agile vs Waterfall is a conversation for another blog.

When working within the agile methodology we can move away from explicit and highly specific planning and into the rapid software testing realm. James Bach and Michael Bolton are pioneers in the Rapid Software Testing space. This is a context-driven software testing methodology that works on the premise that the testing team will have an understanding of the business problem that the product is solving.

This Rapid Software Testing method allows the testing team to conduct a more informal, risk-based approach to planning the testing of software. Sam explains how to take a risked-based approach to planning UI tests in her /dev/world presentation.

Exploratory Testing

Exploratory testing* is where a tester interacts with software to compare it with their mental models of expected outcomes.

Exploratory testing is a part of all testing regimes because it focuses on human-based feedback. Having people run through the software with intention of finding brokenness helps to highlight any design, UI or experience elements that are confusing or hard to use. This is something that automated testing scripts aren’t able to handle.

Exploratory testers will explore how well the software solves the end-users problem. They will highlight any high-risk issues that will impact the value of the product. Skilled exploratory testers will work on a continuously iterative testing approach where the exploration can help form a plan that guides the direction of future testing efforts. This continuous feedback loop allows testers to improve the quality of their testing rather than just the quantity.

Exploratory testing is based on human observations. This is incredibly useful for user experience focussed testing but less practical for testing that needs to be repeated over time or scaled up to assess performance.

Whilst the benefits of having human based feedback far outweigh the cons it is important to note that human error will always occur. Hopefully using a risked-based approach with an experienced exploratory tester will negate some of these issues before your users find them.

Automated testing

Automated testing (sometimes called checking) involves using software tools or scripts to compare the expected results for how a program should run with actual outcomes. It leverages the computational power available today to automate testing that would have to have previously been conducted manually.

definition of automated testing; involves using tools or scripts to help asses expected behaviour of software

For example, an automated testing suite may be used to check that ‘a + b = c’. To test this we could run a sample of known numbers through the function and ensure that the outcomes were as expected.sudo code for 3 unit tests

This is an extremely simplified example of automated testing but this idea can be expanded to include all of the predefined actions within a software.

Automated testing has numerous benefits including increasing the amount of test coverage available within a budget or timeframe, automating basic or low-level tasks, reducing human error when checking outcomes, reusable tests and leveraging the computational power of machines.

If automated testing suites aren’t maintained in line with the development cycle it can lead to outdated tests that return incorrect outcomes. This can lead to less secure testing coverage and a loss of team morale and trust in the testing cycle.  Additionally, automating poorly designed tests only increases the rate that these misaligned tests are run.

Automated testing has been hailed as the holy grail of the software development world of late. It fits in well with the continuous integration methodology and allows for quick iterations of products. However, it’s important to keep in mind that automated testing isn’t a magic bullet. Instead, automated testing should be used in conjunction with other tools to gain more time for your testers to focus on high-value tasks that are outside the scope of automation.

Unit Testing

Unit testing is about isolating the components in your software and testing them individually.

definition of unit testing, isolating components to test individually

Unit testing is a powerful tool when used early in the development cycle. This is a tool often employed by developers to ensure that the code they’re writing works.

Like all of the tools that we have explored today unit testing in only one option that a development team will employ during as part of their testing process.

Conclusion

Anyone doing testing has a plethora of tools available to them when conducting a thorough testing cycle. Often we get caught up in the discussion of Exploratory Testing vs Automated Testing but we need to bring this back to what is the tool that will best fit this testing scenario. I’ve only scratched the surface here of what can be involved with software testing.

Software testing is a continuous feedback loop. Everyone does testing on some level, and one of the goals of testing is to highlight any issues or risks that might impact the perceived value of the product. Ultimately we want to help deliver quality products on time and on budget.


*The intricacies about whether exploratory testing should be referred to as ‘testing’ has been discussed extensively within the software testing industry. Please note: for ease of understanding Sam will be referring to all testing without scripting as ‘exploratory testing’ within this article.

**Sam commissioned Stefanie C. from upwork to help write this article, there might be a follow up blog on Sam’s experiences using freelancers to help outsource work later and why she would do this

Response to “the end of manual testing”

This blog is a response to this blog by Michael Bolton titled “the end of manual testing”. While attending EuroSTAR in Copenhagen in 2017 I had the joy of having a good chat to Michael about this topic.

Let me speak from my experiences; I’m bored in my current role and I’m looking for a new job. I have recent experience in observing what the Sydney market is demanding vs how I view myself. If I could choose my own label I would call myself a product risk investigator because it reflects my views in the value of what a tester brings to the table. Now I personally don’t refer to myself as a “manual tester” even though most of my work has fallen under what the market would call “manual testing”.

So how do I market myself for new job prospects? You can check out one of my recent software testing resume’s here. Did you notice that I never refer to myself as a “manual tester”? I try to highlight my technical skills in roles that the market would call “manual testing” because of the negative conations associated with the label of “manual tester”.

As part of my job hunting efforts I reached out to the recruiter who placed me in my current role. He seems to have a good pulse on the Sydney job market for testers; I reached out to him on October the 31st and the next day he had scheduled me a job interview with a startup in Sydney. He then secured me 2 other interviews and I had 3 interviews in 3 days just before I came to Copenhagen for Eurostar this week. I’ve already received positive feedback from 1 of these interviews. However all 3 of these roles have had a focus on trying to find a technical tester who can help the company test API’s through some sort of coding efforts. No one appears to be hiring a pure “manual tester”. Some job descriptions that I’ve read have said stuff along the lines of, “some manual testing will be required but the focus of this role is definitely writing code”. These type of job descriptions trigger warnings in my head along the lines of, “maybe these guys don’t understand skilled testing?”. I almost feel like this phrase “manual testing” is a dirty word. Maybe I should just come out as a loud and proud Manual Tester? Next week I’m aiming to have 3 more interviews before assessing all of my options and moving forward. These next 3 interviews have been set up via applying to jobs via LinkedIn/Seek/Career portals, reaching out to an old colleague and reaching out to recruiters who are involved in the meetup space.

In Summary; I’m being pushed towards more technical testing because those skills are more marketable and I’m struggling to market myself as a skilled exploratory tester. I agree with Michael; “manual testing” doesn’t exist but finding a company who shares my understanding of testing is pretty hard. Now why is that?

I don’t enjoy my job

I have a few issues with my current job:

  1. I feel insanely guilty for not finding my current role to be the most exciting thing I’ve ever done. I’m contracted out to a client, and I can’t officially say who the client is but there is just so much hype around this client for supposedly being the best tech company to work for and sometimes it doesn’t always match that hype. I find my work boring, a little repetitive and not exactly challenging me. However it’s meant to be the highlight of my career. And when anyone asks me, “how’s working at X?”, I feel like I should respond with, “it’s amazing…” and go on to list all of the amazing benefits.
  2. The scale of the Client makes it hard to feel like an essential part of a team.
  3. In my current position it would be nearly impossible for me to convert to a full time employee and I don’t even know if I want to pursue that path.
  4. My current employer isn’t as supportive of my conference speaking engagements as I’d like them to be.

 

Don’t get me wrong, there are many things I enjoy about the role;

  1. My favourite thing about my job is getting to tell people that I work on a product that nearly every one uses. I’ve always enjoyed working on tangible products.
  2. the free food/coffee barristers is pretty sweet too.
  3. Being surrounded by smart people who care about the products they are building.
  4. It’s an amazing thing for my personal brand

 

I also feel like I haven’t been in my current position long enough to really justify looking for a new job. I’ve only been on the Clients project for around 10 months. I know there’s a bit of a stigma for changing jobs too often. This thought also makes me feel like there’s something wrong with me for wanting to change. I’ve experience some mental stress in my current role that’s not really founded in much; from bouts of imposter syndrome to not being motivated to work.

I think I’ve come to realise that the following things would be important for me when I do look for a new job:

  1. Feeling a part of a well gelled team, I know this takes work and I always go the extra mile to build a team culture. My energy for work comes from other people. I’m not that type of “highly motivated self starter” person who is satisfied with the sit in the corner on a computer all day doing work type of role.
  2. Being in an environment where I feel comfortable being vulnerable. E.g. I’ve had struggles with depression so being open about mental health is important to me.
  3. Getting supported for community engagement, e.g. having speaking at conferences as part of my role would be an ideal thing to have.
  4. Be in a learning environment, there are many companies that say “we are constantly learning” but often don’t put that into practice. There’s often a disconnect between what companies say they do and what they actually do.

 

I think I’m in a really good position to be very picky about my next role. I can see 2 potential and appealing paths forward. Either get more technical or get more people oriented.
By technical I mean I could start leaning towards mobile app development, growing my interest in Data Science, picking up some more DevOps skills, get into test automation etc.
By people orientated I mean moving towards product development, maybe picking up some user research skills, maybe looking for something with a support focus etc. It would be awesome to incorporate some sort of outreach/education efforts as part of my next role.

While going through highschool/uni I worked in supermarkets for 7 years. I actually think that work was more engaging than most of my work in tech has been. I’m missing that strong connection to people in my current role.

So I’ll be keeping my eyes open to any new positions and I have a good sense of what I’m looking for.

Agile and Stress

I was having a chat with an old colleague on LinkedIn today (Brian Osman) and we were talking about Agile. The question was, “how does Agile at Tyro differ from <Client>*?” My conclusion was that by focusing on certain “Agile” rituals an artificially high stress environment can be easily created.

Here is this conversation:

Bosman:

Samster! How’s it going at EPAM? All good I hope 🙂 . Hey just a question – do you guys *do* agile and if so how does it compare to Tyro?

 

Samster:

There’s lots of people in EPAM who do agile but in the <Client> office in my little corner there is no mention of agile, no Kanban boards, no daily standups (in my team). Every team is encouraged to develop their own process. E.g. my manager at <Client> often doesn’t come into the office until 10:30-12pm so a 9am standup just wouldn’t work. He has kids and often works from home. I know the EPAM cloud support guys we have here in Sydney have standups and mostly deal with support tickets so their work is fairly regular.

So with my manager we have a sync doc that I fill out every day on what I’m working on with a breakdown of roughly how much time I’m spending on tasks. Roughly half my time should be spent on bug triaging and trying to mainstream that process across some of the Sydney teams (about 70 developers/designers/PM’s and researchers) and the other half of my time spent on helping those teams with feature testing. This sync doc is in place of a daily standup and we have a sync up once a week where we go through it and look for improvements. There’s no sprints, no regular retrospectives. The <Client> guys will tend to work based on quarters (3 month cycles). Higher up management will set certain goals for the quarter and every team will set their own goals that hopefully line up with those.

So atleast in my little bubble at <Client> there is very little”formal” Agile but product excellence is a heavy part of the culture.

The funny thing with Tyro’s agile is it didn’t really support flexibility. A bit of an oxymoron in a way.

The context of <Client> and Tyro is vastly different too. Often the guys in the Sydney <Client> office have to collaborate with people all over the world, so they will often have conference calls with guys from Mountain View in the morning, often taking these calls at home and maybe conference calls with India in the afternoon. I have a weekly sync with one guy in India who helps lead the team that is responsible for communicating with public transport providers. The ops team for <Client> project that I’m working on is based in Seattle.

On a side note, I have heard stories of <Client> bringing in agile experts for coaching/consulting to learn about it. They usually bring in the guys that write the books on the stuff because <Client> has that kind of pull.

 

Bosman:

That’s awesome! It’s something like I just read from a <Client staff member> and something I like…lower case agile. Figure out the problem yourself and make your own rules (but don’t break the law….and be nice!). Thanks Sammy! I’m writing a report for a team I’m coaching at <Somewhere> and they want to compare this with places like <Client>. I think they’ll be surprised 😊 .

 

Samster:

Yeah. Otherwise things are going pretty well. No one has kicked me out yet. There have been a few days where I felt I wasn’t good enough to be at <Client> but it wasn’t really as stressful as starting at Tyro was. I think a lot of companies aspire to have a culture like <Client> but that’s actually pretty expensive, but nicely liberating. I think companies also often miss the point of <Client>’s culture, I guess they see the free food and games rooms and think that’s all you need to emulate <Client>’s culture.

I think the whole point of agile, is the being able to adapt to a rapidly changing environment. In the context of <Client>, sure there are some areas of the business that are constantly evolving but focusing more on product excellence tends to put a slower spin on things. There’s a focus on facilitating fast feedback but it’s to help make work easier, not to adapt to the market. There’s no mantra of ship it quickly, but there is a huge amount of effort put into supporting engineers get new code into production. Even <Client> uses a giant spreadsheet of manual test cases for Android/iOS <Client> maps apps on top of all of their other checks and balances.

There is a lot of process involved with getting a new feature into <Client> maps.

Maybe the over emphasis on”Agile”, and adapting quickly creates an artificially high stress environment. There’s always this push to beat competitors to market, to get this work out the door this sprint, to be constantly, “Go Go Go”. People don’t function well under stressful situations and that constant stress can’t be that healthy.

 

Bosman:

I agree with your point …. I don’t know if they’ll meet their growth targets and I don’t know if staff will feel happy about being ALWAYS under the pump so to speak…

 

*<Client> = I work client side as a contractor. I can’t officially say who this client is (because contract) but I get to help test an android app you might use for mapping and navigation. The Client is pretty well renowned in the tech space.

Where’s your local experience?

We had an amazing Sydney testers discussion group today on sharing stories around our job hunting experiences. One comment that came up was on local experience and I wanted to elaborate a little more on it.

I would say there are atleast two ideas behind why someone would care about local experience. Unconscious bias and culture fit. I have helped with interviewing some people in my last company so I have some insights to share. I would say that unfortunately there is some bias against people who don’t have English as their first language. People will use the rationale of poor culture fit to justify their unconscious bias.

I would say that unfortunately in Australia we have a bias against people from India, the massive call centre outsourcing efforts have not left us with great impressions. So there is an association with strong accents and having many issues with communication. Someone who’s applying for a job here with an Indian background just has a few extra hurdles to overcome, it’s not really fair but by asking for local experience we can justify to ourselves that you wouldn’t be a good cultural fit. We also imagine that you might be coming from a larger hirachical corporate structure, where there is a lack of innovation/self motivation. We imagine that if you did have an issue with your boss here you wouldn’t let anyone know about it because that’s what we understand about your culture.

Agile

change is the only guarantee in life, how can being agile help you; add value, facilitate fast feedback and imrpove processes

Here’s my rant on Agile:

Agile is a bit of a buzz word these days. If you aren’t doing Agile, there is a tendency to think you are missing out. At it’s core Agile is about adapting to change. As a tester, I’m often asking myself, “How can I add value?”. You can be doing all the Agile in the world but if people aren’t collaborating, you are still going to have a shitty experience/product. One way we testers can adapt to change is to facilitate faster feedback. Tests/checks that run during the build phase are an examples of facilitating faster feedback, continuous integration/delivery is also another example.

There are a lot of Agile ceremonies out there and people saying they have the “Best” way to do Agile. THERE IS NO BEST WAY! But Agile is about people, collaboration and products that add value. You don’t have to have an Agile business to try and introduce some Agile practices into your team. I enjoy the Agile ceremonies that encourage collaboration, reflection and adapting to a changing environment.

Some people think Agile is anti-process. This is not true in my experience. It can be some of the most process driven work that I’ve been exposed to but it’s about creating and tweaking the right process for the right context. Ensuring that any process adds value and making sure it supports collaboration rather than hinders it.

I think businesses have a tendency to use Agile as an excuse to change requirements at the last minute but still expect high quality software. Changes need to be backed by evidence (validated learning) and have buy in from the teams building it and cannot change at the whims of some manager. And don’t get me started on the over emphasis of Automation Testing as the only “Valid” form of testing in many Agile development teams.

/End rant

Do you have any rants on Agile you’d like to share? What has worked for you? What hasn’t worked so well?

My take home message is to question the value you add and also seek out ways to facilitate faster feedback.

Further Reading:

Agile Manifesto