Tester Talks: Insights from the 2022 Software Testing & Quality Report

Check out this recording of Tester Talks: Insights from the 2022 Software Testing & Quality Report, where QA experts from the TestRail team break down the main takeaways from the report, share their unique points of view on testing in 2022, and reveal the actionable insights that QA teams like yours can use to improve your test management processes.

Meet the TestRail panelists:
Diogo Rede: Solution Architect
Shanu Mandot: Community Manager
Simon Knight: Lead Product Manager

Each panelist answered questions asked by the moderator, TestRail Content Marketing Manager Hannah Son. Watch the full replay above, or read through some of the highlights below. Download the full Software Testing & Quality Report for a more complete picture of the industry.

Question: Where do testing and QA fit within the agile and DevOps methodologies?

Diogo: I think testing actually fits everywhere. Whether it’s executing your regression, smoke tests, predefined test cases, or even exploratory testing, some other types of testing, or automated testing, testing fits everywhere.

DevOps is more of a culture and a set of practices that allow us to break the silos between development and operations. So this means that there’s usually one team that does the full cycle from development to deployment, monitoring, supporting, etc. And this is the perfect example of where we can use continuous testing techniques. So yeah, testing is basically everywhere.

Shanu: Before agile and DevOps, QA and testing were important parts of the software development lifecycle, but with agile and DevOps, we have seen the significance of QA increase tenfold. Now, QA processes are introduced in the very early stages because the time to market is reduced and all of a sudden, the quality of an application has become very important for all stakeholders. So, QA is a very integral part of DevOps, and without a solid QA strategy in place, it’s nearly impossible to deliver continuous value.

Simon: Testing is pervasive throughout the entire cycle in the same way that DevOps is a set of practices that engineers use to try and shorten the software delivery lifecycle, ultimately intending to deliver high-quality code continuously. Testing is the set of practices that aims to ensure the right thing is being built and that it’s being built in the right way. And you know, as such, compliments that set of DevOps practices in a very good way.

Question: The rise of agile and DevOps has pushed teams to automate as much as possible. What does that mean for manual testing?

Simon: I think that manual testing always has a place. Everybody tests. Developers test while they’re coding. Because they want to make sure that their code’s working. Business Analysts test their ideas and hypotheses in various ways. Everybody’s testing throughout everything they’re doing, and a lot of that testing is manual. So, testing always has a place that is not always with a dedicated tester, it’s not always with somebody who might class themselves as a professional tester or have the “tester” title.

In the agile and DevOps world, testing is democratized to the entire team and that’s as it should be. However, that challenges professional testers to expand upon their existing skill sets. I think skilled testers should definitely recognize that manual testing doesn’t scale very well, so they should learn to leverage other opportunities.

Shanu: There are a lot of biases around this topic, but from my experience, there is a need to enrich current skill sets or upgrade the tools that we have to become more tech-savvy and efficient in our testing process. Now, QA engineers have to think from a 360 point of view for all different types of testing. It’s important to have these diverse skill sets. Although it has changed, manual testing is not dead, it just doesn’t exist in its traditional fashion.

Diogo: I’ve been automating for nine years now, and manual testing is here and will be here for a long while. I have been very connected to the QA leadership in the companies I have worked for, and every time someone says, “We don’t have anything to test right now,” the problem is usually a lack of vision. It’s not that things are fully automated, it’s more that people are not taking advantage of the time that automation is giving them to learn more or do those accessibility tests that sometimes are talked about, but no one ends up doing. There’s just so much to invest in, other than just manual or even automated testing.

Question: What advice would you share with a QA team that wanted to improve their test automation?

Shanu: Not everything should be or can be automated. Sometimes it’s not a wise decision to automate. I also think that having an experienced automation engineer helps set the direction, kick-off strategies, and set best practices. It helps to have some person with that knowledge on the team to improve the process.

Secondly, domain knowledge it’s very important for creating scenarios because no matter how good you are at automation, if you are missing particular domain-specific scenarios, it won’t be worth the effort.

The third thing is, when we start automation, we want to achieve everything, so having a clear big picture in mind is very important. When building a model, we need to have very strong architecture so that it’s easily scalable when the time comes.

Diogo: My biggest piece of advice would be to plan and create the strategy before anything else. From my experience, what I’ve seen over and over is that people will face the same problems when they are automating tests that they’re already facing when they’re executing tests manually. So, make sure you have all your data sets. Make sure you have your environments ready. From a technical perspective, automation will allow you to do more stuff than you do manually, but if you mess up your data, you’ll run into trouble. It’s test maintenance hell and sometimes that even leads to abandoning automation.

Make sure there’s good visibility into your test results. If no one has access to your test results, then it’s like your automated tests don’t exist. And if they don’t exist, people won’t credit you for automation, there won’t be further investment in automation, etc.

Simon: Stay focused on what the objectives are for actually introducing and building out your test automation. Most teams want test automation in place so that they’ve got increased confidence in what it is that they’re actually about to release. So, one of the key goals of your test automation strategy should be to provide the necessary level of information to the right people at the right time. Staying focused on those kinds of objectives is helpful in steering your test automation strategy in the appropriate direction. Making sure that your tests are prioritized appropriately and targeting the right areas that are important to your business is super important.

If you’re ever in doubt as to where to start with your test automation, take a risk-based approach. Don’t focus on requirements coverage or something like that; focus on risks and start by building out your test automation focused on the highest areas of risk.

Question: What are the 2-3 primary objectives you consider when designing your QA strategy?

Simon: Increasing test coverage in the right places: There’s no point increasing test coverage if it’s not addressing the right things, if it’s not focused on the right risks, or if it’s covering things that aren’t necessarily important to the business or to the other members of your team.

Focusing on risk: Design tests to address risks, prioritize those risks, and focus on coverage of risks, rather than requirements.

And, if you have an automated test strategy, try to automate those tests at the lowest level possible that makes sense in the context of whatever it is that you’re doing.

Everybody owns quality on an agile team: The tester is not the sole person responsible for the quality of a finished product. Just because you’re doing those things as a tester doesn’t make you the person who is solely responsible. In my view, it’s the tester’s primary responsibility to make sure that all of the necessary testing is being performed, not necessarily to do all of that testing themselves or to be solely responsible for it and the QA strategy should make that clear.

Shanu: Firstly, to have a clear set of goals is very important and that mainly depends on the organization, culture, and sometimes, processes at the team and project level. Having objectives is not enough. We need to think a little more about how we can achieve these objectives. So we need to be very clear about what KPIs we will use to measure these objectives and to know whether we are progressing toward them or away from them. Second, automate when necessary. Sometimes the ROI from automation is very low and we don’t need to automate.

Diogo: One objective that I think everyone should take into consideration is reducing bugs in production. This is sort of like the Holy Grail. We know that it’s virtually impossible to have no bugs, but it depends on the type of business you’re working in. So if it’s something safety-critical, then you should probably be very focused on increasing your test coverage during every scenario and having a really, really low rate of bugs in your production environment.

The second objective is to make testing more efficient. If you make testing more efficient, then you have shorter release cycles. You can automate more tests, implement continuous integration, implement continuous deployment models, or even shift left because shift left is also about having faster feedback and fixing things sooner.