How Eventbrite Built and Scaled their QA Team
Founded in 2006, Eventbrite has come to support more than 949K organizations and creators around the world, handling more than 309M tickets for 4.7M+ events in over 180 countries in 2019. Eventbrite’s laser-focus on delivering quality to customers has been a key driver of their success, which culminated in their IPO on September 20th, 2018.
The Challenge
When the QA team at Eventbrite started their free trial of TestRail in early 2016, the company had already entered the hyper-growth phase that would propel them to their stock market listing two years later. They were in a position similar to many QA teams at high-growth startups: managing their test cases through a series of cobbled-together spreadsheets.
As Eventbrite SQA Engineering Manager, Andrew Smelser, recalls, “The start of QA at Eventbrite was from the grassroots. We began with just writing tests down in a free tool like Google Sheets.”
The problem was that as Eventbrite exponentially grew, using spreadsheets to test became a bottleneck and a liability. Spreadsheets were fast and flexible, but they didn’t scale. They were hard to organize, were difficult to report on, and prone to error.
The Solution
Instead, the QA engineering team needed to find a solution that could:
Help accelerate release cycles without sacrificing quality as the pressure to develop new features—and potential risk of new defects—was only going to continue increasing.
Scale their ability to support a constantly growing engineering department, standardize tracking and reporting around key quality metrics, and help maintain clear, concise, and meaningful communication between disparate departments.
Offer a competitive price point and flexible workflow that allowed team members to onboard quickly as the team scaled up or down in response to the needs and circumstances of the business.
Andrew Smelser
SQA Engineer
Visibility
According to Eventbrite’s SQA Engineering Manager, Andrew Smelser, “As Eventbrite grew to the point where it needed a dedicated QA tool, we turned to TestRail.”
The first step was to centralize testing efforts. “We recommended that teams use TestRail because we could have all the information together in one easy-to-access place,” explained QA Engineer Diego Girotti.
Using TestRail as a hub improved visibility into testing efforts, as well as traceability and auditability of testing history. TestRail also allowed those outside of QA and development greater insight into Eventbrite’s testing work and key business metrics such as test coverage, regression testing percentages, and automation.
Smelser summarized: “You gain a lot of visibility when you have this platform versus doing it in sheets with varying permissions.”
Flexibility
One of the most significant advantages of TestRail over spreadsheets was its flexibility and intuitive workflow. Even though some of the software developers on the team were resistant to using a new tool, TestRail’s ease of use won them over, particularly with the ability to import tests from external sources.
According to Smelser, “TestRail is not prescriptive,” and does not demand one rigid method of use. Individual teams at Eventbrite use TestRail in the way that suits them best, while still managing to track and store all of their test artifacts in a single tool.
This freedom was a massive benefit to Eventbrite. Utilizing TestRail did not necessarily mean replacing previous workflows entirely; rather, TestRail provided an extension of those workflows and helped to make them more efficient.
Compatibility with Test Automation
For the Eventbrite team, this meant that their overall test management strategy has remained stable, consistent, and dependable, despite changing tools and environments. As Eventbrite moves forward, TestRail provides the stability to support company initiatives like increasing test automation and implementing a CI/CD pipeline.
Customers want to know if their projects are ready to ship and TestRail allows the team at Cardinal Peak to demonstrate –with confidence- that they have coverage of the project’s unique requirements and that the product has passed all tests.
Pricing
For Eventbrite, this meant that the team could scale quickly without having to slog through lengthy pricing or contract negotiations. They were never “locked-in” to any user count; they paid for what they used, and as new business challenges arose, they could adjust quickly.
According to Smelser, “By being able to prepay, and then deduct monthly charges against that balance based on how many seats we actually used, we’ve been able to stretch our contract a couple of extra months. And obviously, the opposite has been true for us too: we’ve eaten our balance quickly sometimes because we grew so quickly. But we’re really happy with that flexibility.”
Andrew Smelser
SQA Engineer
Shift in the Culture around Quality
While visibility and flexibility were significant benefits for the Eventbrite team, the real value of TestRail came from the fact that adopting TestRail helped the team successfully change the culture around quality across the company.
“At the business level, we’re shifting the culture of ownership around quality,” explained Smelser. “It is a big part of our agile initiative: that quality belongs to everyone. The ideal is that all development teams own quality.”
This movement — for quality to be owned by every developer, and not strictly by QA — took shape in a push to “shift left” testing across the board. “The goal is to be testing earlier and more frequently,” explained Smelser. And at the center of that shift was TestRail.
TestRail empowered the Eventbrite QA team to collaborate with Product and Engineering to scope new sprints and build quality into the development process from the outset. Getting developers involved in testing also helped focus their development on real outcomes for end users, leading to better code, faster releases, and increased value to the customer.
This new paradigm shifted the role of the QA members within the organization. It allowed the QA team to contribute at a larger scale through the formation of an “on-demand” testing team that could assist with the quality assurance needs of multiple teams across the organization. If certain functional areas or integrations seemed to be taking too long to test, failing repeatedly, or resulting in higher numbers of customer-reported incidents over time, the on-demand team could “jump into TestRail and see what [the team was] testing,” according to Smelser, and advise on different strategies to make improvements.
Finally, by standardizing test management in a centralized system and giving everyone on the team access, the Eventbrite QA team was able to better promote their goal of constant improvement around testing and QA. By reviewing metrics around the quantity and severity of defects identified across various sprints using TestRail, the team could estimate how long it would take to develop, test, and ship high-quality releases more accurately.
They were also able to include more cross-functional stakeholders in conversations around quality more easily, connecting the dots between product, engineering, QA, and support to analyze the impact of new changes on the overall quality of Eventbrite’s ecosystem of apps.
The Results
TestRail’s flexibility and usability allowed cross-functional team members across Eventbrite to quickly onboard new users and integrate new development tools while maintaining a stable but fast-paced testing environment.
Tracking all manual and automated testing operations through one central platform increased visibility into testing as a whole, and helped the team monitor the quality of their products with each new release.
This visibility then shaped business-level conversations around Eventbrite’s overall software development and delivery process itself, facilitating a culture shift and driving more meaningful adoption of agile processes across the team. Getting the whole team involved in testing and QA allowed Eventbrite to scale more efficiently while constantly improving the quality of the product for end-users (even as the Eventbrite’s TestRail user count scaled more than 400% in less than four years).
Ultimately, TestRail continues to be the right test management solution for Eventbrite for three reasons:
TestRail has been able to scale with Eventbrite’s test management needs, from high-growth start-up to publicly-traded company.
TestRail provides a balance between features and usability that makes it easy to use for Eventbrite’s QA & Engineering teams.
TestRail has a flexible, competitive pricing model that works in Eventbrite’s favor.
Andrew Smelser
SQA Engineer
Join over 10,000 QA teams using TestRail to release flawless products, faster
3D Systems
Solitea
