How We Mastered Unit Testing at Our Startup

Problem Statement:

At a startup where I joined as the 4th engineer, we faced a production issue every other weekend, which would be every 4th release, primarily due to new team members lacking context on different workflows. Adding documentation for the product and code was necessary but insufficient to ensure quality.

Solution:

Given our team size of 6-7 engineers, I identified unit testing as the most effective solution. Consequently, I introduced the first set of unit tests for the backend and created testing pipelines on gitlab. Created a plan around the same considering the constraint that It was also essential to onboard engineers to this new process with minimal impact on KTLO (Keeping the Lights On), ensuring no disruption to planned projects and timelines.

The plan:

• I received a go ahead from the Head of Engineering for this.

• I devised a phased approach to facilitate a smooth transition:

  1. Initial Phase: Require at least one test case for a merge request (MR) to be approved.

  2. Intermediate Phase: Increase the requirement to a few test cases for MR approval.

  3. Advanced Phase: Ensure all new functions, classes, and APIs are covered with test cases.

  4. Final Phase: Mandate that all code changes are covered with test cases.

Execution

Since I was reviewing most of the MRs soon into the company, it was easy for me to ask the developers to add test cases. I provided references on how to do this with the test cases I had added and sometimes sat with the developers to guide them through the process. This approach was mostly effective. However, about a month into the process, a developer faced a significant challenge. He had a substantial set of changes dependent on a lot of existing code, which required writing factories for numerous models to create a few test cases. This raised the question of whether it was worth investing time to add test cases immediately or defer them. In cases when you are trying to create a mindset shift or a change in process, it’s worth having an example of going above & beyond to be ingrained in the team's culture, so I collaborated with them to ensure this was done.

Progress and Impact on the Codebase:

The codebase was built over four years by three engineers, before the initially had about 28% test coverage since Python unit testing assumes coverage of function and class definitions. In the first quarter, much of the effort went into creating the base and factories, reaching approximately 35% coverage.

In the second quarter, we started seeing significant improvements. The increased emphasis on unit tests and the detection of breaking code changes helped us achieve about 45% coverage. Over the next two quarters, we continued to make progress, achieving around 10% and 7% additional coverage, respectively, bringing us to 62% coverage after one year.

While the percentage coverage increased gradually, we focused on improving the quality of the test cases themselves, covering more scenarios, and adding API integration tests. In the following 1.5 years, we reached 78% coverage, resulting in a fairly stable backend for the product.

I am executing a similar plan at another b2b software startup with a larger engineering team. Would share new learnings from that as a separate blog post.