How We Mastered Unit Testing at Our Startup

Chirag Choudhary — Sat, 01 Jun 2024 18:30:00 GMT

Problem Statement:

At a startup where I joined as the 4th engineer, we faced a production issue every other weekend, which would be every 4th release, primarily due to new team members lacking context on different workflows. Adding documentation for the product and code was necessary but insufficient to ensure quality.

Solution:

Given our team size of 6-7 engineers, I identified unit testing as the most effective solution. Consequently, I introduced the first set of unit tests for the backend and created testing pipelines on gitlab. Created a plan around the same considering the constraint that It was also essential to onboard engineers to this new process with minimal impact on KTLO (Keeping the Lights On), ensuring no disruption to planned projects and timelines.

The plan:

I received a go ahead from the Head of Engineering for this.
I devised a phased approach to facilitate a smooth transition:
1. Initial Phase: Require at least one test case for a merge request (MR) to be approved.
2. Intermediate Phase: Increase the requirement to a few test cases for MR approval.
3. Advanced Phase: Ensure all new functions, classes, and APIs are covered with test cases.
4. Final Phase: Mandate that all code changes are covered with test cases.

Execution

Since I was reviewing most of the MRs soon into the company, it was easy for me to ask the developers to add test cases. I provided references on how to do this with the test cases I had added and sometimes sat with the developers to guide them through the process. This approach was mostly effective. However, about a month into the process, a developer faced a significant challenge. He had a substantial set of changes dependent on a lot of existing code, which required writing factories for numerous models to create a few test cases. This raised the question of whether it was worth investing time to add test cases immediately or defer them. In cases when you are trying to create a mindset shift or a change in process, it’s worth having an example of going above & beyond to be ingrained in the team's culture, so I collaborated with them to ensure this was done.

Progress and Impact on the Codebase:

The codebase was built over four years by three engineers, before the initially had about 28% test coverage since Python unit testing assumes coverage of function and class definitions. In the first quarter, much of the effort went into creating the base and factories, reaching approximately 35% coverage.

In the second quarter, we started seeing significant improvements. The increased emphasis on unit tests and the detection of breaking code changes helped us achieve about 45% coverage. Over the next two quarters, we continued to make progress, achieving around 10% and 7% additional coverage, respectively, bringing us to 62% coverage after one year.

While the percentage coverage increased gradually, we focused on improving the quality of the test cases themselves, covering more scenarios, and adding API integration tests. In the following 1.5 years, we reached 78% coverage, resulting in a fairly stable backend for the product.

I am executing a similar plan at another b2b software startup with a larger engineering team. Would share new learnings from that as a separate blog post.

Add Django Coverage and Quality pipelines in Gitlab

Chirag Choudhary — Tue, 02 May 2023 19:24:41 GMT

Introduction

If you have a Django project and you want to leverage Gitlab's pipelines to see your test cases running or see the code coverage percentage. This article will guide you step by step.

Code Coverage setup

For the code coverage you need to create a gitlab.yml file in your repo & add the following code:

coverage:
    stage: test
    only:
      - merge_requests
    services:
      - mysql:5.7
    before_script:
        - python3.7 -m pip install --upgrade pip
        - python3.7 -V
        - pip3.7 install -r requirements.txt
    script:
        - coverage run -m pytest
        - coverage run manage.py test

This is assuming that you are using MySQL service for the DB, if you use a different DB or don’t use one feel free to remove it.

With this you should be able to see the pipelines running & the logs below:

But wait, where is the coverage percentage?

For that, we need to enable the coverage report with coverage report -m

This would help you see the coverage of each file

Hmm… it is good but hard to read. Let’s generate the report in HTML, make it public & export it to GitLab's artifacts:

coverage:
    stage: test
    only:
      - merge_requests
    services:
      - mysql:5.7
    before_script:
          - python3.7 -m pip install --upgrade pip
        - python3.7 -V
        - pip3.7 install -r requirements.txt
    script:
        - coverage run -m pytest
        - coverage run manage.py test
        - coverage report -m
        - coverage html
    artifacts:
        paths:
            - htmlcov/
    when: always

pages:
    stage: coverage
    dependencies:
        - coverage
    script:
        - mv htmlcov/ public/
    artifacts:
        paths:
            - public/
    when: always
    only:
      - merge_requests

Please note that we are only running the pipelines when a merge request is raised or a change is made. Not after the MR is merged.

This helps us see the job artifacts with line by line covereage for each merge request.

So this is a lot of clicks to see the coverage, let’s get it parsed by cobertura (fun fact! It's Spanish for coverage) & into the GitLab UI. Let’s update the gitlab.yml, we are adding the xml report & cobertura specific report.

coverage:
    stage: test
    only:
      - merge_requests
    services:
      - mysql:5.7
    before_script:
          - python3.7 -m pip install --upgrade pip
        - python3.7 -V
        - pip3.7 install -r requirements.txt
    script:
        - coverage run -m pytest
        - coverage run manage.py test
        - coverage report -m
        - coverage html
        - coverage xml
    artifacts:
        paths:
            - htmlcov/
        reports:
          cobertura: coverage.xml
    when: always

pages:
    stage: coverage
    dependencies:
        - coverage
    script:
        - mv htmlcov/ public/
    artifacts:
        paths:
            - public/
    when: always
    only:
      - merge_requests

We need to then configure the Gitlab repo’s settings > CI/CD

https://gitlab.com/codebards/uplevel/-/settings/ci_cd

With this you should be able to see the coverage for each merge request:

References:

https://docs.gitlab.com/ee/ci/unit_test_reports.html

https://docs.gitlab.com/ee/user/project/merge_requests/test_coverage_visualization.html
https://stackoverflow.com/a/56675636/2084765

Code Quality

For adding a code quality pipeline in your repo add the following in the Gitlab.yml

code_quality:
   rules:
     - if: '$CODE_QUALITY_DISABLED'
       when: never
     - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' # Run code quality job in merge request pipelines
     - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'      # Run code quality job in pipelines on the master branch (but not in other branch pipelines)
     - if: '$CI_COMMIT_TAG'                               # Run code quality job in pipelines for tags
   artifacts:
     paths: [gl-code-quality-report.json]

Create a codeclimate.yml file in your repo & add the following content

plugins:
  pep8:
    enabled: true

Reference: https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html