Test automation in CI

Automated tests are only valuable if they run on every change and people trust their result. In continuous integration, the test suite is a shared tool, so its speed and reliability affect the whole team.

Order for fast feedback

Run the cheapest checks first and stop early when they fail:

1. linting, formatting and type checks, taking seconds;
2. unit tests, taking seconds to a couple of minutes;
3. integration tests against real dependencies, taking minutes;
4. a small end-to-end suite covering critical journeys.

A developer should know about most failures within about ten minutes of pushing. Slow pipelines get bypassed, batched or ignored.

Parallelize and cache

• Split tests across machines (sharding) and run independent jobs in parallel.
• Cache dependencies keyed on the lockfile, so installs take seconds.
• Run only what’s affected for large repositories, using the build tool’s dependency graph, but still run everything before release.

Parallel tests must not share state. Give each test or shard its own database schema or container, and never depend on the order tests run in.

Test data

Tests that depend on shared, long-lived data break whenever someone changes it. Instead:

• create the data each test needs, in the test or through small factory helpers;
• use realistic but synthetic data, never copies of production personal data;
• reset state between tests, for example with transactions that roll back or by truncating tables.

def test_rejects_expired_coupon(db):
    coupon = make_coupon(db, expires_at=yesterday())
    order = make_order(db, total=50)
    with pytest.raises(CouponExpired):
        apply_coupon(order, coupon.code)

The test shows everything that matters to it, and nothing else.

Flaky tests

A flaky test fails sometimes with no code change. Flakiness is poisonous: once people learn that red sometimes means nothing, they stop reading failures and real bugs slip through.

Common causes and fixes:

Track flaky tests. Quarantine one (skip it in the blocking suite and open a ticket with an owner) instead of adding automatic retries that hide the problem.

Quality gates

Decide what blocks a merge and keep that list short and meaningful: tests pass, types check, no new critical vulnerabilities, perhaps a coverage threshold on changed code. Coverage is a guide, not a goal: 100% line coverage can still miss important cases if the assertions are weak.

Make failures easy to act on

When a CI test fails, the report should say what failed and why without re-running anything locally:

• clear assertion messages with expected and actual values;
• logs, screenshots or videos attached for end-to-end failures;
• a way to reproduce with one command, for example with the seed and test name printed.

Treat the pipeline like production code: review changes to it, keep it fast, and fix a red main branch before starting anything new.