FlakeProof

Master the art of eliminating flaky tests. Learn Zig systems programming and Playwright trace debugging from an expert mentor who has diagno...
Caloocan, PH
Created byProfile pictureatticbulk
1 joined
Profile picture
@atticbulkProfile pictureJun 10
Pinned post

Welcome to FlakeProof — Start Here

Welcome to Zig × Playwright: Trace-Debug Flaky Tests.


You're here because flaky tests are costing your team real time, real trust, and real money. Let's fix that.


How This Course Works


18 lessons across 5 chapters, building from foundations to a real-world capstone project:


  1. Foundations — Understand what makes tests flaky and why it matters

  2. Playwright Trace Viewer — Master the tool that turns invisible failures into visible evidence

  3. Zig for Test Infra — Build fast, memory-safe debugging tools with systems-level precision

  4. Combining Zig + Traces — Automate flake detection and reproduction

  5. Advanced Patterns + Final Project — Fix real flaky suites and prove it with a 50-run zero-failure test


Lessons unlock sequentially. Complete each one before moving on.


What You'll Need


  • Node.js 18+ and npm

  • Playwright installed (npm init playwright@latest)

  • Zig 0.12+ ()

  • Docker (for the environment parity chapter)

  • A test suite with flaky tests (or use the provided practice repo)


Get Help


Drop questions in the Mentor Chat — I respond within 24 hours on weekdays.


Your Goal


By the end of this course, you'll have a repeatable system: Trace → Analyze → Categorize → Fix → Verify. No more re-running CI and hoping for green.


Let's kill some flakes. 🛡️

Profile picture
@atticbulkProfile pictureJun 10

The 3 Categories of Flaky Tests (and How to Identify Which One Is Killing Your CI)

Every flaky test falls into one of three categories. Once you know which category you're dealing with, the fix becomes obvious.


I've debugged flaky test suites at scale for years. Here's the framework I use every time.


---


Category 1: Timing Flakes


Symptom: Test passes locally, fails in CI. Or passes 8/10 times.


Root cause: Your test assumes something will be ready "by now." Under load, "by now" is a lie.


How to identify:

  • The failing assertion involves UI state after a user action

  • Adding a sleep(3000) "fixes" it

  • Failures correlate with CI runner CPU load


The pattern:

// ❌ This is a timing flake waiting to happen
await page.click('#submit');
await page.waitForTimeout(2000);
expect(await page.textContent('.result')).toBe('Success');


The fix: Replace arbitrary waits with deterministic ones:

// ✅ Wait for the actual condition, not an arbitrary duration
await page.click('#submit');
await expect(page.locator('.result')).toHaveText('Success');


Playwright's web-first assertions auto-retry until the condition is met or timeout expires. Use waitForResponse when the flake is network-dependent.


---


Category 2: State Flakes


Symptom: Test passes in isolation, fails when run with other tests. Order-dependent.


Root cause: Test A leaves data behind. Test B assumes a clean slate.


How to identify:

  • Run the failing test alone — it passes

  • Run it after a specific other test — it fails

  • Failures disappear when you disable parallel execution


Common state leaks:

  • Database rows from previous tests

  • Browser cookies/localStorage persisting across tests

  • Server-side cache not cleared

  • Uploaded files from previous test runs


The fix: Isolate state per test:

test.beforeEach(async ({ request }) => {
  await request.post('/api/test/reset');
});


For database-heavy suites, wrap each test in a transaction and rollback after.


---


Category 3: Environment Flakes


Symptom: Test passes on your machine, fails in CI. Or passes in CI but fails on a teammate's machine.


Root cause: Different OS, browser version, timezone, locale, available memory, or network config.


How to identify:

  • Failure is 100% reproducible in one environment, 0% in another

  • Screenshots from CI show different font rendering or layout

  • Failures correlate with CI infrastructure changes


Common culprits:

  • Different Chromium versions between local and CI

  • Timezone-dependent date formatting

  • macOS case-insensitive filesystem vs Linux case-sensitive

  • CI containers with 2GB RAM vs your 32GB workstation


The fix: Docker. Pin everything:

FROM mcr.microsoft.com/playwright:v1.42.0-jammy
# Same browser, same OS, same everything


---


The 5-Minute Diagnostic


When a test flakes:


  1. Run it alone 10 times. All pass? → State flake. Something else is contaminating it.

  2. Run it in CI 10 times. Local passes, CI fails? → Environment flake. Lock your environment.

  3. Neither?Timing flake. Find the implicit wait and make it explicit.


This takes 5 minutes and saves hours of guessing.


---


I built a full course combining Playwright's Trace Viewer with Zig-powered analysis tools to systematically eliminate flaky tests — from trace recording through building custom analyzers to a capstone project debugging a real flaky suite. If flaky tests are costing your team CI time, check it out.

Profile picture
@atticbulkProfile pictureJun 10

The 3 Categories of Flaky Tests (and How to Identify Which One Is Killing Your CI)

Every flaky test falls into one of three categories. Once you know which category you're dealing with, the fix becomes obvious.


I've debugged flaky test suites at scale for years. Here's the framework I use every time.


---


Category 1: Timing Flakes


Symptom: Test passes locally, fails in CI. Or passes 8/10 times.


Root cause: Your test assumes something will be ready "by now." Under load, "by now" is a lie.


How to identify:

  • The failing assertion involves UI state after a user action

  • Adding a sleep(3000) "fixes" it

  • Failures correlate with CI runner CPU load


The pattern:

// ❌ This is a timing flake waiting to happen
await page.click('#submit');
await page.waitForTimeout(2000);
expect(await page.textContent('.result')).toBe('Success');


The fix: Replace arbitrary waits with deterministic ones:

// ✅ Wait for the actual condition, not an arbitrary duration
await page.click('#submit');
await expect(page.locator('.result')).toHaveText('Success');


Playwright's web-first assertions auto-retry until the condition is met or timeout expires. Use waitForResponse when the flake is network-dependent.


---


Category 2: State Flakes


Symptom: Test passes in isolation, fails when run with other tests. Order-dependent.


Root cause: Test A leaves data behind. Test B assumes a clean slate.


How to identify:

  • Run the failing test alone — it passes

  • Run it after a specific other test — it fails

  • Failures disappear when you disable parallel execution


Common state leaks:

  • Database rows from previous tests

  • Browser cookies/localStorage persisting across tests

  • Server-side cache (Redis, in-memory) not cleared

  • Uploaded files from previous test runs


The fix: Isolate state per test. The simplest approach:

test.beforeEach(async ({ request }) => {
  // Reset database to known state before each test
  await request.post('/api/test/reset');
});


For database-heavy suites, wrap each test in a transaction and rollback after.


---


Category 3: Environment Flakes


Symptom: Test passes on your machine, fails in CI. Or passes in CI but fails on a teammate's machine.


Root cause: Different OS, browser version, timezone, locale, available memory, or network configuration.


How to identify:

  • Failure is 100% reproducible in one environment, 0% in another

  • Screenshots from CI show different font rendering or layout

  • Failures correlate with CI infrastructure changes


Common culprits:

  • Different Chromium versions between local and CI

  • Timezone-dependent date formatting (en-US vs en-GB)

  • macOS case-insensitive filesystem vs Linux case-sensitive

  • CI containers with 2GB RAM vs your 32GB workstation


The fix: Docker. Pin everything:

FROM mcr.microsoft.com/playwright:v1.42.0-jammy
# Same browser, same OS, same everything — locally and in CI


---


The Diagnostic Flowchart


When a test flakes:


  1. Run it alone 10 times. All pass? → State flake. Something else is contaminating it.

  2. Run it in CI 10 times. Local passes, CI fails? → Environment flake. Lock your environment.

  3. Neither?Timing flake. Find the implicit wait and make it explicit.


This takes 5 minutes and saves hours of guessing.


---


Going Deeper


I built a full course that teaches you to combine Playwright's Trace Viewer with Zig-powered analysis tools to systematically eliminate flaky tests. You'll build a custom trace analyzer, learn deterministic wait patterns, state isolation strategies, and complete a capstone project debugging a real flaky suite.


If flaky tests are costing your team time, check it out.