What is flaky unit tests and how to mitigate it.

Unit tests are supposed to give us confidence in our code. But when tests pass sometimes and fail other times without any code changes, they become frustrating instead of helpful. These unreliable tests are known as flaky tests.

In this post, we’ll explore:

• What flaky unit tests are

• Common causes of flakiness

• Mocking vs. Not Mocking: How Each Can Cause Flaky Tests (And How to Avoid It)

• Strategies to make your tests more reliable

🔍 What Is a Flaky Unit Test?

Flaky tests are tests that produce different outcomes for the same code under the same conditions. They are unreliable and can lead to a lack of confidence in the testing process. Both mocking and not mocking components can potentially introduce flakiness if not done carefully, but they have different characteristics.

🧯 Common Causes of Flaky Unit Tests

1. Asynchronous Code

If your test doesn't properly wait for promises, timeouts, or async operations to complete, results will be inconsistent.

2. Shared or Leaky State

If one test affects another due to shared global variables, caches, or mocks not being reset, the outcome becomes unpredictable.

3. External Dependencies

Tests relying on APIs, databases, or system time can fail due to network latency or environment differences.

🔁 Mocking vs Not Mocking: A Hidden Source of Flakiness

1. Mocking Components:

🧪 Potential for Flakiness

Mocking helps isolate the unit under test by replacing dependencies with simplified versions. But mocking brings its own risks:

• Out-of-sync mocks: If your mocks don’t reflect the real component’s current behavior, your tests can pass incorrectly.

• Overly complex mocks: Overengineering your mocks can make them fragile and prone to bugs.

Example: Suppose you're testing a component that uses a <UserProfile /> child component. If you mock <UserProfile /> to always return a success state, but the actual component has loading or error states, your test may not catch real-world issues.

2. Not Mocking Components (Shallow Rendering):

🧪 Potential for Flakiness

Testing real components or rendering them deeply ensures your tests are closer to how users experience the app. But:

• Uncontrolled side effects: Child components may have side effects (e.g. network requests) that affect test reliability.

• Unstable async behavior: Not properly handling asynchronous actions can cause tests to fail randomly.

Example: If a child component fetches data from an API and the test doesn’t wait for the fetch to complete, assertions might run too early, resulting in intermittent failures.

🛠️ How to Mitigate Flakiness:

Regardless of the testing approach, there are strategies to mitigate flakiness:

1. Maintain Mocks Carefully:

   • If you choose to mock components, keep the mocks up-to-date with the actual component behavior.

   • Regularly review and update mocks when component implementations change.

2. Handle Asynchronous Operations:

   • For asynchronous code, ensure proper handling of promises, timeouts, and other async operations in your tests.

   • Use tools like await or utilities provided by testing libraries to handle asynchronous behavior.

3. Isolate Tests:

   • Ensure that each test is independent and doesn't rely on the state left behind by previous tests.

   • Avoid sharing state between tests.

4. Use Realistic Data:

   • If you're not mocking, use realistic data in your tests to simulate real-world scenarios.

   • Mimic the conditions of your application as closely as possible.

5. Regularly Review and Refactor Tests:

   • Periodically review and refactor your tests to ensure they remain reliable as your codebase evolves.

🎯 Final Thoughts

In summary, both mocking and not mocking can potentially introduce flakiness if not handled carefully. The key is to strike a balance, maintain tests regularly, and ensure they reflect the actual behavior of the application components.

There’s no single best approach—mocking and not mocking both have their place. The key is understanding the trade-offs and being intentional about your choices.

Well-maintained, reliable tests not only prevent bugs but also build confidence in your code. A little extra care in your testing strategy goes a long way.

Thanks for reading

If you have faced flaky tests and have tips of your own, i would like to hear about it in the comments.