TL;DR: Steer clear of non-deterministic tests..

Problem

• Lack of determinism

• Eroding confidence

• Time squandered

Solution

1. Ensure that the test is fully deterministic. Eradicate any potential for unpredictable behavior.

2. Eliminate test coupling.

Examples

The terms "Fragile," "Intermittent," "Sporadic," and "Erratic" are often used interchangeably when discussing problematic tests in many development environments.

However, such tests erode the trust developers place in their test suites.

Such tests are best avoided.

Sample Code

Wrong

import org.junit.jupiter.api.Assertions.assertEquals
import org.junit.jupiter.api.Test

abstract class SetTest {

    protected abstract fun constructor(): MutableSet<String>

    @Test
    fun `test add empty`() {
        val set = constructor()
        set.add("green")
        set.add("blue")
        assertEquals("[green, blue]", set.toString())
        // This is fragile since the outcome hinges on the set's
        // ordering, which isn't predefined and can vary based on
        // the set's implementation.
    }
}

Right

import org.junit.jupiter.api.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue

abstract class SetTest {

    protected abstract fun constructor(): MutableSet<String>

    @Test
    fun `test add empty`() {
        val set = constructor()
        set.add("green")
        assertEquals("[green]", set.toString())
    }

    @Test
    fun `test entry at single entry`() {
        val set = createFromArgs("red")
        val result = set.contains("red")
        assertTrue(result)
    }
}

Conclusion

Fragile tests often indicate system coupling and manifest as non-deterministic or unpredictable behaviors.

Addressing and resolving these erratic tests can drain considerable developer time and energy.

Credits

• Code Smell 51 - Fragile Tests by Maximiliano Contieri