处理不稳定的故事，特别是高层集成测试，是一场持续的战斗。

这些示例展示了如何通过多次重新运行故事并检查结果是否相同来检测不稳定的故事。这种回归测试可以有效地与正常的回归测试分开运行，以便在“抖动”报告中获得关于有问题的测试的独立反馈。

这无法检测所有类型的抖动，但它可以有效地检测由以下原因引起的不稳定性，例如

• 没有 order by 的 SELECT 语句以任意顺序返回结果。
• 列出具有不确定顺序的数据结构（例如哈希映射（python 字典））。
• 导致生成和使用随机数的故事。
• 竞态条件。
• 网页响应速度（例如，过快完成的 selenium 点击）。

请注意，抖动“通过”只是关于结果是否 *一致* - 一直失败的故事被认为是通过的，而运行 99 次失败一次的故事被认为是失败的。

代码示例

example1.story

Flaky story:
  steps:
  - Step that fails on fifth run

Consistent failure:
  steps:
  - Step that always fails

带代码

from hitchstory import StoryCollection, BaseEngine, Failure
from pathlib import Path

class Engine(BaseEngine):
    def __init__(self):
        self._tries = 0

    def step_that_fails_on_fifth_run(self):
        self._tries = self._tries + 1
        if self._tries >= 5:
            raise Failure("Flaky story failure!")

    def step_that_always_fails(self):
        raise Failure("Consistent failure!")

运行在第五次尝试时失败的单个故事

flake_result = StoryCollection(Path(".").glob("*.story"), Engine()).with_flake_detection(times=5).named("flaky story").play()

assert flake_result.is_flaky

将输出

RUNNING Flaky story in /path/to/working/example1.story ... SUCCESS in 0.1 seconds.
RUNNING Flaky story in /path/to/working/example1.story ... SUCCESS in 0.1 seconds.
RUNNING Flaky story in /path/to/working/example1.story ... SUCCESS in 0.1 seconds.
RUNNING Flaky story in /path/to/working/example1.story ... SUCCESS in 0.1 seconds.
RUNNING Flaky story in /path/to/working/example1.story ... FAILED in 0.1 seconds.

    Flaky story:
      steps:
      - Step that fails on fifth run

    Consistent failure:

hitchstory.exceptions.Failure

    Test failed.

Flaky story failure!

FLAKINESS DETECTED in 0.1 seconds, 20% of stories failed.

运行每次都失败的单个故事

flake_result = StoryCollection(Path(".").glob("*.story"), Engine()).with_flake_detection(times=5).named("consistent failure").play()

assert not flake_result.is_flaky

将输出

RUNNING Consistent failure in /path/to/working/example1.story ... FAILED in 0.1 seconds.

    Consistent failure:
      steps:
      - Step that always fails


hitchstory.exceptions.Failure

    Test failed.

Consistent failure!
RUNNING Consistent failure in /path/to/working/example1.story ... FAILED in 0.1 seconds.

    Consistent failure:
      steps:
      - Step that always fails


hitchstory.exceptions.Failure

    Test failed.

Consistent failure!
RUNNING Consistent failure in /path/to/working/example1.story ... FAILED in 0.1 seconds.

    Consistent failure:
      steps:
      - Step that always fails


hitchstory.exceptions.Failure

    Test failed.

Consistent failure!
RUNNING Consistent failure in /path/to/working/example1.story ... FAILED in 0.1 seconds.

    Consistent failure:
      steps:
      - Step that always fails


hitchstory.exceptions.Failure

    Test failed.

Consistent failure!
RUNNING Consistent failure in /path/to/working/example1.story ... FAILED in 0.1 seconds.

    Consistent failure:
      steps:
      - Step that always fails


hitchstory.exceptions.Failure

    Test failed.

Consistent failure!

NO FLAKINESS DETECTED in 0.1 seconds after running 'Consistent failure' story 5 times.

可执行规范

从 flaky.story storytests 自动生成的文档。