ZaStoGram_desktop/Telegram/SourceFiles/test/docs/helpers/test_runner.md

12 KiB

test_runner

Harness guide · Helper index

Type: Harness API (may include self-tests).

Source contract: test_runner.h.

Use when

Stages, gated groups, bounded readiness and exact-object actions. Read the contract below and the source header before use.

Contract and usage

Stages, bounded waits, exact-widget actions, prepared capture/inspection (captureAndInspect takes an optional trailing skipReason for its stage), first-class gated skips (skipReason), an inherited fixture gate for a composed group of stages (addGated) with explicitly declared restoration stages (addRecovery, see gated groups and recovery stages), onFinish release hook (and its finish-release self-test), watchdog (TDESKTOP_TEST_WATCHDOG in seconds) and termination.

A permanent scenario module that is compiled only on request, such as the localization capture from the l10n-tdesktop repository, registers itself from a static initializer with RegisterScenarioProvider. Start() asks each provider in turn; the first that returns true has appended its stages and the overlay's SetupScenario is not called. A provider returns false when its own switch, usually an environment variable, is not set.

The stage contract

Runner executes each Stage as three distinct operations:

Member Contract Must not do
run One action whose prerequisites were established by an earlier stage. Dereference an object that this stage is supposed to wait for.
until A pure, repeatable readiness observation. Click, mutate state, emit a PASS/FAIL, or encode the expected product result. The one documented exception is the Test::ForceWindowActive re-assertion in the activation row in failure diagnosis.
then Assertions and the next action after readiness succeeded. Resolve an unguarded replacement for the object that readiness accepted.
timeoutDetails Return the latest observed values and identities. Repeat only “not ready”; omit the values needed to diagnose why.
skipReason Return why this stage does not apply, or an empty string when it does. A non-empty reason writes TEST_RESULT: N/A: <stage> - <reason>, skips run, until and then, and moves on in the same turn. Decide the product outcome. A gate that reads a measurement instead of a precondition turns a would-be FAIL into a silent N/A.

A stage timeout is a harness failure unless the readiness condition itself is the behavior under test. Keep expected geometry, text, pixels, counts, and other falsifiable outcomes in then or captureAndInspect, not in until. Otherwise a real product mismatch is mislabeled as an opaque timeout.

Use Runner::actOnWidget when an action depends on a widget that may appear asynchronously. It polls the resolver and optional readiness predicate, keeps a QPointer to the exact accepted object, and invokes the action once. This removes both the eager-.run crash and the “resolve again in .then” race. The same rule is built into the harness's widget-carrying readings: Test::BoxShellButtons, Test::WindowMappedCapture, Test::PaintingLayerRootResult, Test::ToastSubtreeReading and Test::ContextMenuReading hold QPointer<QWidget>, so a reading kept across a stage boundary answers matched() / resolved() false (a context-menu reading's menu reads null) once its subject is destroyed and still prints its counts, labels, identity and refusal; format a WidgetDescription of the pointer while the reading is still matched, because there is nothing to format afterwards.

The generic startup waits are opt-in:

  • waitEvent("launch_finished") is the normal first gate.
  • waitForSessionReady() is needed only when the scenario consumes a session.
  • waitForChatsLoaded() is needed only when the tested setup consumes the chats list; it is non-fatal and records whether loading completed.
  • waitForChatsLoadedStrict() is appropriate only when chats loading is part of the required fixture or result.

Do not put all three waits in every scenario preamble. An injected box, helper oracle, or direct production seam often needs none of the account or chats gates.

Before the first stage begins, Runner::start() runs the session gate (test_session_gate.h) for every scenario, self-test packs included. It writes one row per account, NOTE: SESSION_GATE: account index=<i> authorized=<0|1> environment=<Production|Test> primary=<0|1> active=<0|1>, then a reading row, and on a pass a decision pass row and nothing else: it changes no result when every authorized account that the primary window, its shown session and domain().active() select is on the Test environment, or when no account is authorized. When one of them selects an authorized account on another environment, it writes exactly one TEST_RESULT: FAIL: fixture gate: session gate - case=... row naming that environment before any TEST_STEP, and every stage then reads TEST_RESULT: N/A: <stage> - fixture gate: session gate refused the scenario (case=<case>) with none of its callbacks run; the run still reaches TEST_COMPLETE. The gate decides once, at start, and is not a continuous account lock: code that SetupScenario itself executes is not a stage and is not gated, so SetupScenario only appends stages, registers releases and configures the gate. Once the gate passed, a stage may switch accounts through the product's Main::Domain::activate(), for example to the production account a declaration authorized (acting as a non-Test account still needs that declaration), and owns switching back before the run ends. The harness's own switch is the Test-session helper's (Test::UseTestSession), inside the gate and before any stage, undone by its last stage.

Gated groups and recovery stages

Runner::addGated(gate, compose) shares one fixture prerequisite with a composed group of stages. It calls compose once, synchronously, before start(), and every stage appended while it runs inherits gate: stages added through add, the sugar waits, actOnWidget, captureWidget, captureAndInspect, and every stage that an Append* or Capture* helper given the runner appends, because all of them append through Runner::add(). A helper needs no skip parameter of its own to be gated.

When an inherited stage begins, Runner::beginStage() reads its reason in this order, and the first non-empty answer wins:

Order Reason Read when
1 The session gate's refusal Always: Runner::start() replaces every stage's reason through RefusedStage, so on a refusal nothing below is read.
2 The outermost open group's gate The session gate passed.
3 Each nested group's gate, outer to inner Every outer gate answered empty.
4 The stage's own skipReason (for captureAndInspect, its trailing parameter) Every group gate answered empty.

A non-empty answer writes TEST_RESULT: N/A: <stage> - <reason>, counts in SkippedCount() and in SCENARIO_RESULT's skipped clause, and skips run, until and then: the stage resolves no widget, polls no readiness, dispatches no action and takes no capture. The gates after it and the stage's own skipReason are not called at all.

Each group latches its first refusal. Its gate is read again at every stage begin while it answers empty; its first non-empty answer is then kept for every later stage of that group. A gate that later answers another reason (an incidental target does not exist) or empty again (a restoration that cleared the overlay's state) changes nothing, so every row of the group names the first useful refusal. A nested group latches its own answer and is never read while an outer group refuses.

The gate is read when each stage begins, not at composition, so the prerequisite must be settled by an earlier stage: await an asynchronous answer (a fixture write, a media gate) in that stage's until and set the state the gate reads in its then. A gate reading a request that is still running answers empty and lets the dependent stage run.

Not inherited: stages appended after compose returns, addRecovery stages, onFinish callbacks registered during compose, and any work compose does other than appending. Building a widget, arming a timer or sending a request inside compose happens at composition whatever the gate later answers: put such work in a gated stage's run, and release what composition acquired through onFinish.

Runner::addRecovery(stage) appends stage with only its own skipReason, even inside an open group. That reason is the restoration's own recovery policy, typically whether an obligation is owed; recovery is never inferred from a skipped stage. Use it for restoration between the legs of a campaign-wide group: leg 1's restoration runs after leg 1's prerequisite refused, while leg 2's stages still read leg 1's first refusal. Outside a group addRecovery behaves like add and keeps the stage independent if a later edit moves it into a group. A restoration helper that appends through add, Test::AppendTestSessionRestore among them, inherits the gate when called inside compose: call it after the group. The session gate's refusal still skips a recovery stage, as it skips every stage.

The appended-helper pitfall: a per-stage skipReason reaches only the stage it is set on. Setting it on an overlay's own wrapper stages, or passing it to captureAndInspect or CapturePopupMenu, never reaches the stages that actOnWidget or another helper appends. In Run 2 of 2026/10/17/open-ones-own-qr-box-without-a-username-with-temporary-only-codes a genuine FLOOD_WAIT_13 refused a privacy fixture write; the overlay's wrapper stages read N/A, but its direct actOnWidget steps ran anyway, a later reopen timed out, and that timeout ended the run before the ordinary restoration stages. The overlay's gate was also a plain string that later stages overwrote, so the first refusal was lost. Compose such stages in addGated instead.

onFinish is unaffected: gating skips stages, never releases. A group refused to the end of the run reaches finish() from the beginStage() skip loop, and the onFinish callbacks run once, as on every other finish path. A genuine timeout of an applicable gated stage, one whose gate answered empty, is still a stage timed out FAIL.

A minimal sketch; ApplyPremise, PremiseSettled, PremiseHolds, PremiseText, RestorePremise, PremiseRestored and AppendTargetChecks stand for the overlay's own code:

struct State {
	QString gate;
	bool owed = false;
};
const auto state = std::make_shared<State>();
runner->add({
	.name = u"fixture: apply the premise"_q,
	.run = [=] { state->owed = ApplyPremise(); },
	.until = [] { return PremiseSettled(); },
	.then = [=] {
		const auto holds = PremiseHolds();
		Test::Check(holds, u"fixture gate: premise holds"_q, PremiseText());
		if (!holds) {
			state->gate = u"fixture gate: premise not established"_q;
		}
	},
});
runner->addGated([=] { return state->gate; }, [&] {
	runner->actOnWidget(
		u"open the target"_q,
		[] { return Test::ReadLiveWidget(u"task.target"_q).widget; },
		[](QWidget *widget) { Test::Click(widget); });
	AppendTargetChecks(runner); // Its stages inherit the gate too.
	runner->addRecovery({
		.name = u"restore the premise"_q,
		.skipReason = [=] {
			return state->owed
				? QString()
				: u"applies=0: nothing to restore"_q;
		},
		.run = [] { RestorePremise(); },
		.until = [] { return PremiseRestored(); },
	});
});

A server fixture such as privacy rules is applied and restored through the fixture transaction, whose page shows the same shape with a real read-before-write transaction. The behavior is measured by Test::AppendStageGroupGateSelfTest (its page).

Scenario design; overlay example; teardown; failure diagnosis.