LintQ Detector Evaluation Report

10 representative examples · 6 detectors · both real engines (A + B) under Soufflé 2.5

Generated 2026-08-16 20:52 UTC · subset of the 17-example challenge corpus (challenges.json)

How this report is presented

Each example below shows, for every detector, four things: the expected ground-truth label, the actual detector behaviour (fired / silent), the derived verdict, and the raw output the engine actually emitted (the literal Warn*.csv rows). The raw output is the truth — it is what the detector printed, not a re-description of it.

correct (FIRE / SILENT as expected) regression (FALSE_NEG / FALSE_POS) documented limitation out of scope (N/A)

Ground-truth label vocabulary. FIRE — the bug class is genuinely present, the detector must warn. SILENT — the bug is intentionally absent, the detector must stay silent (negative example). FALSE_POS — a documented engine limitation: a reset-blind detector is expected to fire here, so the fire is the known, non-buggy manifestation and is counted as a correct positive (reported separately as a "limitation"). N/A — the detector is structurally unable to observe this bug (e.g. Engine A has no MissingReset rule); these rows are excluded from the metrics denominator.

Precision & recall — methodology

A detector is scored only on rows where it is in scope (it is implemented by its engine and the label is not N/A). "Fire" is the positive class. For each in-scope row we compare the expected behaviour to the actual firing:

FALSE_POS rows are treated as expected-fire, so their fire counts as TP (not FP) — this measures each detector against its documented as-built contract, exactly as the CI gate does. They are also tallied in the "Limitation" column so the limitation is never hidden. Precision = TP / (TP + FP); Recall = TP / (TP + FN).

12total TP
0total FP
0total FN
100.0%overall precision
100.0%overall recall

Per-detector metrics

DetectorBug classEngineTPFPFNTNPrecisionRecallF1N/A excl.Limitation
A_DoubleMeasDoubleMeasA1002100.0%100.0%100.0%70
A_GhostComposeGhostComposeA1001100.0%100.0%100.0%80
A_OpAfterMeasOpAfterMeasA4000100.0%100.0%100.0%61
B_DoubleMeasDoubleMeasB1002100.0%100.0%100.0%70
B_MissingResetMissingResetB2006100.0%100.0%100.0%20
B_OpAfterMeasOpAfterMeasB3001100.0%100.0%100.0%60

Bug-class coverage of the eval set

Bug classEval examples covering it
OpAfterMeasbasic_reset_initialised, ex01_op_after_meas_basic, ex02_op_after_meas_reset_safe, ex10_double_meas_2qubitgate
DoubleMeasex06_double_meas_basic, ex07_double_meas_reset_between, ex10_double_meas_2qubitgate
MissingResetbasic_reset_initialised, ex01_op_after_meas_basic, ex02_op_after_meas_reset_safe, ex06_double_meas_basic, ex07_double_meas_reset_between, ex10_double_meas_2qubitgate, ex11_missing_reset_basic, ex14_missing_reset_2qubit_gate
GhostComposeex15_ghost_compose_basic, ex16_ghost_compose_assigned

Per-example results (raw detector output)

Basic OpAfterMeas (single qubit) OpAfterMeas

A gate (h) is applied to q0 after q0 was measured, with no reset between. Both engines should flag it.

Qiskit source (ex01_op_after_meas_basic.py)
from qiskit import QuantumCircuit

# OpAfterMeas: a gate acts on a qubit after that qubit was measured,
# with no reset between along the CFG path.
qc = QuantumCircuit(2, 2)
qc.reset(0)
qc.reset(1)
qc.h(0)
qc.cx(0, 1)
qc.measure(0, 0)
qc.h(0)          # BUG: gate on q0 after measure(0) -> OpAfterMeas
DetectorExpectedActualVerdictRaw detector output
A_OpAfterMeasFIREFIREFIRE
8	qc	h	0	7
A_DoubleMeasN/AsilentN/A
(no warnings emitted)
A_GhostComposeN/AsilentN/A
(no warnings emitted)
B_OpAfterMeasFIREFIREFIRE
7	0	6
B_DoubleMeasN/AsilentN/A
(no warnings emitted)
B_MissingResetSILENTsilentSILENT
(no warnings emitted)

Note: Resets at the top only initialise the qubit; the post-measurement gate is the bug.

OpAfterMeas negative: reset between measure and gate OpAfterMeas

A reset lies between the measure and the later gate, so the gate operates on a re-initialised qubit. A reset-AWARE detector (Engine B) must stay silent; the reset-BLIND Engine A falsely fires -- a documented limitation.

Qiskit source (ex02_op_after_meas_reset_safe.py)
from qiskit import QuantumCircuit

# NEGATIVE for OpAfterMeas: a reset lies between the measure and the later
# gate, so the gate does NOT operate on a post-measurement qubit state.
# A correct (reset-aware) detector must stay silent; a reset-blind detector
# (Engine A) will falsely fire here -- a documented limitation.
qc = QuantumCircuit(2, 2)
qc.reset(0)
qc.reset(1)
qc.h(0)
qc.measure(0, 0)
qc.reset(0)
qc.h(0)          # SAFE: reset(0) between measure(0) and this gate
DetectorExpectedActualVerdictRaw detector output
A_OpAfterMeasFALSE_POSFIREFALSE_POS(limitation)
8	qc	h	0	6
A_DoubleMeasN/AsilentN/A
(no warnings emitted)
A_GhostComposeN/AsilentN/A
(no warnings emitted)
B_OpAfterMeasSILENTsilentSILENT
(no warnings emitted)
B_DoubleMeasN/AsilentN/A
(no warnings emitted)
B_MissingResetSILENTsilentSILENT
(no warnings emitted)

Note: Key cross-engine example: discriminates reset-aware vs reset-blind OpAfterMeas.

OpAfterMeas via 2-qubit gate (cx touches measured qubit) OpAfterMeas

cx(0,1) acts on both q0 and q1, but only q1 was measured earlier. Detector must flag the gate on q1 (the measured qubit) and NOT on q0.

Qiskit source (ex03_op_reach_2qubit.py)
from qiskit import QuantumCircuit

# OpAfterMeas via a 2-qubit gate: cx(0,1) touches BOTH q0 and q1, but only
# q1 was measured earlier. The detector must flag the gate on q1 (the
# measured qubit) and NOT on q0 (which was never measured).
qc = QuantumCircuit(2, 2)
qc.reset(0)
qc.reset(1)
qc.h(0)
qc.measure(1, 1)
qc.cx(0, 1)      # BUG on q1: 2-qubit gate acts on measured qubit q1
DetectorExpectedActualVerdictRaw detector output
A_OpAfterMeasFIREFIREFIRE
7	qc	cx	1	6
A_DoubleMeasN/AsilentN/A
(no warnings emitted)
A_GhostComposeN/AsilentN/A
(no warnings emitted)
B_OpAfterMeasFIREFIREFIRE
6	1	4
B_DoubleMeasN/AsilentN/A
(no warnings emitted)
B_MissingResetSILENTsilentSILENT
(no warnings emitted)

Note: Tests that ActsOn propagation over 2-qubit gates does not over-report to the unmeasured control qubit.

Basic DoubleMeas (consecutive measure of same qubit) DoubleMeas

q0 is measured twice with no operation on q0 between. Both engines should flag the redundant consecutive measure.

Qiskit source (ex06_double_meas_basic.py)
from qiskit import QuantumCircuit

# DoubleMeas: qubit q0 is measured twice with no operation on q0 between.
qc = QuantumCircuit(2, 2)
qc.reset(0)
qc.reset(1)
qc.h(0)
qc.measure(0, 0)
qc.measure(0, 0)    # BUG: redundant consecutive measure of q0
DetectorExpectedActualVerdictRaw detector output
A_OpAfterMeasN/AsilentN/A
(no warnings emitted)
A_DoubleMeasFIREFIREFIRE
7	qc	0	6
A_GhostComposeN/AsilentN/A
(no warnings emitted)
B_OpAfterMeasN/AsilentN/A
(no warnings emitted)
B_DoubleMeasFIREFIREFIRE
4	5	0
B_MissingResetSILENTsilentSILENT
(no warnings emitted)

Note: The canonical redundant-measurement pattern.

DoubleMeas negative: reset between the two measures DoubleMeas

A reset on q0 sits between the two measures, so the second measure is NOT redundant. Both engines must stay silent (an op on the qubit between suppresses DoubleMeas).

Qiskit source (ex07_double_meas_reset_between.py)
from qiskit import QuantumCircuit

# NEGATIVE for DoubleMeas: a reset on q0 sits between the two measures,
# so the second measure is NOT redundant. Both engines must stay silent.
qc = QuantumCircuit(2, 2)
qc.reset(0)
qc.reset(1)
qc.measure(0, 0)
qc.reset(0)         # op on q0 between the two measures -> breaks DoubleMeas
qc.measure(0, 0)    # SAFE: reset(0) between the two measures of q0
DetectorExpectedActualVerdictRaw detector output
A_OpAfterMeasN/AsilentN/A
(no warnings emitted)
A_DoubleMeasSILENTsilentSILENT
(no warnings emitted)
A_GhostComposeN/AsilentN/A
(no warnings emitted)
B_OpAfterMeasN/AsilentN/A
(no warnings emitted)
B_DoubleMeasSILENTsilentSILENT
(no warnings emitted)
B_MissingResetSILENTsilentSILENT
(no warnings emitted)

Note: Confirms the OpBetween / ModifiedQubit guard works as the dual of the reset case in OpAfterMeas.

DoubleMeas suppressed but OpAfterMeas fires on the same cx OpAfterMeas+DoubleMeas

A 2-qubit gate cx(0,1) on q0 lies between two measures of q0: that op correctly SUPPRESSES DoubleMeas, yet cx is itself a gate on q0 applied after the first measure, so it TRIGGERS OpAfterMeas. One detector must fire (OpAfterMeas) while the other stays silent (DoubleMeas).

Qiskit source (ex10_double_meas_2qubitgate.py)
from qiskit import QuantumCircuit

# COMBINED / cross-detector: a 2-qubit gate cx(0,1) on q0 between two
# measures of q0 correctly SUPPRESSES DoubleMeas (an op touches q0 between),
# yet the same cx is a gate on q0 applied after the first measure, so it
# TRIGGERS OpAfterMeas. Demonstrates that one detector fires while the other
# must not.
qc = QuantumCircuit(2, 2)
qc.reset(0)
qc.reset(1)
qc.measure(0, 0)
qc.cx(0, 1)        # gate on q0 between measures -> suppresses DoubleMeas,
                   # but is itself a gate after measure -> OpAfterMeas on q0
qc.measure(0, 0)
DetectorExpectedActualVerdictRaw detector output
A_OpAfterMeasFIREFIREFIRE
6	qc	cx	0	5
6	qc	cx	0	7
A_DoubleMeasSILENTsilentSILENT
(no warnings emitted)
A_GhostComposeN/AsilentN/A
(no warnings emitted)
B_OpAfterMeasFIREFIREFIRE
4	0	3
B_DoubleMeasSILENTsilentSILENT
(no warnings emitted)
B_MissingResetSILENTsilentSILENT
(no warnings emitted)

Note: Cross-detector interaction: the same statement is relevant to two analyses with opposite expectations.

Basic MissingReset (first op is a gate, not a reset) MissingReset

The first operation that touches q0 is h (a gate), not a reset, so q0 is used while uninitialised. Engine B only.

Qiskit source (ex11_missing_reset_basic.py)
from qiskit import QuantumCircuit

# MissingReset: the first operation that touches q0 is a gate (h), not a
# reset, so q0 is used while uninitialised. Engine B only.
qc = QuantumCircuit(2, 2)
qc.h(0)            # BUG: first op on q0 is h, not reset (MissingReset q0)
qc.measure(0, 0)
DetectorExpectedActualVerdictRaw detector output
A_OpAfterMeasN/AsilentN/A
(no warnings emitted)
A_DoubleMeasN/AsilentN/A
(no warnings emitted)
A_GhostComposeN/AsilentN/A
(no warnings emitted)
B_OpAfterMeasN/AsilentN/A
(no warnings emitted)
B_DoubleMeasN/AsilentN/A
(no warnings emitted)
B_MissingResetFIREFIREFIRE
1	0

Note: Engine A has no MissingReset rule; marked N/A there.

MissingReset via a 2-qubit gate (cx uses uninitialised qubits) MissingReset

cx(0,1) is the first operation touching both q0 and q1, so both are used while uninitialised. Engine B must flag q0 and q1 (the cx is a gate, not a reset).

Qiskit source (ex14_missing_reset_2qubit_gate.py)
from qiskit import QuantumCircuit

# MissingReset via a 2-qubit gate: cx(0,1) is the first operation touching
# both q0 and q1, so both are used while uninitialised. Engine B must flag
# q0 and q1 (the cx is a gate, not a reset).
qc = QuantumCircuit(2, 2)
qc.cx(0, 1)        # BUG: cx uses q0 and q1 before any reset (MissingReset q0, q1)
DetectorExpectedActualVerdictRaw detector output
A_OpAfterMeasN/AsilentN/A
(no warnings emitted)
A_DoubleMeasN/AsilentN/A
(no warnings emitted)
A_GhostComposeN/AsilentN/A
(no warnings emitted)
B_OpAfterMeasN/AsilentN/A
(no warnings emitted)
B_DoubleMeasN/AsilentN/A
(no warnings emitted)
B_MissingResetFIREFIREFIRE
1	0
2	1

Note: MissingReset over a multi-target gate.

Basic GhostCompose (discarded compose return value) GhostCompose

qc.compose(sub) is non-mutating in Qiskit, but its return value is discarded, so the composed circuit is silently dropped. Engine A only.

Qiskit source (ex15_ghost_compose_basic.py)
from qiskit import QuantumCircuit

# GhostCompose: qc.compose(sub) is non-mutating in Qiskit, but its return
# value is discarded, so the composed circuit is silently dropped. Engine A
# only.
qc = QuantumCircuit(1, 1)
sub = QuantumCircuit(1, 1)
sub.h(0)
qc.compose(sub)    # BUG: compose() return value discarded (GhostCompose)
DetectorExpectedActualVerdictRaw detector output
A_OpAfterMeasN/AsilentN/A
(no warnings emitted)
A_DoubleMeasN/AsilentN/A
(no warnings emitted)
A_GhostComposeFIREFIREFIRE
5	qc	sub
B_OpAfterMeasN/AsilentN/A
(no warnings emitted)
B_DoubleMeasN/AsilentN/A
(no warnings emitted)
B_MissingResetN/AFIREN/A
1	0

Note: Engine B has no GhostCompose rule; marked N/A there.

GhostCompose negative: return value captured GhostCompose

qc2 = qc.compose(sub) keeps the return value, so the circuit is intentionally retained. Engine A must NOT fire (negative example).

Qiskit source (ex16_ghost_compose_assigned.py)
from qiskit import QuantumCircuit

# NEGATIVE for GhostCompose: the result of compose() is captured in qc2, so
# the circuit is intentionally kept. The detector must NOT fire.
qc = QuantumCircuit(1, 1)
sub = QuantumCircuit(1, 1)
sub.h(0)
qc2 = qc.compose(sub)    # SAFE: return value captured -> no GhostCompose
DetectorExpectedActualVerdictRaw detector output
A_OpAfterMeasN/AsilentN/A
(no warnings emitted)
A_DoubleMeasN/AsilentN/A
(no warnings emitted)
A_GhostComposeSILENTsilentSILENT
(no warnings emitted)
B_OpAfterMeasN/AsilentN/A
(no warnings emitted)
B_DoubleMeasN/AsilentN/A
(no warnings emitted)
B_MissingResetN/AFIREN/A
1	0

Note: Confirms the AssignedStmt guard: only non-assigned compose() calls are flagged.