Code coverage can miss a requirement

How-to Last tested 24 October 2026

A code-coverage report can reach 100 percent while a requirement remains untested because the report counts execution, not whether the required outcome was checked. In the example below, the buggy adder reaches every reported line and branch point without ever testing saturation. A requirement-linked overflow point and an independent expected-value check expose the omission.

Put the requirement ahead of the percentage

The requirement is small enough to state without borrowing its answer from the implementation: with enable low, the result is zero; with enable high, add two unsigned two-bit operands and saturate at three. Saturation is the behavior that prevents a mathematical sum of four from wrapping to zero. A check that compares the result only with a truncated two-bit sum would repeat the bug and could pass for the wrong reason.

The first test supplies one disabled addition and one enabled addition whose sum is two. Those cases exercise both outcomes of the buggy implementation’s only runtime condition. They also satisfy the two outcomes that the test actually checks. Neither case asks for saturation, so the test leaves the most distinctive part of the stated behavior unresolved. The missing requirement is not another line waiting to execute: the implementation omitted the saturation decision altogether.

Separate three questions when reading the report. Which instrumentation points executed? Which requirement situations occurred? Which expected outcomes were compared with the design’s outputs? Here, Verilator supplies the first answer, three original testbench counters supply the second, and the mismatch counter supplies the third. Calling all three “coverage” without their kind would conceal the distinction that makes the example useful.

Run the complete counterexample

Save the complete source as example.sv in a separate working directory. It contains the buggy and corrected elaborations, the stimulus and the expected-value calculation. FIX=0 selects the buggy design; FIX=1 selects saturation. The source bytes stay the same between those elaborations. The testbench is explicitly excluded from Verilator’s code-coverage instrumentation so its bookkeeping does not inflate the design’s report.

// SPDX-License-Identifier: MIT
`timescale 1ns/1ps
module bounded_adder #(parameter FIX = 0)(
  input logic enable,
  input logic [1:0] a, b,
  output logic [1:0] result
);
  logic [2:0] wide;
  generate
    if (FIX) begin : corrected
      always_comb begin
        wide = {1'b0, a} + {1'b0, b};
        if (!enable) result = 0;
        else if (wide > 3) result = 3;
        else result = wide[1:0];
      end
    end else begin : broken
      always_comb begin
        wide = {1'b0, a} + {1'b0, b};
        if (enable) result = wide[1:0];
        else result = 0;
      end
    end
  endgenerate
endmodule
/* verilator coverage_off */
module tb #(parameter FIX = 0);
  logic enable = 0;
  logic [1:0] a = 0, b = 0, result;
  integer extended = 0;
  integer disabled_seen = 0, ordinary_seen = 0, overflow_seen = 0;
  integer mismatches = 0;
  bounded_adder #(.FIX(FIX)) dut(enable, a, b, result);
  task sample(input logic en, input logic [1:0] aa, bb);
    integer expected, total;
    enable = en;
    a = aa;
    b = bb;
    #1;
    total = int'(aa) + int'(bb);
    expected = !en ? 0 : total > 3 ? 3 : total;
    if (!en) disabled_seen = disabled_seen + 1;
    else if (total > 3) overflow_seen = overflow_seen + 1;
    else ordinary_seen = ordinary_seen + 1;
    if (result !== 2'(expected)) mismatches = mismatches + 1;
    $display("enable=%0d a=%0d b=%0d expected=%0d result=%0d",
             en, aa, bb, expected, result);
  endtask
  initial begin
    if ($value$plusargs("extended=%d", extended)) begin end
    sample(0, 1, 1);
    sample(1, 1, 1);
    if (extended) sample(1, 3, 1);
    $display("FUNCTIONAL disabled=%0d ordinary=%0d overflow=%0d",
             disabled_seen, ordinary_seen, overflow_seen);
    $display("CHECKER mismatches=%0d", mismatches);
    $finish;
  end
endmodule

Build the default top with Verilator 5.052 using --binary --timing --coverage-line --top-module tb -GFIX=0 -Wno-fatal example.sv. Run it with +extended=0 and the documented coverage-output argument. Repeat with +extended=1, then rebuild with -GFIX=1 and run the extended case. The example README gives all commands, pinned tool revisions and report filenames. The Icarus run uses the identical HDL; it provides behavioral checks and functional counters, not a native code-coverage percentage.

The counterexample deliberately finishes after printing a mismatch count instead of stopping on its first mismatch. That permits the failing run’s coverage data to be written. A zero process exit status therefore does not mean the design passed: read CHECKER mismatches. An automated regression using this example should classify a nonzero count as a design failure while retaining the report for diagnosis.

Read the three reports without mixing their scopes

These are the literal report categories produced by verilator_coverage_bin_dbg from Verilator 5.052 at an annotation threshold of one hit. The corrected elaboration has a different instrumentation inventory: its summary reports four line points and no branch points. Zero reported points is not zero coverage, and it is not evidence that an equivalent branch metric was collected. Toggle, expression and state-machine coverage were not enabled; the tool’s zero-denominator summary rows are not additional achieved percentages¹.

In the first run, the enabled ordinary case returns two, and the disabled case returns zero. In the second, the added operands are three and one. The independently computed saturated answer is three, but the buggy result is zero. Both simulators print that mismatch. In the corrected elaboration the same added case returns three, and both simulators print zero mismatches across the three cases. These observations concern this small deterministic sequence, not exhaustive arithmetic verification.

Verilator’s annotation summary also reports source lines with all attached points covered. That count is a presentation of the instrumented points, not another independent requirement result. Preserve the tool’s labels and the configured design when comparing runs; do not merge percentages from different elaborations merely because the source filename matches. The recorded outputs retain the commands, input hash, warnings and summaries².

Keep the intentionally broken elaboration as a test of the verification process itself. A useful regression should reject its overflow result while accepting the corrected elaboration under the same stimulus. If a later testbench edit makes both look successful, investigate the expected-value calculation and mismatch classification before trusting the green status. This single seeded defect is a concrete sensitivity check for one requirement; it does not measure the testbench’s ability to detect every possible design fault.

The runs were not warning-free. Icarus reported conservative sensitivity handling for constant selections in always_comb; Verilator flagged the integer extended control used as a condition. Compilation completed, the source was retained and the literal diagnostics remain in the records. The demonstrated results concern the printed stimulus, not a claim that every warning is harmless in other code.

Make the verification plan reviewable

Start a requirement row with an observable promise, an activation condition and a checker. For the saturation row, the activation condition is an enabled mathematical sum greater than three, the expected result is three, and the checker compares the actual two-bit result after the combinational logic settles. The functional counter increments when that activation occurs; it does not increment because the checker passed. That separation preserves evidence of exercised-but-failing behavior.

A useful review record also identifies the design configuration, stimulus source, comparison timing and evidence file. If an expected value came from a reference model, record which arithmetic width and signedness it used. In this example the expected sum is evaluated as an integer from unsigned operands, avoiding the two-bit truncation that created the bug. A more complicated model needs its own validation; agreement between two copies of the same mistaken algorithm is weak evidence.

Do not close an uncovered goal by changing its label. If no test can reach it, first check the stimulus constraints, the observation point and whether the feature exists in the selected configuration. A goal that remains unreachable in simulation needs a separate disposition: directed stimulus, a scoped formal argument, a documented configuration exclusion or an unresolved item. The coverage-hole workflow shows those dispositions with actual seed and proof records.

This example’s three cases are intentionally small. They do not cover every input pair, all changes while disabled, transient glitches or timing. A next test could enumerate the finite operand space and compare every result, but that experiment was not part of this page. Keeping that boundary visible prevents the corrected demonstration from being promoted into arithmetic sign-off merely because the bug used to illustrate coverage was fixed.

Before comparing a later regression, record which top and parameter values produced the original denominator. A changed configuration can remove a branch or introduce new instrumentation points while leaving the overall percentage unchanged. Compare the requirement observations and checker outcomes first, then use the code report to locate execution gaps within that same configuration.

Reuse the meaning when moving from block to SoC

Accellera’s UVM 1.2 User’s Guide describes reusable verification components, including agents, monitors, scoreboards and environments. It also describes active and passive agent roles: a component that drives an interface at block level can observe that interface when a higher-level environment supplies traffic. That is documentation context here; no UVM library, agent or block-to-SoC environment was executed³.

Accellera’s current download page lists the UVM 2020-3.2 reference implementation, dated August 2026. That current implementation listing and the older user guide serve different purposes; this page does not relabel the 1.2 guide as the current implementation or claim compatibility testing against either library⁴.

For this adder, the reusable evidence would be the saturation requirement, the input/output observation and the expected-value rule. An integrated system may need a new way to cause those operands to arrive, plus checks that address decoding, configuration and reset select the intended instance. The block’s old hit count cannot establish that system-level connection. The existing block-to-SoC example follows software-visible actions through two original model instances.

Keep block reports and integration reports distinguishable even when they refer to the same requirement. A block test can isolate an arithmetic error; an integration test can expose an unreachable configuration path or a result delivered to the wrong destination. Neither evidence set should erase the other’s assumptions. Carry the requirement identifier forward, then attach the checks that actually ran at each level and the outcome each observed.

Not covered on this page

• Exhaustive arithmetic correctness, timing, power or physical implementation.
• Executed UVM environments or commercial coverage databases.
• Toggle, expression or state-machine coverage measurements.
• Sign-off criteria for an integrated chip.