Measured convolution ladder: 16K to 4M filter taps.

Every point is an actual measurement. The FFT is twice the filter length: 32K to 8M FFT points. K = 1,024; M = 1,048,576.

Nine filter sizes, three precisions, two signals and two overlap methods give 108 cases. MATLAB uses 53 significant bits; Advanpix uses 113 and 256 significant bits with zero guard digits.

Each size uses the same filter design (DC-normalized Hann-windowed sinc, cutoff 0.13 cycles/sample), stored Q1.31 coefficients, Q1.23 inputs, white-noise seed 741, and the same rule for positioning the transient sequence around block boundaries. The filter and signal lengths grow with the test. Each input spans two hops; all 3 × taps + 1 output samples, including the tail, are checked.

Peak error in bits.

Peak error means the largest absolute sample difference divided by the exact reference signal peak. Equivalent bits lost = p + 20 log10(error ratio) / 6.020599913. These are numerical error ratios, not converter ENOB or a guaranteed worst-case bound.

The summary tables show the larger observed peak error from overlap-add and overlap-save at each size. The full tables below list both methods separately.

White noise.

Transient sequence.

What the intermediate measurements show.

For 256-bit white-noise convolution, RMS error rises by 3.02 dB with overlap-add and 3.02 dB with overlap-save between 256K and 512K taps. The growth is not described by a single smooth logarithmic curve over this range. This experiment does not isolate the cause: FFT size, stored filter and signal length change together.

The largest observed peak-relative loss anywhere in this ladder is 3.125 bits. Peak errors can stay unchanged or decrease at the next size; they are maxima of particular output sequences, not a monotonic error bound. These results apply to the specified stored filter and signals.

The vertical axes show bits lost from each arithmetic type's own working precision. Equal heights therefore do not mean equal absolute errors. A negative loss means the measured error ratio is below 2^-p for that particular case; floating-point precision is not an absolute error floor, and this does not mean the arithmetic acquired extra working bits. Lines connect measured sizes and do not imply measurements between them.

Full measured values.

RMS/RMS uses error RMS divided by reference RMS. Peak/peak uses maximum absolute error divided by reference peak. Peak/RMS uses maximum absolute error divided by reference RMS. These normalizations must not be interchanged, particularly for a sparse transient sequence.

IEEE64 / 53 significant bits.

Advanpix IEEE128 / 113 significant bits.

Advanpix / 256-bit mantissa.

Reference and validation.

The reference uses exact integer polynomial convolution. Carry-free packing and signed-64-bit uniqueness bounds preserve every coefficient. Selected boundary and random samples are checked with exact integer dot products; the complete transient reference also agrees with a separate sparse direct integer convolution. Filter coefficients generated at 113 and 256 significant bits agree after Q1.31 quantization at every size.

Each output difference is calculated after promoting the computed output and exact integer reference to 665-bit precision. Overlap-add sums remain at the tested precision. Overlap-save checks retained samples and the final tail. Output counts and error metrics are checked for every case.

The test environment is MATLAB R2020b Update 2 and Advanpix 5.1.0.15432, with eight configured Advanpix threads.

See the actual 4M endpoint report for measured endpoint details and reference validation.

The same filter design, stored-value precision and signal construction are used across all nine lengths, allowing the tables to show how the measured error changes with filter size.

← Back to the article