Actual 4M-tap convolution measurements.

Measured intermediate ladder, every doubling from 16K to 4M taps.

These are measured results for 4,194,304 filter taps with 8,388,608-point FFTs.

One white-noise signal (seed 741) and one boundary-aligned sequence of impulses and decaying oscillations. Each input has 8,388,610 samples, spanning two hops; full convolution has 12,582,913 output samples. Hop length 4,194,305. Both overlap-add and overlap-save are measured at 53,113 and 256 significant bits: 12 cases in total.

Measured peak absolute sample errors.

Each value is max(abs(computed output − exact integer reference)) over all 12,582,913 samples of that case.

Measured dB and bit values.

Peak/peak divides maximum absolute error by the reference signal peak. Peak/RMS divides it by the reference signal RMS. RMS/RMS divides error RMS by signal RMS. Transient silence changes signal RMS, so these denominators must not be interchanged.

Bits lost = p + error_dB/(20log10(2)), using the stated error normalization. This is numerical error relative to the signal, not converter ENOB or a guarantee for every input. The observed peak is a maximum in these test signals, not a proven worst-case bound.

Exact reference and validation.

The 4M-tap Hann-windowed sinc filter has cutoff 0.13 cycles/sample. Advanpix generates it at both 113 and 256 significant bits; both produce identical stored Q1.31 coefficients. Inputs use Q1.23. This excludes shared input/coefficient quantization and final output quantization from the arithmetic-error measurement.

The reference uses GMP 6.3.0 through gmpy2 2.3.1 to multiply packed integer polynomials exactly. Digit width prevents coefficient carries. Offset terms are removed modulo 2^64; a proved signed64 absolute-partial-sum bound uniquely recovers each true coefficient. No floating-point convolution provides the reference. Results are scaled by 2^-54 only after exact integer computation.

The exact reference is verified independently at selected boundary and random samples using direct integer dot products. The complete transient output also agrees, sample for sample, with a separate sparse direct integer convolution.

Output errors are calculated in chunks after promoting both the computed output and exact integer reference to 665-bit precision. Overlap-add sums remain at the tested precision. Overlap-save excludes discarded samples and checks every valid sample, including the final tail.

Test environment and timing.

The table separates block-filtering time from filtering plus high-precision error analysis. The filter spectrum is reused across the four signal/method combinations at each precision.

Timings exclude filter-spectrum construction, MATLAB startup and exact-reference preparation. Test environment: MATLAB R2020b Update 2 and Advanpix 5.1.0.15432, with zero guard digits and eight configured Advanpix threads.

These measurements isolate arithmetic error in the stated FIR filter, signals and block-processing methods.

← Back to the article