edge-infer benchmarks

Minimal zero-dependency INT8 inference runtime in C++17 — github.com/ericsson-kuma/edge-infer

Intel Core i7-6700 @ 3.40 GHz (Skylake, AVX2+FMA) · g++ 11.4.0 · -O3 -march=native -funroll-loops · single-threaded · median of 5–21 reps after warmup (MAD in tooltips)

2.9×

INT8 end-to-end speedup

LeNet-shaped CNN vs FP32 reference

11.2×

blocked GEMM vs naive

int8, 512×512

0.99903

cosine similarity

INT8 vs FP32 golden, worst of 8 inputs

29,008 B

runtime scratch

static plan, zero alloc in run()

GEMM optimization journey

Throughput of the same matmul as the kernel evolves: naive (i,j,k; strided B) → reordered (i,k,j; streams B/C rows, auto-vectorizes) → blocked (+256×256 K/N tiles, 8-wide unrolled inner loop). Higher is better.

naive reordered blocked

int8 (int32 accumulate) — GMAC/s

fp32 — GMAC/s

INT8 vs FP32 (blocked kernel)

Scalar-widening int8 loses to AVX2-vectorized fp32 at small sizes and only pulls ahead once its 4× smaller footprint wins cache residency — honest crossover, not a marketing chart. Explicit int8 SIMD is top of the backlog.

fp32 int8

End-to-end: LeNet-shaped CNN (1×28×28)

FP32 = direct-convolution reference path; INT8 = quantized graph (im2col + int8 GEMM + saturating requantize) on the static memory plan. Lower is better.

Full results

kernel	dtype	size	median ms	MAD ms	reps	GMAC/s

Numbers regenerate with ./build/bench/edge_bench (writes bench_results.json). Accuracy: max abs error 0.3629, cosine 0.99903 (worst over 8 random inputs). MIT licensed.