Issue 02 · Measurement
Benchmarks That Lie
A number without its conditions is marketing. Here is how to build benchmarks that tell the truth about edge AI.
From the Editor
The response to Issue 01 told me something useful: people are tired of numbers they cannot trust. Every hardware launch arrives with a chart. Every AI platform claims a speedup. Almost none of them tell you the conditions that produced the number, which means almost none of them tell you anything.
This issue is about measurement. How to design a benchmark that survives scrutiny, why duration matters more than peak speed, and how to get a first federal research award without pretending to be something you are not.
Measure honestly. Publish the conditions. Let the work speak.
Feature
Benchmarks That Lie
There is a familiar pattern in edge AI marketing. A device runs a model, a stopwatch records the fastest pass, and that number becomes the headline. It is technically true and operationally meaningless.
Field systems do not run once. They run for weeks, in heat, on unstable power, with other workloads competing for memory and accelerators. A benchmark that ignores those conditions measures the device on its best day. Operators live with its worst.
Five ways benchmarks mislead
- Cherry-picked runs. Reporting the best result instead of the distribution.
- Averages instead of tails. A good average can hide a terrible P99. Operators feel the tail.
- Short duration. Thermal throttling, memory leaks, and log growth do not appear in a ten-minute test.
- Clean conditions. No packet loss, no contention, no faults. Nothing like the field.
- Undisclosed configuration. Power modes, clock settings, and model quantization that the reader cannot reproduce.
Your users will never experience your average. They experience your worst minute.
What a composite score should capture
A single throughput number is easy to game. A composite that weighs several dimensions is much harder. When I designed the scoring behind my published hardware validations, the goal was to reward systems that stay correct and responsive under stress, not just systems that are fast when everything is perfect.
The dimensions that matter for edge decision systems are consistent: latency at the tail, not just the median; sustained behavior over long runs; recovery time after injected faults; correctness of outputs under load; and integrity of the system's own records. A device that scores well on all five is one you can deploy.
Correction, October 2026: the original edition of this issue cited a composite score from my earlier hardware validation. That score is withdrawn. The corrected validation reports no composite score: it publishes direct measurements in real units, with the method, conditions and raw data, at ariaos.dev/research. If you cannot see the conditions, do not trust the number.
Report distributions, not points
The simplest upgrade to any benchmark is to stop reporting a single value. Report the median, the P95, and the P99. Report the worst observed value. Report how many runs you did and over what duration. A reader with those five numbers can make a real decision. A reader with one number can only make a guess.
Disclose the conditions
Every published result should carry a short conditions block: hardware and firmware versions, power mode, ambient temperature if known, model and precision, concurrent workload, network state, and run duration. It takes five minutes to write. It is the difference between a measurement and an advertisement.
Run it long
If there is one change that improves almost every edge benchmark, it is duration. Many failures only appear after hours or days: slow memory growth, log volumes that fill a disk, thermal saturation, and clock drift.
Field Notes
What Days Teach That Hours Cannot
Lessons from a multi-day continuous run under fault injection.
Short tests answer whether a system works. Long tests answer whether it keeps working. Those are different questions, and the second one is the one operators care about.
One of my validations ran continuously for more than two days on a NVIDIA Jetson AGX Orin — 645 of 645 health probes answered — with fault injection along the way and audit integrity checks at the end. The goal was not to watch a dashboard stay green. The goal was to find whatever breaks slowly.
Correction, November 2026: an earlier edition described this as a fourteen-day continuous run on Apple Silicon. That Apple Silicon result is withdrawn; the sustained run cited here is the corrected measurement on NVIDIA Jetson AGX Orin. Method and raw data: ariaos.dev/research.
Things that only show up later
- Slow resource growth. Small leaks that look like noise on day one look like a trend line within a few days.
- Log and cache volume. Anything that writes without rotation eventually becomes the failure.
- Scheduled events. OS updates, certificate checks, and maintenance jobs that never fire in a short test.
- Recovery fatigue. A system that recovers from the first injected fault cleanly may not recover from the tenth.
Day one tells you if it works. Day two tells you if you can leave it alone.
How to run a long test without babysitting it
Automate the observation before you start. Log latency, memory, disk, and temperature on a fixed interval to a file on a different volume than the system under test. Schedule fault injections in advance so they happen without you. Then leave it alone. The most common way to ruin a long run is to keep fixing small things during it, which resets the experiment.
At the end, compare day one against the last day across every metric. Flat lines are the result you want. Any slope is a finding worth chasing.
Why this matters beyond the lab
Field deployments do not get a restart every evening. A system headed for a remote site, a vehicle, or a disconnected facility has to run unattended for as long as the mission requires. Duration testing is the closest thing a lab has to that reality.
Systems Brief
A First-Time SBIR Firm's Reality Check
What small technical firms should know before chasing their first federal research award.
The Small Business Innovation Research program and its sibling, Small Business Technology Transfer, are among the best paths for small firms to fund research the government needs. They are also widely misunderstood by first-time applicants, including, for a while, me.
What the program actually buys
Phase I funds feasibility: can this approach solve this agency's stated problem? Phase II funds development and demonstration of a prototype. Phase III is commercialization with non-SBIR money, and it is where the real long-term value sits. The program rewards a clear problem-solution fit far more than a polished company.
Lessons worth learning early
- Answer the topic, not your roadmap. Reviewers score fit to the stated need. A great technology aimed at the wrong problem loses.
- Open topics are a real opening. Several agencies run open or broad topics that welcome novel approaches outside predefined problems.
- Register early. SAM, the agency submission portals, and login credentials take time. Do it months before a deadline.
- Plan for compliance. Defense work eventually brings cybersecurity requirements such as NIST SP 800-171 and an SPRS score. Know the path before you need it.
- Skipping a round is a valid strategy. Submitting a weak proposal to a poorly matched topic costs more than waiting for a strong one.
The best SBIR proposal reads like the answer to a question the agency already asked.
The transition question
Every reviewer is quietly asking whether this work will ever reach a user. Proposals that name a transition path, a program office, a prime contractor partner, or a commercial market that funds itself, stand out. Build those relationships before the proposal, not after the award.
Toolbox
A Soak-Test Logger in 40 Lines
Record latency over days, then read the tail, not the average.
This pair of scripts probes an HTTP endpoint on an interval, writes every result to CSV, and then summarizes the distribution. Point it at a health or inference endpoint on the system under test and let it run for as long as you can stand to wait.
#!/usr/bin/env bash
# soak.sh: probe an endpoint every N seconds, log to CSV
# usage: ./soak.sh http://host:8080/health 5 soak.csv
URL="${1:?url}"; EVERY="${2:-5}"; OUT="${3:-soak.csv}"
echo "ts,http_code,total_s" >> "$OUT"
while true; do
R=$(curl -s -o /dev/null --max-time 10 \
-w "%{http_code},%{time_total}" "$URL" || echo "000,10")
echo "$(date -u +%FT%TZ),$R" >> "$OUT"
sleep "$EVERY"
done# summarize.py: python3 summarize.py soak.csv
import csv, sys, statistics as st
rows = list(csv.DictReader(open(sys.argv[1])))
ok = [float(r["total_s"]) * 1000 for r in rows
if r["http_code"].startswith("2")]
fail = len(rows) - len(ok)
ok.sort()
pct = lambda p: ok[min(len(ok) - 1, int(p * len(ok)))]
print(f"samples {len(rows)} failures {fail} "
f"({100 * fail / max(1, len(rows)):.2f}%)")
print(f"median {st.median(ok):.1f} ms p95 {pct(.95):.1f} "
f"p99 {pct(.99):.1f} worst {ok[-1]:.1f}")Run the logger on a different machine than the system under test, so the measurement does not compete with the workload it is measuring.
Field Checklist
Benchmark Honesty Check
Run every published number through this list before it leaves the building.
- Results report median, P95, P99, and worst observed value.
- The number of runs and total duration are stated.
- Hardware, firmware, power mode, and model precision are disclosed.
- Concurrent workload and network state during the test are disclosed.
- At least one run lasted long enough to expose slow failures.
- Faults were injected and recovery time was measured.
- Raw data is kept so the result can be re-examined later.
- A skeptic could reproduce the test from the published conditions.
Score yourself honestly. Every unchecked box is a finding, not a failure.
Correction, October 2026: the AriaOS figures cited in the original edition of this issue are withdrawn. Some came from a built-in stress simulator, and some had no source. The figures above are from the corrected validation on NVIDIA Jetson AGX Thor, NVIDIA Jetson AGX Orin and an AMD EPYC enterprise node. Method, limitations and raw data: ariaos.dev/research.
Next issue: Chaos on Purpose — fault injection as a design discipline.
A ResilientMind AI publication. Views are the author's own. Code samples are provided as-is for test environments. © 2026 ResilientMind AI.