Issue 03 · Reliability

Chaos on Purpose

If you have not broken your system deliberately, the field will break it for you — at the worst possible time.

Field Notes from the Edge — By Joseph C. McGinty Jr. — November 4, 2026

From the Editor

There is a moment in every serious test program when you stop trying to prove the system works and start trying to prove it fails gracefully.

That shift is what this issue is about. Fault injection is not destruction for its own sake. It is the fastest way to learn what your system actually does when the assumptions underneath it give way.

We also tackle a term that gets stretched more than almost any other in defense technology: technology readiness level. If you claim one, you should be able to defend it in a room full of skeptics.

Break it on purpose. Write down what happened. Fix it. Repeat.

Feature

Chaos on Purpose

Fault injection turns unknown failure modes into known, tested behavior.

Large internet companies popularized chaos engineering by randomly terminating production servers to prove their services could survive it. The idea is older than the name. Field engineers have always known that the only trustworthy system is one that has already failed in front of you and recovered.

For edge and mission systems, the discipline matters even more. There is no on-call team three floors away. There may be no network to page anyone. The system has to absorb the failure itself.

Start with a hypothesis

Chaos without a hypothesis is just vandalism. Every injected fault should begin with a written prediction: when this component fails, the system will do this, within this time, and the operator will see this. Then inject the fault and compare reality to the prediction. The gaps are your findings.

The fault catalog

Process faults. Kill services, containers, and workers. Does supervision restart them, and does state survive?
Resource faults. Fill the disk, exhaust memory, pin the CPU. Does the system degrade or collapse?
Network faults. Drop, delay, and partition links. Does it switch to local behavior on its own?
Power faults. Pull power mid-write. Does storage come back consistent?
Time faults. Skew the clock. Do certificates, schedules, and logs still make sense?

Every fault you inject in the lab is one you do not have to discover in the field.

Measure recovery, not just survival

Surviving a fault is the minimum. The useful measurements are how long recovery takes, whether any data or decisions were lost, and whether the system told the operator what happened. A system that recovers silently in four minutes may be worse than one that recovers loudly in eight, depending on the mission.

Blast radius first

Begin with small, reversible faults in isolated environments. Expand only when the system handles the small ones cleanly. Always have a kill switch for the experiment itself, and never run uncontrolled chaos on hardware whose data you cannot afford to lose. Discipline is what separates fault injection from an outage.

Make it routine

The real value comes from repetition. A fault that was handled correctly last month may not be handled correctly after this month's changes. Fold your fault catalog into regular testing so that resilience is verified continuously, not once before a demonstration.

The chaos rig I run on an HP ProLiant G8 running Ubuntu 24.04 follows this approach across network degradation and recovery behavior. The Field Notes in this issue explains why I chose such old hardware on purpose.

Field Notes

Why My Chaos Rig Is a Decade-Old Server

Old hardware is not a compromise. For resilience testing, it is a feature.

When people see that my chaos rig runs on an HP ProLiant G8, the reaction is usually polite confusion. Why test a modern AI governance stack on a server generation that has been retired from most data centers?

Because the field is full of old hardware. Mission systems, public sector agencies, and industrial sites routinely run equipment far past its intended life. Software that only behaves well on current-generation hardware is software that will surprise someone, badly, in the places that need it most.

What older hardware exposes

Tighter resource margins that reveal waste and contention a modern box would hide.
Slower storage that turns sloppy write patterns into visible latency.
Aging components that make real hardware faults more likely, which is exactly what a chaos rig needs.
Modern OS on old metal, a combination that surfaces driver and firmware edge cases.

If it runs well on the worst hardware you can find, it will run well on the hardware your customer actually has.

The setup

Ubuntu 24.04 on the G8, with network degradation applied through traffic control, scheduled service failures, and recovery measured from the moment each fault was injected to the moment normal behavior resumed. Nothing exotic. The value was in the patience and the record keeping, not the tooling.

The lesson generalizes. Test where the margins are thin. Your strongest results will come from your weakest hardware.

Systems Brief

TRL Without the Spin

Technology readiness levels are a shared language. Inflating them breaks the language.

Technology readiness levels, originally developed at NASA and adopted across the Department of Defense, describe maturity on a nine-level scale. TRL 1 is basic principles observed. TRL 9 is an actual system proven through successful mission operations. The levels in between are where most claims, and most disputes, live.

The levels that matter most

TRL 4: component validation in a laboratory environment.
TRL 5: component validation in a relevant environment.
TRL 6: a system or subsystem model or prototype demonstrated in a relevant environment.
TRL 7: a system prototype demonstrated in an operational environment.

The jump from 6 to 7 is the one most often blurred. TRL 6 means a representative prototype has been demonstrated under conditions that simulate the real environment. TRL 7 means it has been demonstrated in the real environment. Those are different claims, and reviewers know it.

Claim the level you can defend with evidence, not the level you hope to reach next quarter.

What evidence should look like

A credible readiness claim points to specific tests: the environment, the conditions, the duration, the failures injected, and the results. For edge systems, "relevant environment" should include degraded networks, sustained load, and fault recovery, not just a functioning demo.

Advice for buyers and reviewers

Ask what test supports the claimed level, and ask to see it. A firm that can hand over published methodology and results is giving you something valuable. A firm that cannot is giving you an estimate. Both are fine, as long as everyone knows which one they are holding.

Toolbox

A Safe Random-Failure Injector

Randomly stop containers on a test host, with a dry run and a protect list.

This script picks a random running Docker container that is not on your protect list and stops it, logging every action. It defaults to dry-run mode so nothing happens until you explicitly ask. Use it only on test hosts.

#!/usr/bin/env bash
# chaos.sh: stop one random container, safely
# usage: ./chaos.sh            (dry run)
#        LIVE=1 ./chaos.sh     (actually stop it)
set -euo pipefail
PROTECT="${PROTECT:-dns|proxy|db|monitor}"
LOG="${LOG:-chaos.log}"

mapfile -t CANDIDATES < <(docker ps --format '{{.Names}}' \
  | grep -Ev "$PROTECT" || true)
[ "${#CANDIDATES[@]}" -eq 0 ] && { echo "nothing eligible"; exit 0; }

TARGET="${CANDIDATES[RANDOM % ${#CANDIDATES[@]}]}"
STAMP="$(date -u +%FT%TZ)"

if [ "${LIVE:-0}" = "1" ]; then
   docker stop "$TARGET" >/dev/null
   echo "$STAMP STOPPED $TARGET" | tee -a "$LOG"
else
   echo "$STAMP DRY-RUN would stop $TARGET" | tee -a "$LOG"
fi

Make it a real experiment

Write your hypothesis first: what should restart it, and how fast?
Run it on a schedule with cron during a long soak test.
Compare the chaos log timestamps against your monitoring to measure recovery time.
Expand the protect list for anything whose loss you are not ready to test yet.

Signal vs. Noise

Short Takes

Four ideas worth keeping, in the time it takes to read them.

Uptime is not resilience.

A system can report 100% uptime while quietly making degraded decisions. Measure correctness under stress, not just whether the process is alive.

Demos are not tests.

A demonstration proves something can work once, in good conditions, with its builder driving. Ask for the run that went wrong.

Redundancy has a cost.

Every second copy adds complexity and new failure modes. Add redundancy where loss is unacceptable, not everywhere by reflex.

Recovery should be boring.

If recovering from a common fault requires a hero, the system is not finished. The best recovery is the one nobody notices.

Field Checklist

Chaos Experiment Card

One card per experiment. No card, no chaos.

Hypothesis written: what fails, what should happen, and how fast.
Blast radius limited to a test environment or isolated component.
Abort procedure defined, and the person who can trigger it is named.
Monitoring confirmed working before the fault is injected.
Fault injected and exact time recorded.
Recovery time, data loss, and operator visibility measured.
Result compared against the hypothesis, gaps written down.
Fix scheduled, and the experiment added to routine testing.

Score yourself honestly. Every unchecked box is a finding, not a failure.

Correction, November 2026: an earlier edition of this issue cited AriaOS figures (a composite score, uptime, latency, endpoint and TRL 6 claims) and referred to the HP ProLiant G8 chaos run as a "published validation." Those AriaOS figures are withdrawn — some came from a built-in stress simulator, and some had no source — and should not be cited. The chaos methodology in this issue stands on its own. The corrected validation reports direct measurements in real units, with method, limitations and raw data: ariaos.dev/research.

Next issue: Storage Is the Silent Failure — power, disks, and data that survives.

All issues