Issue 01 · Survivable Systems
Designing for the Day the Network Dies
Most systems are built for the network you have. Mission systems have to work on the network you lose.
From the Editor
This magazine exists because most of what gets written about AI at the edge is written by people who have never watched a system fail in a place where failure matters.
I have. Twenty-plus years across government and corporate work taught me one thing that every field engineer eventually learns: the environment does not care about your architecture diagram. Power flickers. Links drop. Hardware ages. The system either keeps making sound decisions or it does not.
Field Notes from the Edge is a small technical magazine about building systems that survive. Every issue carries one deep feature, one story from the bench, one brief for the people who buy and approve these systems, and one tool you can use the same day.
No hype. No vendor slides. Just what held, what broke, and what we learned.
Feature
Designing for the Day the Network Dies
Every system has a hidden dependency map. For most modern software, the center of that map is a network connection that is assumed to be fast, cheap, and always there. Authentication calls home. Models are served from a region. Logs stream to a collector. Configuration lives in a control plane somewhere else.
That design is fine for a shopping cart. It is a liability for anything that has to keep working in a forward operating location, a substation during a storm, a disaster response site, or a hospital running on generator power. The defense community has a name for these conditions: DDIL, short for denied, degraded, intermittent, or limited bandwidth.
Four failure modes, not one
Teams often treat offline as a single state. It is not. Each of the four DDIL conditions breaks systems differently, and each one needs its own design answer.
- Denied. No connectivity at all, possibly by adversary action. The system must operate indefinitely on what it carries.
- Degraded. The link exists but loses packets and adds latency. Timeouts and retry storms become the real enemy.
- Intermittent. The link comes and goes. State has to survive the gaps and reconcile cleanly when the link returns.
- Limited. The link works but is thin. Every byte competes for priority, so the system has to know what matters most.
A system that handles denied cleanly can still collapse under degraded, because degraded is where software keeps trying and failing. In my testing, a flaky link has caused more cascading failures than a dead one ever has.
A dead link is honest. A degraded link lies to your software all day long.
Design rule one: local first, network optional
The first discipline is to make the network an enhancement instead of a requirement. Decisions that matter should be computed on the device, with the models, policies, and data needed to make them resident on that device. Synchronization becomes something the system does when it can, not something it waits for.
This sounds obvious until you audit a real stack. Count the calls that leave the box during a normal decision cycle. Every one of them is a place where the system can stall when the link degrades. The goal is not zero calls. The goal is zero calls on the critical path.
Design rule two: fail into a known state
When something does break, the system should fall back to a state that is defined in advance and safe by design. Not a crash. Not a hang. Not a best guess. A known mode with known behavior that an operator can recognize and trust.
That means writing down, before deployment, what the system does when each dependency disappears. If the answer for any dependency is "we are not sure," that is the first thing to fix.
Design rule three: keep the record intact
Disconnected systems still have to answer for their decisions. When the link comes back, someone will ask what happened, what was decided, and why. A system that cannot produce a trustworthy record of its own actions during the outage has failed even if every decision was correct.
Local, tamper-evident logging is not a compliance checkbox. It is what lets a commander, an inspector, or an incident reviewer trust the system after the fact.
Design rule four: prove it under stress
None of this matters until it is tested. Not demonstrated on a clean bench, but tested under injected faults, degraded links, and long run times. This is the work I publish at ariaos.dev/research, and it is the standard I hold my own systems to.
/health under full CPU saturation, all three targetsThose numbers came from deliberately hostile conditions, not friendly ones. That is the only kind of number worth publishing.
What to ask before you buy
If you evaluate edge systems for a living, three questions cut through most marketing. What does this system do when the network is gone for a week? What does it do when the network is present but losing a third of its packets? And how will I know, afterward, exactly what it decided while I could not see it?
A vendor who answers all three with specifics has thought about the field. A vendor who answers with a cloud architecture diagram has not.
Field Notes
Four POST Codes and a 64-Core Swap
Upgrading a production homelab server at night, one hexadecimal code at a time.
My main lab server is a 45Drives PRO15E running Proxmox with ZFS, roughly forty containers and VMs, and two full Windows guests. It hosts my code, my staging sites, and the services I depend on daily. Upgrading it is not a weekend hobby. It is a production change.
The change: swap the 16-core EPYC 7282 for a 64-core EPYC 7V12 and take memory from 256GB to 512GB of DDR4-3200 ECC on an ASRock Rack ROMED8-2T board. Same Zen 2 Rome family, same SP3 socket. On paper, a drop-in upgrade.
Code 29: be patient
First power-on sat at POST code 29 long enough to make anyone nervous. With a new CPU and 512GB of new memory, the board has to train every DIMM from scratch. On large memory configurations that can take ten to twenty minutes and may include a self-reboot. The single most useful skill here is not touching anything.
C2 and a CMOS clear
The code moved to C2, still memory initialization, which meant progress. I cleared CMOS anyway to flush training data from the old chip. That restarts the whole process, so the patience bill comes due twice. It was worth it for a clean baseline.
B2: the option ROM trap
Next came B2, legacy option ROM initialization. The HBA spun up the drive bays in staggered groups, which is normal and audible. But the board kept stalling there. The suspect was the Intel Arc Pro B60 GPU, since Arc cards are known to dislike legacy compatibility mode, and a CMOS clear can quietly turn CSM back on.
Pulling the Arc moved the board past B2 to D6, which on AMI firmware means no console output device found. That is not a failure. It is the board telling you it got through option ROMs and simply has no GPU to draw on. The BMC's remote console handles video from there.
The fix list
- Load optimized defaults, then disable CSM for pure UEFI boot.
- Re-enable SVM and IOMMU for virtualization and passthrough.
- Enable Above 4G Decoding and Resizable BAR for the Arc.
- Confirm UEFI boot order points at the Proxmox drive.
- Power down, reinstall the GPU, and boot under pure UEFI.
The BMC is your best friend during a CPU swap. The front panel shows you a code. The remote console shows you the truth.
What this has to do with resilience
Every lesson here maps to the field. Long initialization looks like failure until you know what normal looks like. Settings you depend on can silently revert. One peripheral can block an entire boot. And out-of-band management is what turns a mystery into a diagnosis. Build systems the way you would want this server to behave: observable, recoverable, and honest about what stage it is in.
Systems Brief
DDIL Is a Buying Requirement, Not an Edge Case
For program managers, contracting officers, and anyone signing off on mission technology.
Across defense planning, contested logistics and disrupted communications are treated as the expected condition, not the exception. That shift has a direct consequence for anyone buying software: a system that only works when it can reach a data center is a system that may not work when it matters most.
The gap in most evaluations
Technology evaluations still tend to happen on clean networks, in conference rooms, with the vendor driving. Demonstrations prove a capability exists. They rarely prove it survives. The result is that DDIL behavior goes untested until the first real outage, which is the most expensive possible time to discover it.
Three requirements worth writing down
- Defined offline behavior. Require the vendor to document what the system does when each external dependency is unavailable, and for how long.
- Degraded-link testing. Require test evidence under packet loss, added latency, and intermittent connectivity, not only full outage.
- Local accountability. Require a verifiable local record of decisions made while disconnected, reconcilable when the link returns.
None of these require exotic technology. They require that resilience be stated as a requirement instead of assumed as a property.
If it is not in the requirement, it will not be in the test. If it is not in the test, it will not be in the field.
Why small firms matter here
Resilience engineering is often done best by small, specialized teams who build and test their own systems end to end. Programs that make room for them, through open topics, prototype pathways, and set-asides for veteran-owned and service-disabled veteran-owned small businesses, get access to people who have actually operated in degraded environments.
The ask for buyers is simple. Put DDIL behavior in the requirement, ask for the test data, and weigh it as heavily as features. The systems that survive will sort themselves out.
Toolbox
Break Your Own Network on Purpose
Linux traffic control can turn any test bench into a degraded link in seconds.
The fastest way to learn how your system behaves on a bad link is to give it one. The tc netem queueing discipline, built into the Linux kernel, adds latency, jitter, loss, and reordering to a network interface. Run this on a test host, never on a production uplink.
#!/usr/bin/env bash
# degrade.sh: simulate DDIL conditions on one interface
# usage: sudo ./degrade.sh eth0 [mild|bad|awful|clear]
set -euo pipefail
IF="${1:?interface required}"
MODE="${2:-mild}"
tc qdisc del dev "$IF" root 2>/dev/null || true
case "$MODE" in
mild) tc qdisc add dev "$IF" root netem \
delay 120ms 30ms loss 2% ;;
bad) tc qdisc add dev "$IF" root netem \
delay 400ms 150ms loss 10% reorder 5% ;;
awful) tc qdisc add dev "$IF" root netem \
delay 1200ms 600ms loss 30% rate 256kbit ;;
clear) echo "netem cleared on $IF"; exit 0 ;;
*) echo "unknown mode: $MODE"; exit 1 ;;
esac
tc qdisc show dev "$IF"How to use it
- Start with mild and watch retry counts and timeouts, not just whether requests succeed.
- Run bad for an hour. Degraded links expose memory growth and queue buildup that short tests miss.
- Use awful to find the point where your system should switch to offline behavior on its own.
- Always end with clear, and confirm with
tc qdisc show.
Safety: if you run this over SSH on the same interface you are connected through, you will degrade your own session. Use a second interface or console access.
Field Checklist
DDIL Readiness Check
Print it. Walk it with your team before the next deployment.
- Every external dependency on the critical decision path is listed.
- Offline behavior is documented for each dependency, with a time limit.
- The system has been tested under packet loss, not only full outage.
- Retries use backoff and caps, so a bad link cannot cause a retry storm.
- Fallback modes are defined in advance and recognizable to operators.
- Decisions made while disconnected are recorded locally and tamper-evident.
- State reconciles cleanly when the link returns, and that has been tested.
- Someone outside the build team has tried to break it.
Score yourself honestly. Every unchecked box is a finding, not a failure.
Correction, October 2026: the AriaOS figures cited in the original edition of this issue are withdrawn. Some came from a built-in stress simulator, and some had no source. The figures above are from the corrected validation on NVIDIA Jetson AGX Thor, NVIDIA Jetson AGX Orin and an AMD EPYC enterprise node. Method, limitations and raw data: ariaos.dev/research.
Next issue: Benchmarks That Lie, and how to build ones that do not.
A ResilientMind AI publication. Views are the author's own. Code samples are provided as-is for test environments. © 2026 ResilientMind AI.