Building software that runs a 24/7 operation
Notes on building applications that operations depend on around the clock, where the interesting design question is not uptime but what the system does when something upstream fails.
Key takeaways
- Uptime is the requirement people write down. Degraded-mode behaviour is the one that matters at 3am.
- Store and forward, idempotent writes, backpressure and circuit breakers make failure boring instead of catastrophic.
- The OT boundary is read-only, segmented and deliberate, never a normal business integration.
- With no maintenance window, releases need compatible migrations, versioned interfaces and a flag that turns a change off.
Software written for an office can assume someone will notice when it breaks. Software that a 24/7 operation depends on cannot. The shift crew at 3am is not going to raise a ticket and wait.
The requirement people write down is uptime. The requirement that actually matters is degraded-mode behaviour: what the system does when a dependency it does not control stops responding.
Design for the failure you cannot prevent
- Store and forward at the edge, so a dropped link delays data rather than losing it.
- Idempotent writes, so a retry after a timeout cannot double-count production.
- Queues with backpressure, so a slow consumer degrades throughput instead of collapsing the system.
- Graceful degradation, so the dashboard shows stale data clearly labelled as stale rather than a blank screen or, worse, a confident wrong number.
- Timeouts and circuit breakers on every external call, including the ones to systems that have never failed yet.
The OT boundary is not a normal integration
Reading from control systems and historians carries obligations that a business integration does not. The data flow should be read-only by design, network segmentation should follow the client security standards, with frameworks such as the zones and conduits model in the ISA/IEC 62443 series available where they help, and remote access needs to be deliberate rather than convenient.
The architectural separation between operational and enterprise systems described in ISA-95 is a useful frame here, and OPC UA gives a vendor-neutral way across that boundary without bespoke drivers per site.
Deploying without a window
If the operation never stops, there is no maintenance window to hide a release in. That constraint shapes the whole delivery approach: backward-compatible database migrations, versioned interfaces, and progressive rollout so a bad release affects a fraction of users before anyone notices.
Feature flags are worth the overhead here. Being able to turn a change off without a deployment is the difference between a two-minute recovery and an incident.
Observability, and someone to call
Logs, metrics and traces are table stakes. The part that gets skipped is deciding which conditions justify waking a human, and writing the runbook that tells them what to do about it.
Support is a design input, not an afterthought. If a system genuinely needs 24/7 support, that shapes how much operational complexity is acceptable in the first place. The most reliable feature is the one nobody had to build.
Boring technology, deliberately
Operational software rewards conservative choices. Well-understood databases, mainstream frameworks and simple deployment topologies mean the on-call engineer at 3am is debugging a familiar system rather than a novel one.
Interesting technology is best spent where it creates value for the operation, not on the parts that simply need to keep working.
At 3am, the most valuable property a system can have is being unsurprising.
How we approach an operational build
Define criticality and modes
Agree what must keep working, what may degrade, and what the system does when each dependency fails.
Architect for failure
Store and forward, idempotency, backpressure and a read-only, segmented OT boundary.
Test the failure paths
Exercise the degraded modes deliberately, because untested failure handling is a guess.
Deploy progressively
Backward-compatible migrations, versioned interfaces and feature flags instead of a window.
Operate and support
Alerting that reaches a human, runbooks that tell them what to do, and support sized to the need.
Define criticality and modes
Agree what must keep working, what may degrade, and what the system does when each dependency fails.
Architect for failure
Store and forward, idempotency, backpressure and a read-only, segmented OT boundary.
Test the failure paths
Exercise the degraded modes deliberately, because untested failure handling is a guess.
Deploy progressively
Backward-compatible migrations, versioned interfaces and feature flags instead of a window.
Operate and support
Alerting that reaches a human, runbooks that tell them what to do, and support sized to the need.
Common questions
What does 24/7 support actually involve?
Alerting that reaches a human, runbooks that tell them what to do, and an escalation path sized to the operation's need, up to around-the-clock cover. Support is a design input: the simpler the system, the cheaper it is to hold.
Cloud or on-premises for operational software?
Whichever the operation's connectivity, latency and data obligations support, and we build cloud-agnostic so the answer can change later. Edge components with store-and-forward cover the sites where links drop.
How do you deploy without a maintenance window?
Backward-compatible database migrations, versioned interfaces, progressive rollout and feature flags. A release becomes a series of small reversible steps rather than an event.
Key terms
Plain-language definitions from our glossary for the concepts this article leans on.
Standards and further reading
- ISA/IEC 62443 series, security for industrial automation and control systems (ISA)
- ISA-95 enterprise-control system integration (ISA)
- OPC Unified Architecture, platform-independent industrial interoperability (OPC Foundation)
- ISO 22400-2:2014 KPIs for manufacturing operations management (ISO)
- DAMA Data Management Body of Knowledge (DMBOK) (DAMA International)
- Guidelines library for mining technology and interoperability (Global Mining Guidelines Group)
Related case studies and tools
- Bridging OT and IT with custom software (case study)
- Shutdown coordination platform (case study)
- Time-Series Data Volume Calculator (tool)
Related reading
Working through something like this?
See how we approach these initiatives, or tell us what you are dealing with.