HealthWatch and ECC: catching the errors your server does not report

Servers don’t just fail loudly. They also fail quietly. A single bit flips in a DRAM module, a checksum on disk drifts, a network buffer corrupts. In consumer hardware, these events happen and no one notices. In a lab, a flipped bit can mean a wrong result printed on a signed report.

That’s why regulated labs run ECC memory. It’s RAM that detects and corrects single-bit errors on the fly, and reports multi-bit errors instead of silently corrupting data. And that’s why HealthWatch, our monitoring layer for SENAITE, now ships an ECC probe in its monitor set.

What the probe checks

For every SENAITE host, HealthWatch inspects the platform’s memory configuration and reports whether the installed DIMMs are ECC-capable, and whether the memory controller has ECC enabled and active.

It tracks the rate of corrected single-bit errors. A rising rate is a DIMM heading for failure. It also surfaces uncorrectable errors, which cause the OS to log a machine check and, on a good day, halt the affected process.

The values feed the same dashboard as ZODB size, backup age, and disk growth. A single failing DIMM shows up as a red tile before it becomes a corrupted result.

Why this belongs in a LIMS operations stack

Most infrastructure teams already monitor CPU and disk. Memory health is the blind spot. It’s invisible in cloud dashboards, it’s not in the average uptime metric, and its symptoms (occasional wrong numbers, occasional segfaults) get blamed on the application layer for months.

For labs subject to ISO 17025, GxP, or equivalent standards, “our server had a memory fault” isn’t a defensible answer to “why did the report say 4.2 mg/L when the sample was 42?”. Continuous ECC monitoring is a small piece of infrastructure that pays for itself the first time it alerts.

Getting it

The ECC probe ships with the current HealthWatch release. If you already run HealthWatch on your SENAITE hosts, an update picks it up. If not, this is a good time to look at what it covers. The SENAITE Care page walks through the full monitor set.