Most server failures are loud: a disk dies, a process crashes, the site is down. Memory errors are often quiet. A single bit flips in a memory module, and whatever was stored there is now slightly different. If it was a number on its way into the database, the database stores the wrong number.
We have seen this on a SENAITE server. A flipped bit corrupted a reference inside the object database. The kernel log stayed empty, because the server used memory without error correction, and such memory has no way of noticing, let alone reporting, that a bit changed. The damage only showed up when something tried to read that part of the database.
What ECC does
ECC memory stores extra check bits with every word. The memory controller corrects single-bit errors on the fly and detects errors it cannot correct, instead of handing corrupted data to the application. On Linux, both kinds are counted by the kernel’s EDAC driver and can be read from the file system.
That gives two separate questions for every server that runs a LIMS:
- Does the server have ECC memory at all?
- If it does, are errors being counted?
What HealthWatch checks
HealthWatch answers both, on every host it monitors.
The configuration, once a day. It reads the memory description
the firmware provides (dmidecode -t memory) and records a critical
finding for every installed module whose error correction type is
None. This is the check that would have flagged the server above
long before the bit flipped. It needs dmidecode installed and root
privileges; without them the check records nothing, and the counters
below still work.
The counters, on every poll. It reads the EDAC counters under
/sys/devices/system/edac/mc/ for each memory controller:
- an uncorrectable error (
ue_countabove zero) is critical: the hardware detected an error it could not repair - a rising count of corrected errors (
ce_count) is a warning: the module is degrading and should be replaced before the next error is one it cannot correct - an EDAC driver without any registered memory controller is a warning too. It typically means a chipset that does not support ECC, even if the modules claim to
Findings land on the anomalies tab, next to kernel events such as machine check exceptions, which HealthWatch already picks up from the kernel log. A burst of corrected errors is recorded once per hour, not once per error.
Check your own server
Two commands answer the two questions on any Linux server:
sudo dmidecode -t memory | grep -i "error correction"
grep . /sys/devices/system/edac/mc/mc*/ce_count \
/sys/devices/system/edac/mc/mc*/ue_countError Correction Type: None means no ECC. If the second command
finds no files, the kernel has no EDAC driver for the memory
controller, and corrected errors are not counted anywhere.
On a virtual machine, both commands describe what the hypervisor presents, not the physical memory underneath. Whether the host uses ECC is a question for whoever runs it. For a rented server, it is a question to ask before signing.
Why this matters for a laboratory
A result that changed because of a memory error looks like any other result. It passes verification if it is plausible, and it ends up on a signed report. Afterwards, nobody can show whether the number was entered wrong or changed later, because nothing recorded the change. For an accredited laboratory that is a hard position to argue from.
ECC memory on the LIMS server, and a check that it works, cost little compared to that.
Getting it
The ECC check is part of HealthWatch, which every SENAITE Care plan includes. If you run your own SENAITE server and are not sure what memory it has, start with the two commands above. If they show something you do not like, talk to us.
