The version of SaltbyPep we first installed at a commercial aging facility did not look much like what we have now. The sensor network worked in our lab. The edge device ran for weeks without issue in a controlled test environment. The model made reasonable predictions on our training data. When we put it in a real facility for the first time, three things failed in the first week that we had not thought to test for. This article is about what those were and what it took to fix them.
I am writing this partly as a field note and partly because I see a pattern in hardware product development where the initial deployment is framed as a success story once the problems get solved. The actual experience of the first deployments is more instructive than the retrospective. The problems were not exotic. They were the kind of problems that look obvious after the fact and required several iterations to solve correctly.
The dropout problem we did not anticipate
The first facility where we deployed is in Gyeonggi Province, an aging room in an existing premium beef operation. The facility manager runs two dry-age chambers and one wet-age cold room. The chambers are in a basement level, and the network infrastructure is shared with the rest of the facility, which includes a small retail area on the ground floor that has significant traffic during weekend peak hours.
During the first week, our local edge device started logging sensor dropouts at irregular intervals. Not consistent dropouts that would suggest a faulty sensor cable or a loose connector. Irregular, brief, recurring. The pattern correlated with facility activity patterns but not in an obvious way. It took us several days to identify that the local Wi-Fi access point serving the basement was also handling the retail floor, and the combination of refrigerator compressor cycling and periodic network congestion from the retail floor was causing intermittent association drops between our sensors and the access point.
We had tested our dropout recovery in the lab by pulling a sensor cable. That simulates a clean disconnect and reconnect. The real failure mode was partial: the sensor was physically connected, the firmware believed it was associated, and the data it was sending was being lost between the sensor and the edge device without either party recognizing the loss for up to several minutes at a time. Our sensor health state machine was not designed to catch this category of failure.
Why the recovery protocol took four iterations
The obvious fix was to add a watchdog: if no data arrived from a sensor within a defined window, trigger a reconnection. We added this. It worked until a power event at the facility caused all sensors to reboot simultaneously. When they came back up, all of them tried to reconnect within a few seconds of each other, and the association storm knocked our edge device off the local network for long enough that we lost a continuous window of data.
Fix two was staggered reconnection: sensors came back online with randomized delays distributed across a 90-second window. This prevented the association storm. It introduced a new problem: during the staggered reconnection window, the edge device received partial sensor data that looked like a legitimate state to the model but represented a physically inconsistent condition, because half the sensors were reading the current chamber state and half were coming online from a cold boot with stale firmware state. The model did not know to treat the reconnection interval as an unreliable window.
Fix three was a reconnection-awareness flag in the edge device: when any sensor entered a reconnection cycle, the device marked the current state estimate as provisional and refused to update the cumulative enzyme activity model until all sensors had reported at least two consecutive readings within expected range. This worked well for planned power events but failed for unplanned brief brownouts, where the edge device itself would restart without executing its clean shutdown sequence, leaving the model state file in a partially written condition on the local storage.
Fix four was write-ahead logging on the state file: every model state update was first written to a transaction log, and the main state file was only updated after the log write confirmed. On restart, the device checked whether the transaction log and state file were consistent, and recovered the most recent consistent checkpoint if they diverged. This is a standard database approach and not a novel idea. It took three field failures to arrive at it because we had been treating the state file as an in-memory structure that happened to be persisted, not as a database that required write guarantees.
What we learned about testing for the actual environment
The failures described above share a common structure: we tested for clean versions of failure modes, not for the messy ones that occur in a real facility. Clean disconnect, not partial association loss. Orderly shutdown, not brownout. Single sensor failure, not simultaneous multi-sensor restart.
We did not make this mistake because we are careless about testing. We made it because testing for messy real-world failure modes requires either a realistic environment simulation, which is hard to build for a system that depends on physical sensor hardware and facility-specific network infrastructure, or a large number of real deployments that surface the failure modes empirically. We had neither when we did the first installation. What we had was a technically sound design that worked correctly under the failure modes we had thought to test.
The practical implication for the next installation was to spend a day before commissioning systematically inducing failures in the installed system: cutting the power to individual sensors while the system was live, pulling the edge device power while a model update was in progress, and simulating network congestion by saturating the access point with traffic from a laptop. This destructive validation before handoff became part of our standard installation protocol.
The sensor placement iteration
The hardware reliability issues consumed most of the first two weeks of the initial deployment. The sensor placement question took longer because it required actual aging runs to evaluate.
Our initial sensor placement was based on literature and our own lab testing: temperature sensors at the geometric center of the chamber, mid-height, away from airflow obstructions. Humidity sensors near the fresh-air intake where we expected the humidity to be most variable. This placement is sensible for measuring average chamber conditions, which is what most building management monitoring does.
For aging prediction, we discovered that average chamber conditions are less informative than we expected, and that the distribution of conditions across the chamber matters more. A three-degree temperature gradient from top shelf to bottom shelf in a dry-age chamber, driven by natural convection, produces measurable differences in enzyme activity rate between cuts on different shelves. A center-of-chamber sensor reads the middle of this gradient and systematically mischaracterizes both the hottest and coolest positions.
We added sensors at the chamber corners and at multiple heights in the second facility installation. The additional data from the distributed placement improved our spatial characterization and reduced the variance in our endpoint predictions for that chamber. It also increased cost and installation complexity. Finding the minimum sensor count that captures the distribution adequately without over-instrumenting is still ongoing work. The answer is probably chamber-specific and depends on how well the facility controls airflow.
Replacing judgment with data, and what that actually means
The title of this article comes from how we described the project to ourselves in the early stages: we wanted to replace the expert's subjective feel for an aging batch with objective sensor data. In the first installation, we quickly learned that this framing was incomplete.
The expert's judgment we were trying to replace is not just a read of current conditions. It is an integration of current conditions, history, pattern recognition built from many previous batches, and an intuitive understanding of how the specific chamber, product source, and season interact. We were replacing the integration mechanism, not the judgment criteria. And we were doing it with a model that, at that point, had access to far less historical context about that specific chamber than the aging master did.
We are not claiming we already do this better than the expert. We are building toward it. The honest position is that for novel situations, unfamiliar chambers, and edge cases, the aging master's judgment is still more reliable than our model. Where we add clear value is in consistency across multiple batches in a well-characterized chamber, and in catching sensor-detectable events that a periodic human inspection would miss. That is a useful starting point for what a system like ours should claim to do, and everything beyond it requires a longer track record than we currently have.
The engineering story, in the end, is not about replacing expert judgment. It is about building something that can run continuously, survive the failure modes of real facilities, and accumulate the data needed to eventually earn the comparison.
