Finally figured it out after months when I moved a wire and it started, then move the wire close to another wire and it would no longer start. These were wires that would typically be close, so my guess is one of the wires was now generating enough noise that it was bleeding over into some other system and causing it to fail. As the car bounced around the proximity of the wires changed and lead to the random behavior.
EMF can do some seemingly crazy things. I built a kiln controller and the initial version would sometimes randomly lockup, reboot in the middle of an operation, or do other seemingly "crazy" things. Sometimes even the hardware watchdog would stop functioning.
Turns out contactors pulling in and out a 5000w load generates some strong EMF and sometimes that EMF is enough to cause random glitches to the CPU or other hardware.
Switching to high power solid state relays completely solved the problem while keeping the system compact. The actual silicon transistor was so big you could have drawn the mask by hand and it was attached to a heat sink half the size of an adult fist. I was initially worried about reliability but (knock on wood) 8 years later the system is still working without issue.
Yeah. I wrote the code for a controller that managed the temperature of a lubricant. The heater was a propane-fired burner that was mounted on the same skid. Every so often we'd get a random reboot. Finally, when it was very quiet, I heard a soft 'tic' just before I noticed that the CPU had rebooted again. EMI from the spark gap that ignited the propane was coupling back into the I/O lines and would occasionally reset the CPU.
One of those things where if it had happened 100% of the time we'd have figured it out quickly. But it was so infrequent that no one thought of that as a cause.
We had one access point (unifi) in our datacenter which was consistently failing.
fail, RMA, fail, RMA, probably replaced that thing 5 times. It was also incredibly unreliable.
Meanwhile all the other access points (probably 10 of them in total) had 0 issues.
Eventually realized that the cable for that AP was running perpendicularly over the conduits which fed power into our suite, so, about 1MW of power. Relocated the cable so it was farther from the conduit and it never had an issue again. Makes you wonder about the effect of working in such environments.
Most likely this solid state relay has a builtin flyback diode that was not in the circuit of the contactor. The diode should be as close as possible to the load to reduce the EMF.
Finally figured it out after months when I moved a wire and it started, then move the wire close to another wire and it would no longer start. These were wires that would typically be close, so my guess is one of the wires was now generating enough noise that it was bleeding over into some other system and causing it to fail. As the car bounced around the proximity of the wires changed and lead to the random behavior.