High CPU utilization from `diagnosticsprocess` causes observable robot loop time spikes

[bhall-ctre]

This was tested on the pre-release 2027.0.0-beta13 (192) image.

According to htop, the diagnosticsprocess service uses ~23% CPU core utilization (non-normalized) and elevates the iodaemon CPU core utilization from 6% to ~43%, combining to a net increase of ~60% CPU core utilization. This increase is not tied to usage of the web interface.

CPU utilization with diagnosticsprocess:

CPU utilization without diagnosticsprocess:

This causes observable spikes in robot loop times from things like thread preemption/scheduling when the robot program is not running at RT priority. For example, below demonstrates the timing of a very simple program that does nothing but measure and log how long it takes to call setControl once on a Phoenix 6 device. Everything to the left of 180 seconds is with diagnosticsprocess running, and to the right without it. While there are a couple of timing spikes without the process running, they are acceptably small by comparison (sub-ms).

These timing spikes are also entirely resolved by setting the main robot thread to RT priority 1 (the lowest RT priority), but we'd rather not have teams use RT priority due to the risk of thread starvation of WPILib and vendor threads.

[auscompgeek]

I wonder if there's a recent regression. On the alpha 12 image htop reports iodaemon and diagnosticsprocess are each using roughly 30-35% CPU with no devices connected.

[bhjelstrom]

In an upcoming update, many topics are consolidated into 2 structs, and the NT update rate for fault and battery data is reduced from 500hz to 50hz.

Once we drop the rest of the deprecated IMU fields, we will be here:

There's more room for improvement beyond these changes

[bhjelstrom]