This is what the scheduler latency looks like on our isolated core:
# Total: 000300000 # Min Latencies: 00001 # Avg Latencies: 00005 # Max Latencies: 00059 # Histogram Overflows: 00000
(those are uS!)
Some notes that weren't included in this post:
I do have my LEA-M8T generating time pulses at 16 Hz, with a dpoll of -4 in the chrony config. The poll interval of 16 seconds means it gets 256 samples which also improved the stability.
I do also have that mentioned BH3SAP GPSDO sitting next to me on my desk. I had Claude update the firmware to allow "flywheel" behavior so it continues to generate pulses in the absence of GPS PPS. That's coming in another post. Also had Claude update the firmware to spit out TSIP (trimble proprietary protocol) info for lady heather.
Going to go through the comments and answer a couple other things I see.
Happy to answer any other questions!
That should give you 4-5x less drift than his results (though you could pair the techniques for even better figures)
RTLinux has a module feature to sync the scheduler to an external pin state. It is an obscure feature...
Adding more processors creates a well-known named-problem with metastability:
https://en.wikipedia.org/wiki/Clock_domain_crossing
Real-time is not guaranteed latency, and the Pi is not like Zynq fpga. =3
This is way more direct than spacebar heating.
You could also add a transistor attached to the resistor and a GPIO and use the clock drift as a proxy for temperature. PID is probably enough but since you have a 24 hour cycle you could calculate a baseline heating schedule.
I have test equipment made as recently as the early 2000s that uses a crystal oscillator in an oven as a frequency standard. It takes a good five minutes to fully stabilise.
Always fun new things to learn when doing something "simple" like setting up an NTP server!
I was thinking along these lines as well. Put a metal block on the CPU and oscillator for thermal mass (not sure if separate blocks would be better). Ideally, with a large enough thermal capacity, the block should reach an average temperature and remain there.
Inertia is also good even if the temperature is not constant: clock drift can be measured and compensated. If the temperature rises slowly, the clock speed will increase slowly: the rate can be measured and compensated for. Jitter is the issue here, and thermal inertia should dampen it.
It may also be worth preventing convection from happening on the board. Putting the Pi in a wool sock may not be the best idea depending on its temperature, but an electrically insulating thermal conductor (or an electrical insulation layer + steel wool may do it).
Heatsinks may also be counter-productive (if they have a small thermal capacity), as their temperature depends on room temperature, which changes during the day.
You're right that this is a over-controller oscillator. The goal generally with ovens is to keep heat! (To an extent of course.)
That would likely make it worse. The trick here is that the other cores are running at essentially their maximum temperature and and will dynamically reduce their clockspeed if required to keep from going above that limit. In essence, the environment becomes actively temperature controlled. If the ambient heat goes higher, the cores clock lower, if it gets colder the cores clock higher (up to a point).
If you add too much heat dissipation, the total power used by those cores might not be enough to keep well above ambient.
Author should experiment with liquid nitrogen ;) [1]
[11] https://www.xda-developers.com/liquid-nitrogen-cooling-raspb...
Love this (honestly). Interesting article!
https://www.usenix.org/conference/nsdi22/presentation/najafi
the method of using two PPS signals configured to be some delta time apart to detect jitter is not new. the whole learning the tempco of the crystal thing is like 80 years old.
they also avoided touching the most critical part of the issue - how would you be sure that such learned tempco is accurate enough for the estimated bound.
An advantage of constantly loading at least one core of the CPU might be preventing the deeper power states from kicking in, which should make the RX timestamping latency more stable and improve stability of synchronization of NTP clients.
Or do we have a defined standard eagle these days?
1. using a second hand single frequency Ublox LEA-M8T GNSS receiver is not a good idea in 2025 when a dual frequency GNSS receiver chip can be purchased online for as little as $20-30. buy one of those ublox ZED-F9P receivers with the 2018/2019 firmware, they get you the exact same timing performance as ZED-F9T with QEerr support. PPS jitter is going to drop from 20ns to less than 2ns after qErr compensation.
2. you can replace that highly unstable crystal oscillator with a $10 used OCXO + $20 PLL board and save that PID temperature controller to be used as the secondary oven for that OCXO. people have been doing that for Raspberry Pi for ages.
3. or you can configure your GNSS receiver to output a 10Hz signal from the PPSOUT pin so chrony can get 10 updates per second, GNSS receiver's internal TCXO is going to be more stable than raspberry Pi's crystal oscillator.
4. for more fun - just keep measuring the frequency drift vs. temperature change. a sample set of 24-48 hours of such measurements should be enough to figure out the tempco of that unstable crystal oscillator so you can get chrony to do the drift compensation. from memory, chrony supports such a temperature compensation lookup table to be specified.
good luck and have fun
2 - I've thought about replacing with ocxo but the pads are tiny. My soldering skills aren't nearly as good as my ebaying skills.
3 - it is outputting 16 Hz, chrony is doing dpoll -4 poll 4.
4 - I have done tempcomp (it is not active for the post referenced by this HN post). Fun stuff and probably my next writeup.
For future improvements, a cheap but effective win might be to put a temperature sensor on the oscillator (or two or three in various places). And use that to drive the PID loop.
Even if just experimental & not long term, it would be nice to have data on how strong the correlation is between the cpu & oscillator temperatures. To see their difference and how much that changes over time. Another graph! CPU vs txco (vs ambient?) temperatures over time.
Others have mentioned the temperature compensation. I've done it and it sounds like I should do that for the next writeup. A simple DS18B20 close to the Pi produces reasonable results.
At that point, couldn't we just use the temperature value to compensate for the drift?
Is the Pi going the Pentium 4 route?
The old Intel CPUs were grotesquely inefficient. Every single generation of Raspberry Pi has been well under 5W idle. And just so it's clear, the author is using an old Raspberry Pi 3.
From TFA:
> The RPi 3B’s 19.2 MHz oscillator is physically located near the CPU on the Raspberry Pi board, so by actively controlling CPU temperature, we’re indirectly controlling the oscillator’s temperature.
Also note that the R.Pi can even be further optimised by switching off HDMI.
https://www.pidramble.com/wiki/benchmarks/power-consumption
https://www.jeffgeerling.com/blogs/jeff-geerling/raspberry-p...
I don't think Intel has any "efficient" CPU that can go passively cooled at load though. Maybe Apple can do it for the low end SoCs.
The Pi 3 can go passively cooled and maybe even without a heatsink at load, but the newer Pis can't. Judging by the progression from 3 to 4 to 5, they will reach P4 levels of heat in the name of speed around ... 7?
> And just so it's clear, the author is using an old Raspberry Pi 3.
Yes, the author has harder problems to solve than what I'm whining about. But my concern is a bit related.
The Raspberry Pi 4 can be used without a fan. They are packaged inside keyboards, but both the Raspberry Pi 400 and the Raspberry Pi 500 are passively cooled.
The heatsink is uncomfortable to touch (it's not in a case). Pretty sure it would downclock if i removed it. So it works without a fan, but not without a heatsink (I bet the keyboards have a heatsink for the Pis built in.).
A 3 would have worked fine without any heatsink at all. At least at normal room temperature.
The Raspberry Pi 4 will throttle performance at 80C (if I recall correctly) but it can work without a heatsink. I have a R.Pi 4 working without a passive cooler in an enclosure and it reports 55C-62C most of the time.
> I bet the keyboards have a heatsink for the Pis built in
You are right, both models use a heatsink. The PCB is also different, although the respective CPUs are standard Pi 4 or Pi 5.
Incidentally, the CPU in the R.Pi 400 has a higher clock rate than the standard R.Pi 4, so it performs better.
Does the N100 not fit this criteria?
Intel tried to scale frequency up with the Pentium 4 in the name of performance, and it ended up extremely hot and power hungry. Just like some high end CPUs now, but then it applied to every model from Intel.
I suppose you don't remember when a Raspberry Pi could run fine even without a heatsink, let alone active cooling. That's more recent than the Pentium 4.
The early ‘00s were a wild time. Intel boldly stating they expected to get the P4 up to 10 GHz, AMD having to assign clock speed equivalence ratings for their chips… I also remember thinking the P4EE was insanely priced ($1000, or about $1700 in 2025 USD), but now we have >$10K Threadrippers.
"Hey claude read my previous posts and this script and generate a new blog post about this temp control stuff. Generate whatever plots you want. Here's how you can access my influxdb with the data. Here's how you can ssh into the Pi to get the exact running scripts. Ok here's my wordpress token - upload it and the pictures. Oh it looks like your .md to wordpress failed really bad, read a previous post to find out how to format stuff. Oh the tables are still not right, try again." <-- literally what I spent an hour doing last night and refining.
Maybe people would find that interesting as well!
No
> Why would this be more interesting for you if he'd written the script himself?
Was I unclear? I read blog posts to read thoughts from humans, not from computers.
Then again these are all well known optimizations (core-pinning, frequency-locking, thermal stabilization for oscillators). The interesting part is the actual measurement results over multiple days. That's something you can't get from a single prompt.