Linux system clock and hardware clock accuracy?

This is a followup to the "my Debian Tweaks" post on on Jan 21. Briefly, my PC tends to spend a lot of time disconnected from the Internet, and the default clock management as implemented in /etc/init.d/hwclock.sh doesn't work very well in that case.

Debian timekeeping background

Linux on Intel-based PC's has two timekeepers, often referred to as the system clock and the hardware clock. The system clock is based on a high frequency crystal oscillator and is mainly intended for timing and scheduling programs and such. It doesn't run when the system is turned off, so it needs to be initialized with the current time when the PC is powered on. The hardware clock is a battery-backed circuit that keeps time even when the PC is off; it provides a value to initialize the system clock at power up and is rarely used for anything else. Debian systems assume the system clock is more accurate than the hardware clock, and they update the hardware clock every time the system is powered down (see /etc/init.d/hwclock.sh).

So how accurate are the crystal oscillators for system and hardware clocks?

Relative accuracies of these clocks depend on the specific PC. The only spec I found in a quick search is 500 parts per million (PPM) in the Intel HPET timer spec software-developers-hpet-spec-1-0a.pdf. The system clock may or may not be related to the HPET timer. Timing crystals are usually spec'ed at 100 PPM or better even for cheap crystals, so 500PPM sounds like a really loose spec.
The PC hardware clock typically uses a 32.768 kHz watch crystal and a battery-backed timing circuit (you can see the coin cell battery in the photos in my Dec 24 2012 posting). Watch crystals are readily available with 20 PPM accuracy specs, and the frequency is generally stable with time and temperature. For example, the CFS206 available from Digikey and other distributors is 20PPM accuracy, 3PPM per year max drift and .034 PPM per degree C temperature sensitivity. So timekeeping accuracies of a few PPM should be feasible with some calibration.
The US National Institute of Standards and Technology did an interesting study of wristwatches, tf.nist.gov/general/pdf/2276.pdf, and found that even a cheap[2] ($30) modern watch can achieve better than 1 PPM timekeeping accuracy. Older quartz watches were orders of magnitude worse, which implies that manufacturers have started calibrating the crystals in the last few years. So a PC's hardware clock should be capable of similar performance.

How good are the clocks in my PC?

Here's a quickie experiment with my main PC, an Acer Aspire 5253 notebook  with hardware clock updating disabled in hwclock.sh.  I set the hardware clock on Feb 13, 2013, and it hasn't been touched since. The hardware clock lost 6.227 seconds in 4609045 seconds (about 53 days), which is an average frequency offset of 1.35 parts per million (PPM).[1] In contrast the Linux system clock lost 0.44 seconds in 9103 seconds (about 2.5 hours), which is 48 PPM.
I checked a couple of older systems and their hardware clocks are in the 6 to 10 PPM range. So newer PC's are probably being calibrated like newer watches are. I suspect that the hardware clock is still more accurate than the system clock on the older PC's, but the difference should be less pronounced.

My Setup

The hardware clock in the Acer Aspire is more accurate than the system clock by a factor of 35, so that's why I don't let hwclock.sh update the hardware clock on shutdown; instead I manually set the time using ntpdate-debian every month or so, and manually update the hardware clock at that time. That keeps me within 3 seconds or so of true time, which is more than adequate for my purposes.
There are routines (adjtimex and such) that attempt to improve the calibration of the hardware clock and/or the system clock. They generally assume the oscillators are off by a fixed percentage from the true rate and that frequency changes due to temperature and drift are insignificant. In the 1 PPM range I doubt that that's true, so I don't bother with adjtimex and friends.

Side Notes

[1] I checked my Timex DataLink USB watch at the same time, and it lost about 2 seconds over the same interval. So it's averaging around 0.5 PPM.
[2] And if you want to spend a few thousand dollars, you can get a Grand Seiko SBGX059 which is spec'ed at less than 10 seconds per year, or about 0.3 PPM. Nice, but I think I'll stick with my sub-$100 Timex.