In my case, the problem occurred on an up-to-date raspbian image with the unified firmware installed/built manually.
I have configured the DS1307 RTC. Its on the network 24/7 and using NTP. The ntp logs show this keeps time to within 100ms (and my understanding is that the minor NTP time adjustments are fed back to the RTC. It is networked via an Edup rtl8192cus USB dongle, no USB hubs or anything.