The problem with ntpd is the sudden performance degradations which can occur when conditions deviate from ‘ideal’. We have detailed these robustness issues of ntpd in prior work, including [17, 18]. Simply put, when path delay variability exceeds some threshold, which is a complex function of parameters, stability of the feedback control is lost, resulting in errors which can be large over small to very long periods. Recovery from such periods is also subject to long convergence times […]
Now to the results. As expected, and from the very
first migration, ntpd exhibits extremely large errors (from
-1 to 27 s!) for periods exceeding 15 minutes (see zoom
in middle plot) and needs at least another hour to converge
to a reasonable error level.
from Virtualize Everything but Time, Broomhead ,Cremean,Ridoux,Veitch
TimeKeeper is a “from ground up” implementation of clock synchronization over both PTP and NTP and does not share NTPd limits.
Fate has made me the “money guy” for OpenSSL so I’m going to talk about that for a bit.
As has been well reported in the news of late, the OpenSSL Software Foundation (OSF) is a legal entity created to hustle money in support of OpenSSL. By “hustle” I mean exactly that: raising revenue by any and all means. OSF typically receives about US$2000 a year in outright donations and sells commercial software support contracts and does both hourly rate and fixed price “work-for-hire” consulting as shown on the OSF web site. The media have noted that in the five years since it was created OSF has never taken in over $1 million in gross revenues annually.
Thanks to that publicity there has been an outpouring of grassroots support from the OpenSSL user community, roughly two hundred donations this past week along with many messages of support and encouragement. Most were for $5 or $10 and, judging from the E-mail addresses and names, were from all around the world. I haven’t finished entering all of them to get an exact total, but all those donations together come to about US$9,000.
OpenSSL uses a “give away code and charge for consulting” model that FSMLabs began with in 1999. We couldn’t make it work either.
Measured on Amazon EC2.
Standard deviation from reference time over 20 hours
TimeKeeper: 23 microseconds
NTPd: 6000 microseconds (6 milliseconds)
Cassandra is a highly-distributable NoSQL database with tunable consistency. What makes it highly distributable makes it also, in part, vulnerable: the whole deployment must run on synchronized clocks.
It’s quite surprising that, given how crucial this is, it is not covered sufficiently in literature. And, if it is, it simply refers to installation of a NTP daemon on each node which – if followed blindly – leads to really bad consequences. You will find blog posts by users who got burned by clock drifting. [Holub]
Cassandra labels updates with times and then uses those times to figure out the freshest update. Suppose machine A and B are both working with a database of records. B updates record R, then A updates record R and 3 out of 5 machines holding copies of R get this second update. Then B asks for record R and the read operation gets 5 responses – it can throw away the stale copies of R and forward the fresh copy because the timestamp on the update from A is more recent than those on the older copies. But this only works if the times are right – which means you need solid clock synchronization. Time synchronization clients like NTPd and PTDp (and variants) will silently fail, so the system can lose synchronization without notice. This will silently corrupt data. One of the most sophisticated parts of TimeKeeper is the algorithm to detect incorrect times, to failover where possible, and to alarm where not possible.
See: TimeKeeper, Clouds, Big Data
Miscreants who earlier this week took down servers for League of Legends, EA.com, and other online game services used a never-before-seen technique that vastly amplified the amount of junk traffic directed at denial-of-service targets.
Rather than directly flooding the targeted services with torrents of data, an attack group calling itself DERP Trolling sent much smaller sized data requests to time-synchronization servers running the Network Time Protocol (NTP). By manipulating the requests to make them appear as if they originated from one of the gaming sites, the attackers were able to vastly amplify the firepower at their disposal. A spoofed request containing eight bytes will typically result in a 468-byte response to a victim, a more than 58-fold increase. [Ars Technica]
Servers running TimeKeeper would diagnose such attacks early and ignore them. In fact, whether TimeKeeper is serving NTP or Precision Time Protocol (PTP) it will shrug off many of the exploits that can bring down networks running free software time synchronization. The ubiquitous NTPd free server and client software is designed to expect a well controlled network with a limited range of possible errors and, certainly, without malicious requests. TimeKeeper is designed for an enterprise environment where infrastructure failures are not tolerable. The protocols simply define how clock updates are distributed on the network and the format of the packets, they do not solve problems of network robustness or security. Security, failover, audit trail, manageability and other critical enterprise service properties are the responsibility of the software that implements these protocols.
Managers of enterprise networks that depend on time distribution and synchronization need to determine whether these are critical services or not. If not, then lack of instrumentation, legacy security holes, and in-house patching may be acceptable. On the other hand, if timing is a critical service, then enterprise qualified software components are not optional. This issue should be of particular concern to firms that offer their own customers time as a service because failures on the customer side might be perceived by customers as a failure in the service itself and those failures may even blowback into the provider network.
Management teams are growing more reliant on the ability to immediately access and quickly sort through massive amounts of data to find the information they need – Data Governance for Financial Institution
A “data tiedown” is a reliable and cross-checked timestamp that secures a data item or group of data items to one or more fixed times – such as time of collection, generation, or storage. Data analysis cannot compensate for bad data and distributed processing makes it easy to generate bad data whether from new inputs or by corrupting existing data. As transaction rates increase and data size grows, traditional locking techniques are both more expensive to implement and easier to get wrong. Timestamps can be an important part of data integrity assurance by imposing order on data, even if it is generated or stored by multiple processing elements that are only loosely coupled. But timestamping is in itself a fragile process. One common error mode for networks relying on IEEE 1588 Precision Time Protocol (PTP) for clock synchronization involves a loss or incorrect calculation of the number of leap seconds since the epoch. If data integrity depends on timestamps a sudden jump of 35 or 16 seconds can have a devastating impact. So the integrity of timestamps is something that also needs to be safeguarded. One way to do that is to do what TimeKeeper does – track multiple reference time sources so that the clock can be continuously cross checked against other clocks. The timestamp is then provided with a record of how well it matches secondary, tertiary, or deeper sources. When these all match, the timestamp has high reliability. Errors can also be traced – there is a forensic methodology available to validate time and locate problems. Data is then tied to a timestamp and the timestamp is tied to a log that verifies its accuracy – the combination is an example of a data tiedown.