We had a leap second bug on the options trading desk I worked on that brought down our system. As market makers, we had an obligation to keep providing liquidity so this was a serious issue and exposed us to fines.
Our exchange co-located data centers used GPS for precisely synchronized timestamp generation but the firmware on some of the GPS hardware had a bug and failed to take into account a leap second. When that leap second was to be inserted, they became out of sync by a second.
We generated spot price feeds from each location and a component that consumed these feeds would check to ensure that they were not stale (any data more than 0.5 sec old could not be used as an input for pricing and trading). Well, a lot of exchange feeds started looking stale, and our system stopped quoting on said exchanges.
First there were some murmurs from the traders and within minutes the entire room hit a crescendo of panic. It took, I think, the better part of an hour to debug.
Can you provide the model of the GPS hardware? I work in the same industry and have set up multiple NTP / PTP timer servers that have never failed to account for leap seconds. It's almost always the downstream software that crashes. See the 2012 leap second insertion as an example.
Our exchange co-located data centers used GPS for precisely synchronized timestamp generation but the firmware on some of the GPS hardware had a bug and failed to take into account a leap second. When that leap second was to be inserted, they became out of sync by a second.
We generated spot price feeds from each location and a component that consumed these feeds would check to ensure that they were not stale (any data more than 0.5 sec old could not be used as an input for pricing and trading). Well, a lot of exchange feeds started looking stale, and our system stopped quoting on said exchanges.
First there were some murmurs from the traders and within minutes the entire room hit a crescendo of panic. It took, I think, the better part of an hour to debug.