Using IEEE 1588 timestamps to measure packet arrival times?

[daloev]

I need to measure the arrival time (relative to a PPS signal from White Rabbit equipment) of a subset of UDP packets arriving over a 10G Ethernet interface. We have used Endace cards for packet capture, but they have long been unsupported and they require a PC running an obsolete Linux to work.

I already have a system that uses a Xilinx UltraScale+ ZU4EV doing 10G Ethernet, and it would be wonderful if I could use the plentiful excess resources I have on the chip to capture timestamps and some other information (like MAC addresses) on network traffic.

One approach could be to build something around the SERDES, looking for the packet preamble and getting timestamps and other information. However, it seems to me that IEEE 1588 hardware timestamping is doing that already, and it would be much easier to use that if it suitable than build something from scratch.

Does anybody has experience using Ethernet hardware timestamping for this purpose? How accurate are the hardware timestamps? I would like to have accuracy to 5 ns or less.

[bson]

Isn't packet timestamping typically based on a frame detection pulse on a PHY pin?  This is then used to perform a timer capture and interrupt.  5ns seems awfully optimistic though.

[nctnico]

Microchip (formerly Microsemi, formerly Vitesse) offers switch chips which can reach these kind of timestamp accuracies according to the specs. But these aren't easy to integrate. And without knowing when the UDP packets are send and the network component delays, the 5ns target will be swamped by other unknowns.

[bingo600]

Does anybody has experience using Ethernet hardware timestamping for this purpose? How accurate are the hardware timestamps? I would like to have accuracy to 5 ns or less.

Most of those "stamp" against a 1PPS pulse.
If using GPS, just getting that to 5ns or less could be a challenge.

[ejeffrey]

Yes you can do this.  AMD has some app notes on running PTP on their FPGAs and getting generating accurate time stamps and their Ethernet MAC / PHY IP support iee1588 timing signals.  I have looked into this and talked to an AE about it but not actually done it yet.

Cisco Nexus SmartNICs supposedly just do this out of the box with  built in support for PTP, PPS signals, and high precision time-stamping.  They are basically an US+ FPGA on a PCIe card, so it should be possible to do on your own.  I think they claim 3.4 ns uncertainty, which is just based on the internal clock frequency,. since the true SOP signal is asynchronous from the FPGA timer.

[ejeffrey]

You will need a naturally stable time base on  your FPGA since you need very long integration times to get 5 ns accuracy.  Also unless you use SyncE switches, the uncertainty compounds for each switch you go through.

[ch_scr]

To get 5ns resolution you can timestamp incoming packets with a (wide) counter running at 200MHz. Even a Tang Nano 9k 20€ board can do that nowadays. No need for integration at all in that case.
Still needs a reasonable clock feeding the FPGA PLL, a decent 10MHz TCXO comes to mind.

[bson]

5ns precision and 5ns accuracy are two very different things.  The accuracy is going to be dependent on the jitter of the frame leader, and at some point more precision just allows more precise measurement of this jitter.  It takes very little noise on top of a multi-level signal to create that much jitter.

[bson]

Integration doesn't reduce jitter.  The absolute jitter of N+1 cycles of a signal is exactly the same as N cycles (assuming it's white/unmodulated).

Edit: and also, the jitter of N cycles is the same as that of 1 cycle.

[daloev]

We are measuring relative to the PPS from a White Rabbit slave whose master relies upon a GPSDO. Our measurements have shown that the variation between PTP edges is far below 1 ns, and, thankfully, averages out to zero within the limits of our measurements. I would take the FPGA clock from a slave clock output synchronized to the PPS. In other situations, we use a Metamako (now Arista) switch to timestamp packets, and that is basically a Xilinx FPGA in a box with other components that is doing what I want, but we need something embeddable. I should do as ejeffrey has done and talk to an AMD AE.

[ejeffrey]

We are measuring relative to the PPS from a White Rabbit slave whose master relies upon a GPSDO. Our measurements have shown that the variation between PTP edges is far below 1 ns, and, thankfully, averages out to zero within the limits of our measurements. I would take the FPGA clock from a slave clock output synchronized to the PPS. In other situations, we use a Metamako (now Arista) switch to timestamp packets, and that is basically a Xilinx FPGA in a box with other components that is doing what I want, but we need something embeddable. I should do as ejeffrey has done and talk to an AMD AE.

You also have the advantage that the white rabbit systems use synchronous ethernet as well.  I haven't used it, but my understanding is that it means that the ethernet transmit/receive clocks are synchronized to the internal FPGA clock and the PPS.  In normal ethernet, the start-of-packet can come at any time relative to the internal clock and the deserialization and clock domain crossing are major sources of jitter.   So 5 ns should be pretty achievable.

[nctnico]

We are measuring relative to the PPS from a White Rabbit slave whose master relies upon a GPSDO. Our measurements have shown that the variation between PTP edges is far below 1 ns, and, thankfully, averages out to zero within the limits of our measurements. I would take the FPGA clock from a slave clock output synchronized to the PPS. In other situations, we use a Metamako (now Arista) switch to timestamp packets, and that is basically a Xilinx FPGA in a box with other components that is doing what I want, but we need something embeddable. I should do as ejeffrey has done and talk to an AMD AE.

You also have the advantage that the white rabbit systems use synchronous ethernet as well.  I haven't used it, but my understanding is that it means that the ethernet transmit/receive clocks are synchronized to the internal FPGA clock and the PPS.  In normal ethernet, the start-of-packet can come at any time relative to the internal clock and the deserialization and clock domain crossing are major sources of jitter.   So 5 ns should be pretty achievable.

Yes, WR uses SyncE but one of the things you need to account for is phase shift due to the receiver needing to shift (bit slide) a number of UI (link frequency periods) to sync the recovered clock to the data stream. When taking 16 bit words from the link's datastream (recovered clock = link frequency / 20) and each UI beying 800ps, this results in a variable delay between 0ns and 20x800ps = 16ns.

Another route is to go to CERN's OHWR website and use one of the reference designs to implement a WR node with multiple ethernet ports where one port is used to synchronise to the WR switch. The traffic on the other ports can be timestamped using the internal TOD counter + bit slide.

[daloev]

Yes, WR uses SyncE but one of the things you need to account for is phase shift due to the receiver needing to shift (bit slide) a number of UI (link frequency periods) to sync the recovered clock to the data stream. When taking 16 bit words from the link's datastream (recovered clock = link frequency / 20) and each UI beying 800ps, this results in a variable delay between 0ns and 20x800ps = 16ns.

Another route is to go to CERN's OHWR website and use one of the reference designs to implement a WR node with multiple ethernet ports where one port is used to synchronise to the WR switch. The traffic on the other ports can be timestamped using the internal TOD counter + bit slide.

I have considered the idea of implementing a WR node because that would let me eliminate the add-on WR card I currently use and using it to timestamp packets would be a bonus. I just noticed BabyWR (https://ohwr.org/project/babywr) and that looks very interesting for its simplicity compared to a CUTE-WR-A7. However, it is still under development.

[tggzzz]

Does anybody has experience using Ethernet hardware timestamping for this purpose? How accurate are the hardware timestamps? I would like to have accuracy to 5 ns or less.

Light travels ~2m in 5ns. So, which end of the cable are you planning on timestamping? If timestamping both ends, which is "correct"?

Then add protocol stacks, interrupts, cache misses, TLB misses....

Then add multiple devices connected to a network switch. Then add network bridges...

You need to understand the consequences noted by Leslie Lamport 45 years ago. Start with
https://en.wikipedia.org/wiki/Lamport_timestamp
https://www.cs.cornell.edu/courses/cs6410/2018fa/slides/14-lamport-clocks.pdf
http://lamport.azurewebsites.net/pubs/time-clocks.pdf

[aeg]

Light travels ~2m in 5ns. So, which end of the cable are you planning on timestamping? If timestamping both ends, which is "correct"?

Both are correct. If you're an ordinary clock and you want to know what time it is, then you adjust the timestamp according to the path delay, as described in IEEE 1588.

Then add protocol stacks, interrupts, cache misses, TLB misses....

None of these come into play. Timestamping is by the network hardware.

Then add multiple devices connected to a network switch. Then add network bridges...

Residence time correction is performed using the correctionField value inserted in the packet by IEEE 1588 compliant switches.

The devil is in the details, but not those details.

[tggzzz]

Light travels ~2m in 5ns. So, which end of the cable are you planning on timestamping? If timestamping both ends, which is "correct"?

Both are correct. If you're an ordinary clock and you want to know what time it is, then you adjust the timestamp according to the path delay, as described in IEEE 1588.

So IEEE1588 has managed to circumvent the limitations defined by Lamport? I'm impressed; very impressed 

That would be equivalent to solving the Byzantine General's problem.

Then add protocol stacks, interrupts, cache misses, TLB misses....

None of these come into play. Timestamping is by the network hardware.

OK to some extent, but when exactly? With 5ns being the target, is that the beginning of packet, end of packet?

Then add multiple devices connected to a network switch. Then add network bridges...

Residence time correction is performed using the correctionField value inserted in the packet by IEEE 1588 compliant switches.

The devil is in the details, but not those details.

[ejeffrey]

So IEEE1588 has managed to circumvent the limitations defined by Lamport? I'm impressed; very impressed 

That would be equivalent to solving the Byzantine General's problem.

What?  No. That would only apply if you allowed the network to be an arbitrary adversary.  PTP is only used when you have control over the network.  The goal is to bypass areas of understood non-determinism, primarily buffering and switch fabrics, and measure only deterministic links, the transceivers and network cables themselves.  So switches measure the ingress and egress time and that gets applied as a correction.

OK to some extent, but when exactly? With 5ns being the target, is that the beginning of packet, end of packet?

The normal reference point is when the first byte of the Ethernet header after the preamble as it passes the PHY.  The main thing is to measure input and output packets consistently, and to either eliminate or separately measure path asymmetry due to e.g. dispersion in fibers.

Fwiw, I just measured jitter between a normal Intel 10G/1000base-T NIC operating at 1 gigabit and an FPGA and got less than 100 ns jitter even passing through a switch that doesn't do residency correction.  This isn't too surprising since there was no other traffic at the time.  You wouldn't expect a lot of buffer based jitter, but it's still surprisingly good with little effort.  Getting to 5ns is challenging but possible.

[tggzzz]

So IEEE1588 has managed to circumvent the limitations defined by Lamport? I'm impressed; very impressed 

That would be equivalent to solving the Byzantine General's problem.

What?  No. That would only apply if you allowed the network to be an arbitrary adversary.  PTP is only used when you have control over the network.  The goal is to bypass areas of understood non-determinism, primarily buffering and switch fabrics, and measure only deterministic links, the transceivers and network cables themselves.  So switches measure the ingress and egress time and that gets applied as a correction.

Any network with the possibility of contention is an an adversary.

But on reflection, I think Byzantine generals is a distraction in this context.

Where there is a finite time-of-flight, partial ordering is the best you will ever be able to achieve.

It will be interesting to see what the standards bodies define - and don't define - w.r.t. defining time on the moon relative to time on the earth. And Mars. And Alpha Centauri.

OK to some extent, but when exactly? With 5ns being the target, is that the beginning of packet, end of packet?

The normal reference point is when the first byte of the Ethernet header after the preamble as it passes the PHY.  The main thing is to measure input and output packets consistently, and to either eliminate or separately measure path asymmetry due to e.g. dispersion in fibers.

Fwiw, I just measured jitter between a normal Intel 10G/1000base-T NIC operating at 1 gigabit and an FPGA and got less than 100 ns jitter even passing through a switch that doesn't do residency correction.  This isn't too surprising since there was no other traffic at the time.  You wouldn't expect a lot of buffer based jitter, but it's still surprisingly good with little effort.  Getting to 5ns is challenging but possible.

Jitter is relevant, but isn't the principal conceptual problem with the idea of a single unique time on all interconnected processors.

[nctnico]

So IEEE1588 has managed to circumvent the limitations defined by Lamport? I'm impressed; very impressed 

That would be equivalent to solving the Byzantine General's problem.

What?  No. That would only apply if you allowed the network to be an arbitrary adversary.  PTP is only used when you have control over the network.  The goal is to bypass areas of understood non-determinism, primarily buffering and switch fabrics, and measure only deterministic links, the transceivers and network cables themselves.  So switches measure the ingress and egress time and that gets applied as a correction.

Any network with the possibility of contention is an an adversary.

That is why PTP traffic bypasses packet switching and goes through a seperate pipe allowing a switch which supports being an IEEE 1588 transparent clock, to correct the timestamps for any internal delays. This results in the switch being (almost) invisible where it comes to time transfer (= time synchronisation) between two points no matter the amount of traffic.

[nctnico]

Yes, WR uses SyncE but one of the things you need to account for is phase shift due to the receiver needing to shift (bit slide) a number of UI (link frequency periods) to sync the recovered clock to the data stream. When taking 16 bit words from the link's datastream (recovered clock = link frequency / 20) and each UI beying 800ps, this results in a variable delay between 0ns and 20x800ps = 16ns.

Another route is to go to CERN's OHWR website and use one of the reference designs to implement a WR node with multiple ethernet ports where one port is used to synchronise to the WR switch. The traffic on the other ports can be timestamped using the internal TOD counter + bit slide.

I have considered the idea of implementing a WR node because that would let me eliminate the add-on WR card I currently use and using it to timestamp packets would be a bonus. I just noticed BabyWR (https://ohwr.org/project/babywr) and that looks very interesting for its simplicity compared to a CUTE-WR-A7. However, it is still under development.

You could send the creator of the project an email to see what is possible where it comes to joining forces. I've met the creator a few times and he is a reasonable guy (employed by a research institute to do this kind of work).

[ejeffrey]

Any network with the possibility of contention is an an adversary.

There is no contention on an individual ethernet link, at least in a switched network.  At any given time, either a specific packet is being transmitted or it is not.  Even in a hub, either the packet is transmitted or not, a collision cancels both packets, it doesn't delay them non-deterministically. 

Where there is a finite time-of-flight, partial ordering is the best you will ever be able to achieve.

Depends on what you mean.  There is always a limit to synchronization and resolution.  So there will be events that are so close you can't distinguish which is "actually first".  But you can still assign a total order -- every event has a timestamp, timestamps *trivially* have a total order.  That isn't the problem -- the problem is preserving causal ordering -- no node should be able to receive a message about the future.  But that's conceptually easy, it only requires synchronization to be better than the shortest hop delay.

It will be interesting to see what the standards bodies define - and don't define - w.r.t. defining time on the moon relative to time on the earth. And Mars. And Alpha Centauri.

PTP itself doesn't require a specific reference frame, or even a specific rate.  It's almost universal to use something derived from GPS, which means that the second is relative to TAI, aka the SI referenced to MSL.  But you can run it with your own time reference, for instance if you have your own atomic clock and want to run it with your own local SI second rather than the MSL reference TAI.  Or you can use a different or arbitrary time offset if you want.

The primary goal of PTP is to ensure that every clock in the system is running at the same rate.  It's basically a way to distribute PPS signals without a dedicated network.  The secondary goal is to make sure that their offsets are equal, which requires a description of the reference plane where the time will be measured, typically the PHY.

Jitter is relevant, but isn't the principal conceptual problem with the idea of a single unique time on all interconnected processors.

There is no conceptual problem with having a unique time defined everywhere, and I don't know why you think there is.  Because time is different in different reference frames due to GR and SR, you *do* have to pick a reference frame, but once you do it's perfectly well defined how to synchronize clocks everywhere.

[aeg]

I have to admit that I kind of missed the point in my previous reply. You get path delay correction and residence time correction on PTP packets themselves, but OP was asking about timestamping random UDP packets, presumably not from the PTP peer.

[ejeffrey]

That's true.  You can have an accurate clock and get a precise timestamp of the packet arrival.  Whether that means anything depends on how the packet got there.

[tggzzz]

Where there is a finite time-of-flight, partial ordering is the best you will ever be able to achieve.

Depends on what you mean.  There is always a limit to synchronization and resolution.  So there will be events that are so close you can't distinguish which is "actually first".  But you can still assign a total order -- every event has a timestamp, timestamps *trivially* have a total order.  That isn't the problem -- the problem is preserving causal ordering -- no node should be able to receive a message about the future.  But that's conceptually easy, it only requires synchronization to be better than the shortest hop delay.

So Leslie Lamport got it wrong?

Nowadays even short hops are "long".

It will be interesting to see what the standards bodies define - and don't define - w.r.t. defining time on the moon relative to time on the earth. And Mars. And Alpha Centauri.

PTP itself doesn't require a specific reference frame, or even a specific rate.  It's almost universal to use something derived from GPS, which means that the second is relative to TAI, aka the SI referenced to MSL.  But you can run it with your own time reference, for instance if you have your own atomic clock and want to run it with your own local SI second rather than the MSL reference TAI.  Or you can use a different or arbitrary time offset if you want.

GPS on the moon?  Mars?

What happens to the second in a very different gravity well?

What happens when the path length is continually changing? Obviously that implies the relative velocity can be positive and negative.

There is no conceptual problem with having a unique time defined everywhere, and I don't know why you think there is.  Because time is different in different reference frames due to GR and SR, you *do* have to pick a reference frame, but once you do it's perfectly well defined how to synchronize clocks everywhere.

The concept of a single reference frame for Earth, Mars, and the Moon is not trivial.

[ejeffrey]

So Leslie Lamport got it wrong?

No, you have just misunderstood his work.  The opening lines of the abstract for "Time, Clocks, and the Ordering of Events in a Distributed System" read:

The concept of one event happening before another
in a distributed system is examined, and is shown to
define a partial ordering of the events. A distributed
algorithm is given for synchronizing a system of logical
clocks which can be used to totally order the events.

Once you assign timestamps you have a total order.  As opposed to the partial order, which only declares event A to precede B if there is a message going from A to B (possibly indirectly).  Without the a clock, there is a ambiguous range of "concurrent" events limited by the communication latency, but defining clocks defines an order for the concurrent events.

What happens to the second in a very different gravity well?

Well, you can either define an independent timekeeping reference on a different celestial body or you can define that it should stay in sync with earth standard time.  This is much the way that GPS clocks experience a vastly different gravity and velocity compared to NIST atomic clocks in Boulder which in turn have very different gravitational potential to an observatory in Paris.  Very likely the latter would be chosen, that's how NASA currently does time synchronization for spacecraft.