Troubleshooting an odd issue on a custom SFP+ media adapter

[Multiple Cheese Slices]

Hello everyone!

I'm working on a project with specific use-case. I'll give you some background first, in case it would be of any importance.  (If you don't care about the ramblings, feel free to skip to the technical part below )
I need to convert multi-mode fibers coming out of a device (Let's call it device A) on one end to single-mode for transmission over a long distance, and back to multi-mode again on the other end (Let's call it device B, although in my current test setup it is a Xillinx ZCU102 board with SFP cages), all in a way that would be transparent to the packets/users' data, just as if the two devices were connected together directly.
The Device A has a multi-mode transciever built-in, so I cannot simply swap it for a single-mode one and have a SM fiber running all the way from A to B (except for the current test setup I have right now, the same will be true for device B about the transciever being non-swappable).
The data sent over the fibers will not consist of standard ethernet packets, but rather a custom type of a frame.

I have taken a look around commercial media converters, and they're surprisingly expensive - few hundred to few thousand dollars per channel not counting the transcievers! There's also lots of other digital stuff going on there like network monitoring or some other fancy options, I don't need all that. What I need are a lot of channels for relatively cheap (Let's say, Below 18USD per channel not including the transcievers. I need a few hundred channels, so the costs start to add up quick...).

I have also found this https://mikrotik.com/product/crs326_24s_2q_rm mikrotik 24-port sfp+ network switch which in theory could be used to bridge two ports together in effect making a cheap media converter, however I believe that it will drop all the packets if it fails to recognize them as valid ethernet packets (and again - it's not ethernet, and encapsulation is unfortunately not possible for this specific use case). Of course I have tried contacting mikrotik support about whether this could be configured to work, but unfortunately haven't gotten any helpful answers.

Due to this, I have decided to make my own media converter prototype to prove that it can be done cheaply. One that is as dumb & simple as possible. My idea was to connect two SFP transcievers directly to each other (RX and TX crossed, of course), so that - for example - a multi mode signal would get converted to an electric signal using the MM SFP, which then would enter the SM SFP to be sent further down as a single-mode signal. It is supposed to work the exact same way in the opposite direction, of course.
I am working with 10Gbit SFP transcievers for the media conversion and the ZCU102 (device B).


I test the connectivity of device A and device B using some firmware running on both ends which does some unrelated stuff, but requires a connection between the two devices to function and is constantly sending packets back and forth. I am able to read out the connectivity / whether the packets are being sent from A to B and from B to A by inspecting signals on the ZCU102 board using VIO - a virtual probe monitor. Only catch is, that I first need to be able to send packets from B to A, to then send packets from A to B.


I have designed, ordered and assmebled a PCB implementing my idea  It has a few layout variants (electrically identical), just for good measure.

(I have also later found this https://www.pcbway.com/project/shareproject/SFP_to_SFP_Fiber_Media_Converter_include_TYPE_C_to_TTL_terminal_interface_ef5f247d.html - pretty much the exact same thing which I've built. Although I would be more surprised if nobody thought of this before, actually).



This is the part to skip to just for the technical details and the actual question I have

I have tested this setup with the following SFP Transcievers / in the following ways: (underlined stuff is what is on my PCB converter)

1. I have tried starting with the final go-to setup, MM coming in from A, coming into the first conversion (MM<->SM), then the SM fiber going into the second SFP-SFP conversion, and the outgoing MM hooked up to device B. This did not allow for data transfer, so I have done more experiments.
A <-MM fiber-> ATGBICS AFBR-709SMZ-C <-copper-> Dell FTLX1471D3BCL-FC <-SM Fiber-> Dell FTLX1471D3BCL-FCA <-copper-> TGBICS AFBR-709SMZ-C <-MM Fiber-> B

2. I have tried using only MM SFPs and fibers, but still with my pcb adapter in between. In effect, there is an optical->eletrical->optical "conversion" going on, but we are using the same type of SFP and fiber all the way - this is just to prove that the SFPs can in fact talk to each other directly. Here the packets flow both ways no problem (both A to B and B to A). The idea itself seems to be fine, appears to be some electrical SFP compatibility issue.
A <-MM fiber-> ATGBICS AFBR-709SMZ-C <-copper-> ATGBICS AFBR-709SMZ-C <-MM Fiber-> B

3. Tried the exact same thing as in 2., exact same SFP model but a different manufacturer. This one does NOT work for some reason. Odd.
A <-MM fiber-> Avago AFBR-709DMZ <-copper-> Avago AFBR-709DMZ <-MM Fiber-> B

4. I have tried the setup with just one SM-MM conversion, including a SM SFP in the B (ZCU102)'s sfp cage, of course. This one behaves really weird! It allows me to transmit packets from B to A but not from A to B
A <-MM fiber-> ATGBICS AFBR-709SMZ-C <-copper-> Dell FTLX1471D3BCL-FC <-MM Fiber-> B(also with the Dell FTLX1471D3BCL-FC)

5. Tried the same thing as in 4, but with the Avago. Does not work in any direction. Seems that these Avago SFPs are particularly picky.
A <-MM fiber-> Avago AFBR-709DMZ <-copper-> Dell FTLX1471D3BCL-FC <-MM Fiber-> B(also with the Dell FTLX1471D3BCL-FC)

6. This one works only with B->A, not the other way around. It appears that the Avago (Like the Dell FTLX) has troubles receiving stuff directly from another SFP.
A <-MM fiber-> ATGBICS AFBR-709SMZ-C <-copper-> Avago AFBR-709DMZ <-MM Fiber-> B

7. This one doesn't work at all, presumably because the packets from B aren't getting to A, for A to be able to send stuff back.
A <-MM fiber-> Avago AFBR-709DMZ <-copper-> Avago AFBR-709DMZ <-MM Fiber-> B

The results do not seem to differ based on what SFP is used in the ZCU102 board (as long as it is the correct type (SM or MM), of course) or which of the three layouts is used, so at least that can be ruled out. The behavior is also 100% deterministic - the SFP combinations which work, always work, and the ones which don't - never don't. It doesn't seem to matter whether I use a long 400m roll of fiber to connect the stuff nor a short 1m fiber, so I don't think it's that either. Things could be worse

One thing I have stumbled upon is the common mode or biasing voltage that the differential pairs are apparently supposed to have. I have probed the ZCU102 (which is a commercial device, it works with all the previously mentioned SFPs no problem) with a DC voltmeter, and found that:
Voltage on an inactive (transciever not inserted into the cage) SFP port on the ZCU102, GND to RD+/- is 0.3V
Voltage on an active (some transciever inserted into the cage)  SFP on the ZCU102, GND to RD+/- is 0.8V
Voltage on an inactive SFP on the ZCU102, GND to TD+/- is 1.2V
Voltage on an active SFP on the ZCU102, GND to TD+/- is 0.7V

So I have done a bit more digging. It appears that SFP+ uses a CML (Common Mode Logic) signalling standard (at least accoring to the SFP+ MSA (SFF-8431.pdf)). All my transcievers are SFP+, since they are 10Gbps.


CML appears to need 50 pull-up resistors to 1.2V for both traces of the differential pair near the TX. Most PHYs, including the one built into the Xillinx FPGA and this https://ww1.microchip.com/downloads/aemDocuments/documents/UNG/ProductDocuments/DataSheets/VMDS-10297.pdf#page=34 example PHY, seem to have these built-in.


I have tried adding these on my PCB  , but the behavior during tests did not change at all. That's a shame, I thought this could have been a big lead.


I'm out of ideas what to try next and have no clue why this is working with some models while not with others  . I want to make this design fool-proof - to make it able to work with any SFP+ transciever. Do you think there's anything else worth looking into here? Anything I've missed or done incorrectly? Stuff to try?

Thanks in advance for you advice on this! 

[jm_araujo]

Have you configured the Avago to the correct bit-rate? Scanning the datasheet it seems it supports both 1Gb and 10Gb, user selectable using the Rate Select pins, which some SFPs ignore (like the FTLX1471D3BCL, according to it's datasheet)
From the datasheet:

The rate select can be done through either the hardware
pins or the software access to the A2h page of EEPROM
map. The user can refer to the Appendix for details of
rate select.

[Multiple Cheese Slices]

That is a very good point, I am surprised that I have missed this. I have just assumed that the avago and atgbics would be the same if the model matches, but apparently not.
Right now these pins are floating, I will try pulling them up to 3.3V and see if it works.

When it comes to the SM SFP, I am only able to find a datasheet for the Finisar variant (10G only), not Dell (which is the one I have). As seen by the example of the MM transciever, the available data rate settings can vary depending on the manufacturer. I will try playing with these also on the SM transciever. Maybe the Dell is defaulting to some other data rate too...

This is a really good lead, thank you!

[Multiple Cheese Slices]

Tried the new setup with both RS0 and RS1 being pulled up to 3.3V, both with and without the 50 pullup resistors to 1.2V.

Bad news, this does not seem to affect the tests in any way. I have tried previous setups No. 2, 3, 4 and 7 (with the RS0 & 1 pulled to 3.3V at first through a 10k resistor, and then directly with no resistance in between). No difference in results from my last post.

Odd, I would expect something to change here. Maybe it was an issue, but not the main issue?

[Multiple Cheese Slices]

There's a thing which I think may be worth noting

I don't have the equipment to properly measure the high-speed 10GHz signal directly, best I can access is a 20GSa/s, 1GHz-bandwidth scope (LeCroy/Teledyne Waverunner 8104) & 1GHz differential probe (LeCroy AP034).

Regardless of that, I have done some probing to perhaps get a look at least at how the signals coming out of different SFPs look compared to each other, even if the signal is so distorted that this would not be a good reading overall.

I have found that these waveforms can vary depeonding on what SFP is connected, and whether it is connected to device A or device B (there is a 400m long cable connecting to device B, therefore the optical signal is more attenuated than a short connection coming from device A) this may or may not be the culprit here.

Example measurement 1: This is a signal in the ATGBICS-to-ATGBICS configuration (setup No.2 from my original post)

The upper waveform is around 635mV pk-pk. It is the singal coming out of the TX pair of the ATGBICS AFBR-709SMZ-C connected through a long (400m) MM fiber to the device B (ZCU102 board).
The waveform on the bottom (stored in memory on the scope for a rough reference in next measurements) is a signal coming out of the ATGBICS AFBR-709SMZ-C that is connected through a short fiber (about 40cm) to the device A.

It appears that (what I assume to be) the optical signal strength affects the electrical signal strength on these SFPs. They were supposed to have limiting amplifiers, but I guess there is a limit of what they can do

Example measurement 2: (setup No.7 from my original post)

What makes me worried, is that different SFPs seem to be affected by the attenuation (assuming that the attenuation is the cause of this) very differently. This is a signal that the Avago AFBR-709DMZ spits out based on what is being fed to it through the 400m MM fiber from device B.
The electrical signal on my board is 400mV pk-pk.
(ignore the bottom waveform, it's the one stored in memory from the previous measurement)


Example measurement 3: (setup No.7 from my original post)

This is what the Avago outputs based on the signals it is getting from device A (short 40cm fiber). Sightly better than the previous outcome, but still a little more attenuated compared to what the ATGBICS typically outputs.
around 665mV pk-pk
(Again, we just care about the upper waveform)

I am making my measurements by directly probing the differential pairs which connect the SFP cages together. I have scraped off a bit of the soldermask to get an electrical connection.



I have checked the datasheets of the MM transcievers, and for both of them the input voltage swing can vary from 180mV to 700mV. If I am not mistaken, this means that the input at its least senstive may require 700mV or more differential voltage on the input to register the signal as high. I assume the actual sensitivity/threshold varies based on the manufacturer, model and maybe even some silicon lottery effects to some extent.

This, along with the measurements which in some conditions do not cross the 700mV threshold would seem like one of the potential problems with the connectivity.

The solution coming to mind would be to use some buffer like this (of course this one is too slow, needs to be at least 10GHz), which could accept a wide range of differential input voltages, while always outputting some voltage which all SFPs would accept. https://www.analog.com/media/en/technical-documentation/data-sheets/ADN2890.pdf

Best one I could find which is capable of 10GHz is this differential amplifier. I'm not sure if using it is a good idea, I think a simple buffer would be better here, but I can't find any for 10GHz - not even fanout clock buffers seem to go that high. Or maybe I'm just bad at searching 
https://www.mouser.fr/datasheet/2/609/6409fb-2955701.pdf

What are your thoughts? Do you think using a buffer or an amplifier like this is a good idea?
Do you think these measurements are meaningful at all?

[bson]

I have also found this https://mikrotik.com/product/crs326_24s_2q_rm mikrotik 24-port sfp+ network switch which in theory could be used to bridge two ports together in effect making a cheap media converter, however I believe that it will drop all the packets if it fails to recognize them as valid ethernet packets (and again - it's not ethernet, and encapsulation is unfortunately not possible for this specific use case).

If it's not a valid ethernet frame, then it will likely be ignored by any switch.  It needs the preamble to lock on, MAC addresses to switch, and the CRC to validate. 

But on the whole, even though I don't think a switch will help you, I have the CRS326-24G-2S-RM, which is 24 x RJ gbe and 2 x 10G SFP+, and it works as one would expect.  Their management software is competent, and it can be operated as either a smart switch (dumb switch out of the box) or a basic router.  I don't think it can mix or partition the ports (e.g. switch some, route others).  I set it up with just a browser aimed at it directly.  Great for the price.

[Multiple Cheese Slices]

Quick update:

Apparently the key-word to search with when buffering high-speed signals are redrivers.
I have found this thing. It is intended for USB and such, but looking at the voltage levels, I think it should be fine for the SFP+ use case. https://www.ti.com/lit/ds/symlink/tusb1002a.pdf
 I plan on ordering a few and seeing if it fixes the issue.

Do let me know whether you think think could or could not work, your thoughts are appreciated! 

[Multiple Cheese Slices]

I got good news!

It's a little embarrassing, but the source of the issue appears to have been a very simple, stupid thing. 
I have been pulling the Tdis pins on the SFP transcievers down to GND using a 10k resistor. Apparently it is too weak to actually fully enable the SFP all the time. It appears that due to manufacturing differences, some SFPs did not fully enable their transmitters - which most likely was the actual cause of the weak signal (some SFPs include internal pull-up resistors stronger than the 10k pulldown on my board). This pin seems to be more analog, than I have assumed.

I have connected all Tdis pins directly to GND, The setup No.1 now works fine.
A <-MM fiber-> ATGBICS AFBR-709SMZ-C <-copper-> Dell FTLX1471D3BCL-FC <-SM Fiber-> Dell FTLX1471D3BCL-FCA <-copper-> ATGBICS AFBR-709SMZ-C <-MM Fiber-> B

What is funny, is that
A <-MM fiber-> Avago AFBR-709DMZ <-copper-> Dell FTLX1471D3BCL-FC <-SM Fiber-> Dell FTLX1471D3BCL-FCA <-copper-> Avago AFBR-709DMZ <-MM Fiber-> B
still does not function....

However, replacing either one of the two Avagos with an ATGBICS makes the system function as expected.
A <-MM fiber-> ATGBICS AFBR-709SMZ-C <-copper-> Dell FTLX1471D3BCL-FC <-SM Fiber-> Dell FTLX1471D3BCL-FCA <-copper-> ATGBICS AFBR-709DMZ <-MM Fiber-> B

A <-MM fiber-> Avago AFBR-709DMZ <-copper-> Dell FTLX1471D3BCL-FC <-SM Fiber-> Dell FTLX1471D3BCL-FCA <-copper-> ATGBICS AFBR-709SMZ-C <-MM Fiber-> B
both work fine.

The Setup No. 7
A <-MM fiber-> Avago AFBR-709DMZ <-copper-> Avago AFBR-709DMZ <-MM Fiber-> B does also function just fine.


Based on the above tests, I assume that Avagos still have a problem with signal strength, so placing in redrivers still might be a good idea - especially if the final version of this converter would be expected to function in an actual "real" scenario. I guess all the troubleshooting done now could have just saved a lot of headache later down the line.