Hello everyone!
I'm working on a project with specific use-case. I'll give you some background first, in case it would be of any importance. (If you don't care about the ramblings, feel free to skip to the technical part below
)
I need to convert multi-mode fibers coming out of a device (Let's call it device A) on one end to single-mode for transmission over a long distance, and back to multi-mode again on the other end (Let's call it device B, although in my current test setup it is a Xillinx ZCU102 board with SFP cages), all in a way that would be transparent to the packets/users' data, just as if the two devices were connected together directly.
The Device A has a multi-mode transciever built-in, so I cannot simply swap it for a single-mode one and have a SM fiber running all the way from A to B (except for the current test setup I have right now, the same will be true for device B about the transciever being non-swappable).
The data sent over the fibers will not consist of standard ethernet packets, but rather a custom type of a frame.
I have taken a look around commercial media converters, and they're surprisingly expensive - few hundred to few thousand dollars per channel
not counting the transcievers! There's also lots of other digital stuff going on there like network monitoring or some other fancy options, I don't need all that. What I need are a lot of channels for relatively cheap (Let's say, Below 18USD per channel not including the transcievers. I need a few hundred channels, so the costs start to add up quick...).
I have also found
this https://mikrotik.com/product/crs326_24s_2q_rm mikrotik 24-port sfp+ network switch which in theory could be used to bridge two ports together in effect making a cheap media converter, however I believe that it will drop all the packets if it fails to recognize them as valid ethernet packets (and again - it's not ethernet, and encapsulation is unfortunately not possible for this specific use case). Of course I have tried contacting mikrotik support about whether this could be configured to work, but unfortunately haven't gotten any helpful answers.
Due to this, I have decided to make my own media converter prototype to prove that it can be done cheaply. One that is as dumb & simple as possible. My idea was to connect two SFP transcievers directly to each other (RX and TX crossed, of course), so that - for example - a multi mode signal would get converted to an electric signal using the MM SFP, which then would enter the SM SFP to be sent further down as a single-mode signal. It is supposed to work the exact same way in the opposite direction, of course.
I am working with 10Gbit SFP transcievers for the media conversion and the ZCU102 (device B).
I test the connectivity of device A and device B using some firmware running on both ends which does some unrelated stuff, but requires a connection between the two devices to function and is constantly sending packets back and forth. I am able to read out the connectivity / whether the packets are being sent from A to B and from B to A by inspecting signals on the ZCU102 board using VIO - a virtual probe monitor. Only catch is, that I first need to be able to send packets from B to A, to then send packets from A to B.
I have designed, ordered and assmebled a PCB implementing my idea It has a few layout variants (electrically identical), just for good measure.
(I have also later found
this https://www.pcbway.com/project/shareproject/SFP_to_SFP_Fiber_Media_Converter_include_TYPE_C_to_TTL_terminal_interface_ef5f247d.html - pretty much the exact same thing which I've built. Although I would be more surprised if nobody thought of this before, actually).
This is the part to skip to just for the technical details and the actual question I haveI have tested this setup with the following SFP Transcievers / in the following ways:
(underlined stuff is what is on my PCB converter)1. I have tried starting with the final go-to setup, MM coming in from A, coming into the first conversion (MM<->SM), then the SM fiber going into the second SFP-SFP conversion, and the outgoing MM hooked up to device B. This did not allow for data transfer, so I have done more experiments.
A <-MM fiber->
ATGBICS AFBR-709SMZ-C <-copper-> Dell FTLX1471D3BCL-FC <-SM Fiber->
Dell FTLX1471D3BCL-FCA <-copper-> TGBICS AFBR-709SMZ-C <-MM Fiber-> B
2. I have tried using only MM SFPs and fibers, but still with my pcb adapter in between. In effect, there is an optical->eletrical->optical "conversion" going on, but we are using the same type of SFP and fiber all the way - this is just to prove that the SFPs can in fact talk to each other directly. Here the packets flow both ways no problem (both A to B and B to A). The idea itself seems to be fine, appears to be some electrical SFP compatibility issue.
A <-MM fiber->
ATGBICS AFBR-709SMZ-C <-copper-> ATGBICS AFBR-709SMZ-C <-MM Fiber-> B
3. Tried the exact same thing as in 2., exact same SFP model but a different manufacturer. This one does
NOT work for some reason. Odd.
A <-MM fiber->
Avago AFBR-709DMZ <-copper-> Avago AFBR-709DMZ <-MM Fiber-> B
4. I have tried the setup with just one SM-MM conversion, including a SM SFP in the B (ZCU102)'s sfp cage, of course. This one behaves really weird! It allows me to transmit packets from B to A but not from A to B
A <-MM fiber->
ATGBICS AFBR-709SMZ-C <-copper-> Dell FTLX1471D3BCL-FC <-MM Fiber-> B(also with the Dell FTLX1471D3BCL-FC)
5. Tried the same thing as in 4, but with the Avago. Does not work in any direction. Seems that these Avago SFPs are particularly picky.
A <-MM fiber->
Avago AFBR-709DMZ <-copper-> Dell FTLX1471D3BCL-FC <-MM Fiber-> B(also with the Dell FTLX1471D3BCL-FC)
6. This one works only with B->A, not the other way around. It appears that the Avago (Like the Dell FTLX) has troubles receiving stuff directly from another SFP.
A <-MM fiber->
ATGBICS AFBR-709SMZ-C <-copper-> Avago AFBR-709DMZ <-MM Fiber-> B
7. This one doesn't work at all, presumably because the packets from B aren't getting to A, for A to be able to send stuff back.
A <-MM fiber->
Avago AFBR-709DMZ <-copper-> Avago AFBR-709DMZ <-MM Fiber-> B
The results do not seem to differ based on what SFP is used in the ZCU102 board (as long as it is the correct type (SM or MM), of course) or which of the three layouts is used, so at least that can be ruled out. The behavior is also 100% deterministic - the SFP combinations which work, always work, and the ones which don't - never don't. It doesn't seem to matter whether I use a long 400m roll of fiber to connect the stuff nor a short 1m fiber, so I don't think it's that either. Things could be worse
One thing I have stumbled upon is the common mode or biasing voltage that the differential pairs are apparently supposed to have. I have probed the ZCU102 (which is a commercial device, it works with all the previously mentioned SFPs no problem) with a DC voltmeter, and found that:
Voltage on an inactive (transciever not inserted into the cage) SFP port on the ZCU102, GND to RD+/- is 0.3V
Voltage on an active (some transciever inserted into the cage) SFP on the ZCU102, GND to RD+/- is 0.8V
Voltage on an inactive SFP on the ZCU102, GND to TD+/- is 1.2V
Voltage on an active SFP on the ZCU102, GND to TD+/- is 0.7V
So I have done a bit more digging. It appears that SFP+ uses a CML (Common Mode Logic) signalling standard (at least accoring to the SFP+ MSA (SFF-8431.pdf)). All my transcievers are SFP+, since they are 10Gbps.
CML appears to need 50
pull-up resistors to 1.2V for both traces of the differential pair near the TX. Most PHYs, including the one built into the Xillinx FPGA and
this https://ww1.microchip.com/downloads/aemDocuments/documents/UNG/ProductDocuments/DataSheets/VMDS-10297.pdf#page=34 example PHY, seem to have these built-in.
I have tried adding these on my PCB
, but the behavior during tests did not change at all. That's a shame, I thought this could have been a big lead.
I'm out of ideas what to try next and have no clue why this is working with some models while not with others
. I want to make this design fool-proof - to make it able to work with any SFP+ transciever. Do you think there's anything else worth looking into here? Anything I've missed or done incorrectly? Stuff to try?
Thanks in advance for you advice on this!