Need some advice on transferring data from LVDS to ethernet, around 60 mbit/s

[rstofer]

Spurred on by this thread, I've picked up an FPGA board for the first time in ages and started working on some code to talk directly to a MII interface. The board I'm using is a  is a Digilent Arty, (Artix-7 with a  TI DP83848J 10/100 Ethernet PHY on it). The design itself will be tiny - just a block ram holding the data packet, the packet CRC calculator, and the logic to send the nibbles to the PHY.

The H/W should easily be able to hit about 90% of the link bandwidth (90Mb/s) - well over what the OP needed.

After reading Datasheets, and tinkering last night I'm just about bring the PHY out of reset and then start sending UDP packets. I'll let you know how I get on over the weekend -I  don't really have any plans but to send a packet every 40ms or so, as I don't want to hose the house's wireless (as it will be cabled to the my ADSL router).

I'd be very interested in your results (and your code if it's available).  I looked at the LVDS input problem and it could be a challenge.  The first issue is whether or not one of the signals is a clock.  Without a clock, the stream is asynchronous and sampling can be done using some tricky embedded hardware and a clock generator.  The thing is, both approaches I looked at use IDELAY elements and the maximum tap is only out around 3 nS.  This approach works for Gb inputs but not low Mb.  There's a lot of information at Xilinx about how to grab these LVDS signals but, again, most of it seem to be for much faster signals.  More study needs to be done.

With a clock in the datastream, things might get a little easier but we still have the problem of crossing into the FPGA clock domain.  Usually this is just 2 or 3 D-flops in series so it might not be too bad.

[hamster_nz]

I'd be very interested in your results (and your code if it's available).  I looked at the LVDS input problem and it could be a challenge.  The first issue is whether or not one of the signals is a clock.  Without a clock, the stream is asynchronous and sampling can be done using some tricky embedded hardware and a clock generator.  The thing is, both approaches I looked at use IDELAY elements and the maximum tap is only out around 3 nS.  This approach works for Gb inputs but not low Mb.  There's a lot of information at Xilinx about how to grab these LVDS signals but, again, most of it seem to be for much faster signals.  More study needs to be done.

With a clock in the datastream, things might get a little easier but we still have the problem of crossing into the FPGA clock domain.  Usually this is just 2 or 3 D-flops in series so it might not be too bad.

Once working I'll slap it on my Wiki.

The input side shouldn't be too hard, even without a clock. If there was a clock to follow I would oversample with a serdes block at something close to 330Mb/s, and then subsample every five or six bits, You can then nudge the sample frequency to keep transitions away from where you are sampling (eg ...5,6,5,6,5,6,5,6,5,6... if the source clock perfectly matches reference, or ...5,6,5,5,6,5,6,5... if the source clock is running a little faster.

[hamster_nz]

Well that wasn't hard - sending hand crafted packets through the DP83848J PHY at 100Mb/s on the Digilent Arty FPGA board, using the MII.

http://hamsterworks.co.nz/mediawiki/index.php/ArtyEthernet

I'll keep correcting the data packet (mainly add checksum) over the next few days.

Resource usage is 22 slices and half a BRAM block

[Rasz]

Unrelated to my current work project, I intend to buy soon enough some FPGA board. Currently, I already own a papilio pro, papilio pro duo, minispartan+. Although I never had really the time to touch them....

and this is your main problem, not the lvds, speed or implementing network layers
Lets face it - you are lazy. You think buying more expensive boards will somehow impart knowledge on you without doing any actual work.
hamster_nz already demonstrated this is 1-2 evenings project, but you rather procrastinate while thinking about $500 dev kits

[asgard20032]

Unrelated to my current work project, I intend to buy soon enough some FPGA board. Currently, I already own a papilio pro, papilio pro duo, minispartan+. Although I never had really the time to touch them....

and this is your main problem, not the lvds, speed or implementing network layers
Lets face it - you are lazy. You think buying more expensive boards will somehow impart knowledge on you without doing any actual work.
hamster_nz already demonstrated this is 1-2 evenings project, but you rather procrastinate while thinking about $500 dev kits

Currently I have no development board at my home with which i can try to do ethernet development. And an internet phy is not something i can easily add to a papilio board. It require good pcb, short trace, length matched. I am currently trying to implement it on my fpga at job. But contrary to hamster, my interface is rgmii, not mii, so it send data on both edge of the clock. This is a bit more complicated. For someone with experience with fpga, like hamster, it is also easy. It is my first real FPGA project.

[asgard20032]

Well that wasn't hard - sending hand crafted packets through the DP83848J PHY at 100Mb/s on the Digilent Arty FPGA board, using the MII.

http://hamsterworks.co.nz/mediawiki/index.php/ArtyEthernet

I'll keep correcting the data packet (mainly add checksum) over the next few days.

Resource usage is 22 slices and half a BRAM block

I am currently in the process of writing my own packet. I am a little bit confuse by your example.

Code: [Select]

-- Ethernet source  DE:AD:BE:ED:01:23
              when x"001C" => data <= "1110"; -- E
              when x"001D" => data <= "1101"; -- D
              when x"001E" => data <= "1101"; -- D
              when x"001F" => data <= "1010"; -- A
              when x"0020" => data <= "1110"; -- E
              when x"0021" => data <= "1011"; -- B
              when x"0022" => data <= "1111"; -- F
              when x"0023" => data <= "1110"; -- E
              when x"0024" => data <= "0001"; -- 1
              when x"0025" => data <= "0000"; -- 0
              when x"0026" => data <= "0011"; -- 3
              when x"0027" => data <= "0010"; -- 2
It look like you are sending the byte in the correct order, but with nibble inverted, why?

[Rasz]

Currently I have no development board at my home with which i can try to do ethernet development.

you already said you have three never touched ones, all are ok for this

And an internet phy is not something i can easily add to a papilio board. It require good pcb, short trace, length matched.

no they dont, I did phy on a breadboard with 5cm jumper wires

I am currently trying to implement it on my fpga at job. But contrary to hamster, my interface is rgmii, not mii, so it send data on both edge of the clock. This is a bit more complicated. For someone with experience with fpga, like hamster, it is also easy. It is my first real FPGA project.

is this really your excuse? $15 piece of hardware that can be fedexed overnight is stopping whole project?
http://www.waveshare.com/DP83848-Ethernet-Board.htm
http://www.oddwires.com/ethernet-phy-module-dp83848

I know Im an asshat, but you need someone to slap you across the face with a dead fish. In the words of our great prophet:

[hamster_nz]

Well that wasn't hard - sending hand crafted packets through the DP83848J PHY at 100Mb/s on the Digilent Arty FPGA board, using the MII.

http://hamsterworks.co.nz/mediawiki/index.php/ArtyEthernet

I'll keep correcting the data packet (mainly add checksum) over the next few days.

Resource usage is 22 slices and half a BRAM block

I am currently in the process of writing my own packet. I am a little bit confuse by your example.

Code: [Select]

-- Ethernet source  DE:AD:BE:ED:01:23
              when x"001C" => data <= "1110"; -- E
              when x"001D" => data <= "1101"; -- D
              when x"001E" => data <= "1101"; -- D
              when x"001F" => data <= "1010"; -- A
              when x"0020" => data <= "1110"; -- E
              when x"0021" => data <= "1011"; -- B
              when x"0022" => data <= "1111"; -- F
              when x"0023" => data <= "1110"; -- E
              when x"0024" => data <= "0001"; -- 1
              when x"0025" => data <= "0000"; -- 0
              when x"0026" => data <= "0011"; -- 3
              when x"0027" => data <= "0010"; -- 2
It look like you are sending the byte in the correct order, but with nibble inverted, why?

Ignore Rasz, he got out on the wrong side of bed

It is done that way because it works

Hardware engineers tend to design as if things are very long numbers, and software engineers think as though it is a series of bytes.

Just remember low nibble has to go to the wire first, then the high nibble.

I had a lot of stuff backwards when I got my first packet out on the wire!

[asgard20032]

For the interface, i will use a DDIO. with a dual buffer that will hold the current lower nibble and upper nibble. On rising edge, i will update my upper nibble and send my lower nibble, and on falling edge, i will update my lower nibble and send the upper nibble. For the clock, i will use a pll to shift it a little bit. 2 IP core : PLL and DDIO. I just finished doing the test packet (although, just a simple test like you, not the full application with LVDS). But now, the compiler don't like my const array. It say i got error at the ' symbole. : reg [some:size]someRegisterName[some:size] = '{lot of nibble}

So now its more the verilog syntax. I use this project to also learn verilog. The artyx board you are using is pretty nice by the way. Most modern terasic board either pack lot of feature, but with everything interesting (SD card, ethernet...) connected to the  HPS (DE1-soc, DE0-nano-SoC), or are very minimalist(DE0-nan0) or costly (DE2-115, got ethernet and all other feature). Their DE1 was perfect, it got everything connected to its SoC (altought lacking ethernet, but got everything else), but its outdated, won't work with newest quartus. If only they could make an updated DE1 non-SoC, like they did with the DE2 (DE2-115). I got a nice book about SoC with FPGA (not arm SoC, but softcore SoC with nios II). The book was made for DE1. If i get the DE1-SoC, i could follow the whole book except for sram part, the green led part, and the sd card part(since its connected to the ARM). But for the sd card and the 8 lacking green led, i will (if i get the board) make a pcb (and use that occasion to learn kicad) with micro sd and green led, and even some other nicy (maybe an esp8266 and a bluetooth). But sram is not that easy to add on another pcb. Not trough normal header, too much capacitance, length, noise... At least, i could do 90% of my book.

But on the digilent side, altought their board got less stuff, they got two affordable recent technology board with ethernet NOT connected to a processor (ARM). Like the artyx you are using, or the nexys 4(with academic discount). But getting an altera board with ethernet would be more nice since I could do a full nios II learning path.

Unfortunately, I can get an academic discount, but shipping it here, import fee and ... will cost more than what I save with my academic discount. I wonder if I could get my academic discount trough local distributor like digikey or mouser.

Any suggestion of nice peripheral i could add to a DE1-SoC to enhance my overall experience and learning?

Now I am pretty confident of the success of my project.

[rstofer]

Well that wasn't hard - sending hand crafted packets through the DP83848J PHY at 100Mb/s on the Digilent Arty FPGA board, using the MII.

http://hamsterworks.co.nz/mediawiki/index.php/ArtyEthernet

I'll keep correcting the data packet (mainly add checksum) over the next few days.

Resource usage is 22 slices and half a BRAM block

Outstanding work!  I downloaded your code and I'll get on to it Monday.  I have a lot to learn and your code will help!

Now all I need are two TCP server sockets and one TCP client socket and I'm good to go with my project.  I'll have to think about how to get there.  Maybe a small CPU with a minimal stack like uIP or maybe lwIP.  Maybe the ZPU project core will work.  I already have it running on a Spartan 3 and it has the advantage of having ZPU-GCC.

[hamster_nz]

Unrelated to my current work project, I intend to buy soon enough some FPGA board. Currently, I already own a papilio pro, papilio pro duo, minispartan+. Although I never had really the time to touch them....

Oh,don't feel too bad about the idle boards... until you get a reason to pull them out they always sit idle. You are most probably busy playing with other things... I've got a Parallella, a Raspberry Pi Zero, a Raspberry Pi B 3, an original Raspberry Pi B, and a Pine64 all of which have only been booted a few times, and a BeagleBone Black that has never been out of it's ESD bag - so I fully understand

It looks like I'm just throwing a few bits at the PHY to make it work, but there is a lot more to it than that - knowing how Ethernet addressing works, Ethernet broadcasts, IP headers, IP broadcasts, UDP ports, header checksums, (must add the CRCs in!). Luckily I've done most of it before, so all I really have to worry about is getting the PHY working. If you were coming at it cold it would be a month of spare time researching and reading standards, and playing around with code.

Feel free to ask any questions - I'll point you at Wikipedia or the original documentation where I found things out if I can.

[chris_leyson]

It require good pcb, short trace, length matched

I wouldn't worry too much about matching trace lengths. For a start, the twisted pairs in CAT5e cable have slightly different lengths because the pairs are twisted at different pitches to reduce crosstalk. I might get around to measuring this with a TDR one day.
Also, the symbol rate for 1000base-T is 125 megabaud, so a 10mm difference between two traces in a differential pair corresponds to 1.5 degree phase error, that's in free space so it's a ballpark estimate. I wouldn't worry to much about trace lengths.

[nctnico]

Please use an array with numbers instead of having a seperate line of code for each line. Please use something like this:
output <= array_var(index_counter) ;

If index_counter is a unsigned number type then you can use it as an index directly.

[hamster_nz]

Please use an array with numbers instead of having a seperate line of code for each line. Please use something like this:
output <= array_var(index_counter) ;

If index_counter is a unsigned number type then you can use it as an index directly.

You are more than welcome to take the code and do whatever you like with it - convert it to an array, read it in from a file, type in binary or tap it out in morse code - whatever. You can even send it back to me and I'll post it up too.

However for my hacking I wanted to count individual bytes and to do math on the offsets and play with it. I also wanted to be able to comment every nibble for those who might decide to use it as a reference for their own implementation. I made a conscious decision to do it that way as it allows others to simple pass data into the module and merge it into the packet  (super inefficient I know, but it will still be smaller than a Microblaze).

In short, I'm not got to rework it just for you, but you can :-)

(Oh, and I am pretty sure you need to use "output <= array_var(to_integer(index_counter));" if using IEEE.NUMERIC_STD.ALL )

[hamster_nz]

Pretty much finished the design (http://hamsterworks.co.nz/mediawiki/index.php/ArtyEthernet) - added a module to perform then CRC32 calculation for frames and another  to add the frame's preamble. There is now a simple way to overwrite what is currently empty nibbles with whatever data you want to send, and a signal that can be asserted to trigger sending of a new frame.

Over 100baseT, with small packets (UDP with 16 bytes of user data) it sends around 130,000 packets per second - one every 7us - for about 2MB/s of user data. This rate of packets is enough to make WireShark and my laptop groan, and it seems to drop about 1 in 5,000 packets due to the interface being flooded or other system activity.

If it was changed to send 1,518-byte frames (the biggest allowed with standard Ethernet) the protocol overhead goes down, allowing it to send 12,026,092 bytes of user data per second, 50% more than what the original poster wanted.

[rstofer]

Pretty much finished the design (http://hamsterworks.co.nz/mediawiki/index.php/ArtyEthernet) - added a module to perform then CRC32 calculation for frames and another  to add the frame's preamble. There is now a simple way to overwrite what is currently empty nibbles with whatever data you want to send, and a signal that can be asserted to trigger sending of a new frame.

Over 100baseT, with small packets (UDP with 16 bytes of user data) it sends around 130,000 packets per second - one every 7us - for about 2MB/s of user data. This rate of packets is enough to make WireShark and my laptop groan, and it seems to drop about 1 in 5,000 packets due to the interface being flooded or other system activity.

If it was changed to send 1,518-byte frames (the biggest allowed with standard Ethernet) the protocol overhead goes down, allowing it to send 12,026,092 bytes of user data per second, 50% more than what the original poster wanted.

I look forward to trying it later today.  I had a couple of issues with the original, probably related to Vivado 2016.1.  Or, more likely, just me...

[hamster_nz]

I've added a sequence number to the packets, and been capturing in groups of 65,537, and when sending at full speed sometimes there are blocks of missing packets. Around 800 packets go missing at a time, along with a jump in the time-stamps against them. It is missing about 5 or 6 ms of data in bursts.

So I tried a few different options:

6,250 packets per second - 0 missing, 0 missing, 0 missing, 0 missing, 0 missing
24,700 packets per second - 0 missing, 0 missing, 0 missing, 0 missing, 0 missing
50,000 packets per second - 0 missing, 0 missing, 0 missing, 0 missing, 0 missing
100,000 packets per second - Dropped packets, 0 missing, 0 missing, 0 missing
150,000 packets per second - 11420 missing, 4472 missing, 3306 missing, Dropped packets, 0 missing.

(dropped packets were when Wireshark reported packets as 'dropped', based on NIC statistics. Missing is when expected packets were not in the capture).

So it looks like my laptop's onboard NIC is only good for up to 100,000 packets per second or so before it can't keep up.

[Rasz]

(dropped packets were when Wireshark reported packets as 'dropped', based on NIC statistics. Missing is when expected packets were not in the capture).

So it looks like my laptop's onboard NIC is only good for up to 100,000 packets per second or so before it can't keep up.

Dont know about windows, but linux struggled for a long time when it comes to packet capture - every single packet generated new syscall switching cpu context and compared it individually against filter settings = huge overhead. Maybe, just maybe its because of this. You can try netcat, dump all traffic to a file and count manually (or write python/whatever parser)

nc -u -l 4096 > dump.bin

[hamster_nz]

Yeah, I know I'm flogging a dead horse... but...

I've got a Gigabit PHY on a Nexys Video spitting bits - 978,000,000 of them per second. I'll make a new GitHub repo for it tomorrow at https://github.com/hamsternz?tab=repositories and upload the code.

[rstofer]

Yeah, I know I'm flogging a dead horse... but...

I've got a Gigabit PHY on a Nexys Video spitting bits - 978,000,000 of them per second. I'll make a new GitHub repo for it tomorrow at https://github.com/hamsternz?tab=repositories and upload the code.

I had been wondering if a 100 Mb network was going to work.  It doesn't for me, I only get about 70,000 small packets per second or about 33 Mb throughput.  I didn't chase down the dropped packets.  This with the older version and 58 byte packets.  It's odd that this happens, the PC is fairly capable and has two RealTek GBe PHYs.

Clearly, expanding the packet helps a lot but GBe really shines.  You're about 16 times the requirement falling to 5 times when something happens.  Pretty much the same problem with the 100 Mb PHY.  Somewhere, on the recceiving end, things slow down for a short while.

It is certainly possible to keep up with 60 Mb on the wire.  I'm not entirely certain that the receiver will be able to process the packets as fast as they arrive but that's not the problem of the sender.

When I eventually get my Zybo board, I want to revisit this.

[stmdude]

Guys, I'm out of my league here when it comes to the FPGA stuff, but I'm quite knowledgeable when it comes to networking equipment.

When you're testing this, make sure you only have a cable between your board and PC. Network switches do wierd and wonderful things to your packets, depending on how they're built.

First off, all switches (and L3 switches (aka, routers)) have two upper limits.  The first one is the throughput of the backplane. A good enterprise-grade switch has enough backplane capacity to fill all ports with both TX and RX traffic. I.e, an 24port GigE switch should have at last a 24*2Gbps backplane.
Most decent switches today have this, unless it's made of plastic, and bought because it was the cheapest.
However, the second upper limit varies _a lot_ between different manufacturers and models, and it's the PPS throughput (packets per second). This limit can be remarkably low. I've seen enterprise-grade GigE switches from very reputable vendors have PPS rates of <3Mpps.
If you exceed the PPS, the switch will drop your packets.
Essentially, the PPS is set by how quickly the switch can look up an entry in its table of "which MAC-address is on which port".

Also, RealTek NICs aren't exactly the bees knees. I'm surprised you get as few dropped (as in, dropped by the NIC) packets as you do, when sending that many packets (not bits, packets).

[rstofer]

There is something odd going on and, at the moment, I don't have the tools/skills to figure it out.  Why should my PC only grab 33 Mbps of UDP packets which are summarily dropped in the stack for lack of a client to eat them when I can handle 300 Mbps of routed TCP packets from Xfinity's speed test site 100 miles away?  FWIW, I just watched the speed test with Wireshark and I see a little over 300 Mbps sustained.  They are, of course, maximum length packets.

I did run the cable directly between the FPGA board and the PC.  I would have rather used a hub and had Wireshark running on a 3rd device but I'm not even sure if they make 100 Mbps hubs.  I just searched, they do make them but they are pricey.

The second incantation did result in packets whereas the first did not - for me...  But I still can't get the packets into Wireshark on Debian.  I wanted to see if there was a difference in performance.  The latest version, with large packets, will be more interesting but I don't have an FPGA board with a GBe PHY - yet.

I'm reasonably convinced that a sustained 60 Mbps can not be achieved on a 100 Mbps network.  It's pretty clear that it can be done on a 1000 Mbps network but we still haven't seen the application that will have to process that stream.  I don't know enough about the application to determine whether UDP (and dropped packets) will be acceptable or whether TCP will be required.  I would expect TCP to add additional overhead with ACK packets and other protocol requirements.  TCP also requires that the FPGA honor all the requirements and that implies that the PHY/MAC will have to receive data as well.  Blasting bits out of the FPGA is interesting (I didn't know how to deal with the PHY) but receiving bits will be a lot more challenging.

[asgard20032]

For the rgmii interface, I need to send a logical derivative of tx_en and tx_err on the tx_ctl line on the txglk falling edge.

What is a logical derivative?

[stmdude]

@rstofer:  If you want to play around with protocols and packet-sizes, and have access to two PCs, you should have a look at iperf ( https://iperf.fr/ ). It'll allow you to generate as much traffic as you'd like, and receive it as well if you'd like. It'll show you where the bottlenecks of your particular system is, if it's Mbps or PPS.

60Mbps on 100Base is definitely possible. It's not even in the realm of difficult if you're using the maximum MTU.

If you want to have a nice protocol for high-bandwidth streaming to look at, check out UDT ( http://udt.sourceforge.net/ ). People are doing crazy things with it, like streaming infiniband over Ethernet. It has no issues with saturating 10GigE links. People have done FPGA implementations of it, so it seems applicable to your project.

[rstofer]

@rstofer:  If you want to play around with protocols and packet-sizes, and have access to two PCs, you should have a look at iperf ( https://iperf.fr/ ). It'll allow you to generate as much traffic as you'd like, and receive it as well if you'd like. It'll show you where the bottlenecks of your particular system is, if it's Mbps or PPS.

I'll take a look.  I really only have modest interest in this 60 Mbps problem.  I need a full TCP/IP stack to support a couple of server sockets and a client socket.  But I will need the entire TCP/IP protocol and I would very much like to have DHCP.  I think the Zybo board with the dual core ARM will be the way to handle the stack and the FPGA fabric can implement my CPU project which interacts with the sockets.  I have been waiting for an interesting FPGA board to come along that could handle the soft CPU and the required networking.  I think the Zybo may be that board.

60Mbps on 100Base is definitely possible. It's not even in the realm of difficult if you're using the maximum MTU.

Yes, you would have to use maximum, or nearly maximum, packets but I was thinking end-to-end up through the application layer.  It would be even more difficult if TCP is required.  Maybe it's time for another test.  There's some reason I can't get my UDP rate over 32Mbps.  At 70,000 pps with 58 byte packets, it would seem that all I need to do is increase the packet size.  Yes the pps will drop but probably not as fast as the bps increases.  More testing to do...

If you want to have a nice protocol for high-bandwidth streaming to look at, check out UDT ( http://udt.sourceforge.net/ ). People are doing crazy things with it, like streaming infiniband over Ethernet. It has no issues with saturating 10GigE links. People have done FPGA implementations of it, so it seems applicable to your project.

FPGAs are amazing devices.