Ethernet (speed) auto-negotiation

[legacy]

so, I have a couple of days to play with our (my crew of crazy friends) Sonoko project, which has different boards running Linux, and the road-map is still ... sit, relax, and ... yes, it's all still a bloody mess 

Anyway, one of the weird problems we have found is related to the Ethernet auto-negotiation, which is still a mystery.

Each node does bootstrap from the network via tftp, so it's very important that each node uses the maximal speed possible because the kernel image is of 8Mbyte and the ramrootfs is of 64Mbyte.

A lot of Mbyte to be uploaded

Each graphic node has a PowerPC405GP embedded CPU with an MII channel to the Ethernet PHY, which should be responsible for both speed { 10Mbps, 100Mbps} and {half, full} duplex auto-negotiation, while the router-node has three fast-ethernet ports configured to autonegotiate the maximal speed possible, but ... when you look at the configuration reported by U-boot on each of the graphic nodes, it randomly looks like "a deranged dog on speed" (  ).

Here it is an example

Code: [Select]

# tftpboot 800000 kernel
ENET Speed is 100 Mbps - FULL duplex connection (EMAC0)
Using ppc_4xx_eth0 device
u-boot says 100Mbps full duplex!

Does it lie? Well, the Linux router says this

Code: [Select]

# ethtool eth1

Settings for eth1:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 10Mb/s
        Duplex: Half
        Port: MII
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: pumbg
        Wake-on: d
        Current message level: 0x00000000 (0)
        Link detected: no
10Mbps half duplex

100Mbps full duplex != 10Mbps half duplex .... do have to trust the router because a second laptop has just confirmed that the graphic node is *somehow* failing the auto-negotiate, so it has forced the link to be 10Mbps half-duplex.

In fact, the Ethernet PHY is a DP83843 chip which exposes some pins for Auto-Negotiation and other pins are used to indicate Duplex, Link, Receive and Transmit, Link, Collision, and Speed status. By reading their logic status ... yet again confirmed what the router says: 10Mbps half-duplex 

So u-boot does lie!

One of the question is: why? why does it fail? It's not clear how and why the auto-negotiate protocol does work at this point.

-

Perhaps, I cannot even exclude a bug in u-boot, but in my mind, this is less convincing.

-

Anyway, I do not even know how to proceed, and where I have to investigate.

[Daixiwen]

It is possible that the link starts at 100 Mbits/s in full duplex but drops to 10 Mbits/s and half duplex because of a poor connection.
If you use a computer instead, connected to the same router port with the same network cable, which speed do you get?
It could also be a bad soldering problem on the network connector or the transformer on the board.

[edpalmer42]

Check your cables.  Are they home made?  Are they actual Cat5 or better or are they some unlabelled cable that happens to fit.  I've seen people try to use 8-wire flat telephone cables.  Not going to work!! 

One odd thing I saw was with old 3Com 10/100 cards plugged into a Netgear 5 port Gigabit switch.  If you used 1 meter cables, auto-negotiate would fail.  Use 10 meter cables and everything worked perfectly! 

By the way, if it's a private setup that's not going to change, you can always force the ports on one side and/or the other to 100 Mbps.  I'm not sure if you have to do both sides or not.

[legacy]

It is possible that the link starts at 100 Mbits/s in full duplex but drops to 10 Mbits/s and half duplex because of a poor connection.

Code: [Select]

graphic node0 === link2 === router node0 === link1 === laptop0
storage node1 === link3 === router node0

my router says
link1 is 100Mbps
link2 is 10Mbps
link3 is 100Mbps

without changing the physical cables, let's replace the graphic node with a laptop2

Code: [Select]

      laptop1 === link2 === router node0 === link1 === laptop0
storage node1 === link3 === router node0

my router says
link1 is 100Mbps
link2 is 100Mbps
link3 is 100Mbps

It could also be a bad soldering problem on the network connector or the transformer on the board.

yup, it looks like the graphic node0 is triggering the ethernet phy to the lowest configuration possible due to a bad transform.

I am going to replace it.

[CaptCrash]

One odd thing I saw was with old 3Com 10/100 cards plugged into a Netgear 5 port Gigabit switch.  If you used 1 meter cables, auto-negotiate would fail.  Use 10 meter cables and everything worked perfectly! 

wait, this maybe a serious problem here. Cables need to be very short, less than 1 meter, something like 40cm, because all nodes must stay inside a metal case.

I have come across this quite a bit with IT equipment where people use short cables to attach devices to ethernet switches.  My understanding is that for active ethernet devices a minimum cable distance of 1.5m is required (approx 4').
I would expect that if you hard set the speeds of both ethernet ports and use a 1m cable it will probably work fine, otherwise use a longer cable and auto negotiate is likely to be ok.
Either way you would be much better off setting the speeds/duplex so that you dont have items changing on you and negotiating at one end differently to the other.

[Halcyon]

I have come across this quite a bit with IT equipment where people use short cables to attach devices to ethernet switches.  My understanding is that for active ethernet devices a minimum cable distance of 1.5m is required (approx 4').

Thanks for sharing, very interesting. I can't say I've come across this myself but it's not often when I need to connect such devices so close to a switch. That being said, I also use enterprise switches (Cisco, HP etc...) and most of them auto-detect cable length and switch to a low-power mode on shorter runs.

[NiHaoMike]

If the problem appears to be the signal being too strong, what about make an attenuator to reduce it?

[ogden]

I have come across this quite a bit with IT equipment where people use short cables to attach devices to ethernet switches.  My understanding is that for active ethernet devices a minimum cable distance of 1.5m is required (approx 4').

Not true. There is no minimum cable length mentioned in IEEE 802.3. 0.5m or even 0.25m patch cords are widely used in networking and they work. Also - most multi-port routers/switches have some kind of management CPU which is then connected to built-in switch ASIC. Both are active devices. Sometimes those are very close to each other on the PCB, few mm apart. Such kind of PHY connection usually is transformerless, just using AC coupling capacitors.

If the problem appears to be the signal being too strong, what about make an attenuator to reduce it?

Nah. Just urban myth. Typical 100m cat5e attenuation at 100MHz (max specified freq) is 22dB (Belden). Accordingly attenuation of 1.5m is 0.33dB (1/1.08) which is so small that can be ignored.

[magic]

Short cables are certainly not a guarantee of autonegotiation failure because I use some too and they work.

But it wouldn't occur to me that they may cause a failure, so thanks for the warning I guess.

[CaptCrash]

I have come across this quite a bit with IT equipment where people use short cables to attach devices to ethernet switches.  My understanding is that for active ethernet devices a minimum cable distance of 1.5m is required (approx 4').

Not true. There is no minimum cable length mentioned in IEEE 802.3. 0.5m or even 0.25m patch cords are widely used in networking and they work. Also - most multi-port routers/switches have some kind of management CPU which is then connected to built-in switch ASIC. Both are active devices. Sometimes those are very close to each other on the PCB, few mm apart. Such kind of PHY connection usually is transformerless, just using AC coupling capacitors.

If the problem appears to be the signal being too strong, what about make an attenuator to reduce it?

Nah. Just urban myth. Typical 100m cat5e attenuation at 100MHz (max specified freq) is 22dB (Belden). Accordingly attenuation of 1.5m is 0.33dB (1/1.08) which is so small that can be ignored.

In regards to your point about "no minimum cable length mentioned in IEEE 802.3.", thats correct.  But that also means that there is no standard for minimum.  So its up to each chipset maker to handle it in their own manner.  That is the key to the issue.

I came back to this thread as today at a customers, we were having an issue between a Cisco managed switch (older unit 2960 i think, but I'm not 100%) and a HPE Procurve 3810M managed switch (new install).  The customer had configured a 4 port LAG group between the switches and was getting unreliable traffic throughput, with ports in the LAG being unreliable (link loss and packet corruptions).
In testing I cable tested each of the cables, connected with my laptop to each switch with each cable.  These tests all worked fine with a laptop to switch but not switch to switch.
In troubleshooting, I swapped from 50cm cables to 2m cables.  Issues with the LAG group stopped and traffic is now fine.

Changing the cables between the two switches made a difference.

[ogden]

I came back to this thread as today at a customers, we were having an issue between a Cisco managed switch (older unit 2960 i think, but I'm not 100%) and a HPE Procurve 3810M managed switch (new install).  The customer had configured a 4 port LAG group between the switches and was getting unreliable traffic throughput, with ports in the LAG being unreliable (link loss and packet corruptions).
In testing I cable tested each of the cables, connected with my laptop to each switch with each cable.  These tests all worked fine with a laptop to switch but not switch to switch.
In troubleshooting, I swapped from 50cm cables to 2m cables.  Issues with the LAG group stopped and traffic is now fine.

Changing the cables between the two switches made a difference.

Still not buying it. Most likely it was timing/skew problem rather than signal strength. I bet single 50cm cable between switches would work w/o problems until end of the days (of any switch involved). Switch interoperability problems are results of bad karma. That's why you shall stick to one brand only and do not risk your network being cursed by salesman who lost his deal.

[MrMobodies]

I had a similar issue a decade ago with a dual port gigabit Broadbcom 5703 controller that was onboard a Supermicro X5DL8 and I had two of those server boards where the link speed kept on changing randomly on those particular nics and it will do it with any cable but there were options to turn it off in the driver and I think was ethernet wirespeed. No problems after that I did my checks on throughput and no packet errors on the switch port statistics.