Ethernet as System Comm Link

[fourfathom]

I sometimes reminisce about past design decisions, and one that still bugs me, 25 years later, is an inter-board communications link in a telecom product (the Cerent 454, a SONET optical/electrical network switch). 

I was director of systems engineering, and essentially the product architect, and we were figuring out how to  have the individual interface and switching boards and common control boards communicate.  I had proposed using point-to-point 100BASE-T ethernet (backplane connected) with Ethernet switches on the redundant common control cards.  However, my proposal was shot down by our software team, with them trotting out the "Ethernet is an unreliable protocol" argument.  I tried to explain how the underlying physical layer was just as reliable as any other physical interface and protocol -- this wasn't thinwire Ethernet with vampire taps or Tees where collisions were possible, and the switch ICs had more than adequate buffering.  If they wanted more reliability then they were free to add an upper-layer protocol on top of the physical layer I was providing.

But they were adamant.  "Ethernet Is Unreliable", and with me being a hardware engineer and them supposedly being the networking experts, I finally had to give in.  Instead, I gave them a custom design, built into the ASICS we were designing -- in my opinion no more reliable than the Ethernet-switch physical layer I had proposed, but apparently more than adequate for our purposes.

But it still bugs me now.  I believe that the software team had absorbed the "UNRELIABLE" concept during their education, and hadn't really thought about what that meant.  I was already fighting a lot of battles at that stage, but I wish I had stuck to my guns on that one.  It would have been a cleaner solution.

So am I wrong?  If so, why?  I would truly like to learn.

[madires]

No, your idea of using Ethernet was perfectly reasonable. Around the same time Juniper did excatly that. They used Ethernet for connecting the routing engine modules (basically industrial PCs).

[Benta]

I think you're being a bit unfair here.
Back then, several network technologies were in play, especially in Telecoms.
Your colleagues back then would probably not have used the term "unreliable", but rather "non-deterministic". And that's a fair characterization of Ethernet.
Especially in Telecoms, where correct data package time delivery is essential, I understand their concern.
Other technologies in the running back then were Token-Ring (deterministic) and FDDI (deterministic), a few others, plus any amount of proprietary formats.

[fourfathom]

I think you're being a bit unfair here.
Back then, several network technologies were in play, especially in Telecoms.
Your colleagues back then would probably not have used the term "unreliable", but rather "non-deterministic". And that's a fair characterization of Ethernet.
Especially in Telecoms, where correct data package time delivery is essential, I understand their concern.
Other technologies in the running back then were Token-Ring (deterministic) and FDDI (deterministic), a few others, plus any amount of proprietary formats.

But this wasn't network payload data, just internal configuration and housekeeping data.  They did indeed say "unreliable".

The product used traditional STS-1 and VT-1.5 circuit-switching internally (we designed our own STS-1-granularity 60 GBit/s throughput switch ASICS, and high port-count VT-1.5 switch ASICS, as well as customer-port interface ASICs).  One of our claims to fame was being able to map GigE into SONET, with multi-port switched ethernet on our cards.  Of course we also had OC-3 to OC-192, as well as 12-channel DS3 and DS1 cards).  Any card in any slot (but some slots were more equal than others).  I did some ASIC design, particularly a high-density DS3 chip, using digital clock synthesis and PLL techniques, but mostly I was responsible for system-level design and board-level advise.

You're right, this was all happening at the time when the traditional networks used guaranteed circuit-switched timeslotted channels, and ATM / cells were to be the hoped-for modernization.  TCP/IP was non-deterministic and still being studied for use in traditionally-deterministic applications.  We designed and built an ATM card, but it got dropped before anyone actually deployed it (and good riddance!)

We got acquired by Cisco, and I spent many meetings having to explain circuit-switched networks and various deterministic redundant network failover topologies to the router guys.  Our product actually used those new-fangled network discovery methods to figure out how to best configure itself in a semi-random network topology (it wasn't always BLSR or similar, sometimes it was more of a mesh).

Fun times! (I told you I have been reminiscing!)

[Cerebus]

..., and we were figuring out how to  have the individual interface and switching boards and common control boards communicate. 

You're talking about moving control plane traffic around, right? One presumes that getting the SONET data plane traffic around was the whole point of the switching cards.

But they were adamant.  "Ethernet Is Unreliable", and with me being a hardware engineer and them supposedly being the networking experts, I finally had to give in.  Instead, I gave them a custom design, built into the ASICS we were designing -- in my opinion no more reliable than the Ethernet-switch physical layer I had proposed, but apparently more than adequate for our purposes.

But it still bugs me now.  I believe that the software team had absorbed the "UNRELIABLE" concept during their education, and hadn't really thought about what that meant.  I was already fighting a lot of battles at that stage, but I wish I had stuck to my guns on that one.  It would have been a cleaner solution.

I'm suspecting that at some point one of more of them had heard "Ethernet is not a reliable medium" and failed to realise that means "does not have guaranteed delivery" not has "has high failure rates" or "has high bit error rates". In a chassis environment, where you're in control of everything there is no reason for ethernet to be less reliable [in the common sense] than any other layer 2 protocol/medium. Indeed, the likelihood of systematic failures in a home spun ASIC is likely to be higher than in commodity ethernet chips that have been tested in 1000s of applications rather than just one.

Back then, several network technologies were in play, especially in Telecoms.

You're saying that to a man talking about designing a SONET switch is a bit like saying "Now, Grandma, you hold the egg like this and suck here".

Your colleagues back then would probably not have used the term "unreliable", but rather "non-deterministic". And that's a fair characterization of Ethernet.

Firstly I suspect that he knows what his colleagues said and putting words into their mouths that suit your argument ought to be a bit infra dig. Secondly this is a custom build of a switched ethernet network inside a product, it isn't a general network. It can be made as deterministic as one wishes, if deterministic is even a necessary property for this particular network application.

[fourfathom]

..., and we were figuring out how to  have the individual interface and switching boards and common control boards communicate. 

You're talking about moving control plane traffic around, right? One presumes that getting the SONET data plane traffic around was the whole point of the switching cards.

Yep.  The control plane was completely independent of the data plane (although we had ways to read and write network-control data in the SONET overhead bytes carried by the data plane.)

It was a great time for me, being at a significant transition in networking (and it's still a hybrid network).

[Benta]

Back then, several network technologies were in play, especially in Telecoms.

You're saying that to a man talking about designing a SONET switch is a bit like saying "Now, Grandma, you hold the egg like this and suck here".

Your colleagues back then would probably not have used the term "unreliable", but rather "non-deterministic". And that's a fair characterization of Ethernet.

Firstly I suspect that he knows what his colleagues said and putting words into their mouths that suit your argument ought to be a bit infra dig. Secondly this is a custom build of a switched ethernet network inside a product, it isn't a general network. It can be made as deterministic as one wishes, if deterministic is even a necessary property for this particular network application.

Yes, Cerebus.
Your acerbity after reading a subsequent post brings you no points. Rather the opposite. Go sit on your garden bench and smoke your pipe. Hindsight is wonderful.

[Cerebus]

Back then, several network technologies were in play, especially in Telecoms.

You're saying that to a man talking about designing a SONET switch is a bit like saying "Now, Grandma, you hold the egg like this and suck here".

Your colleagues back then would probably not have used the term "unreliable", but rather "non-deterministic". And that's a fair characterization of Ethernet.

Firstly I suspect that he knows what his colleagues said and putting words into their mouths that suit your argument ought to be a bit infra dig. Secondly this is a custom build of a switched ethernet network inside a product, it isn't a general network. It can be made as deterministic as one wishes, if deterministic is even a necessary property for this particular network application.

Yes, Cerebus.
Your acerbity after reading a subsequent post brings you no points. Rather the opposite. Go sit on your garden bench and smoke your pipe. Hindsight is wonderful.

Hadn't read it when I wrote that and I think you mean "perspicacity". Also doesn't alter the point about determinism. And no thank you, I'll pass on the cancer bowl.You might think you're on here to "win points" but I'm most certainly not, and having watched the number of spats you've got into with people recently perhaps you are. Tone it down, you're not making yourself look good.

[Cerebus]

It was a great time for me, being at a significant transition in networking (and it's still a hybrid network).

About that time I took delivery of my first STM-16 SDH ADM and a bunch of STM-16 and STM-4 circuits, it all sounded so fast at the time. Three years later everything was replaced with the next generation/speed step up of kit and so it repeated henceforth. It was also the time that us operators were just starting to avoid ATM if we could, despite the fact that a lot of equipment vendors were still pushing it hard - got fed up with paying "cell tax". About three 1/2 years ago I did a consulting stint for one of the really big telcos and there wasn't a single ATM circuit left anywhere near the core network, and precious few near the edges either as far as I could tell. Precious little left on their network that looked like PDH or SDH or any of the end to end circuit switched technologies except in the places where some customers were still on E1s.

[fourfathom]

Yeah, I had some time before that designed VHF radios and I thought that was high speed tech. All too soon our digital bit rates were way faster than my VHF carrier freqs.

Guys, please don't fight on my account.   I appreciate  all this feedback.

[nctnico]

Yeah, I had some time before that designed VHF radios and I thought that was high speed tech. All too soon our digital bit rates were way faster than my VHF carrier freqs.

Same here! A couple of decades ago I started out with a 4MHz home computer and nowadays I'm designing circuits / boards with memory busses running at 1.6GHz and high speed interconnects running at several GHz. It boggles my mind that it works despite the fact that the math, simulations and practical trials confirm it works. Interesting times for sure 

[mansaxel]

Your colleagues back then would probably not have used the term "unreliable", but rather "non-deterministic". And that's a fair characterization of Ethernet.

Firstly I suspect that he knows what his colleagues said and putting words into their mouths that suit your argument ought to be a bit infra dig. Secondly this is a custom build of a switched ethernet network inside a product, it isn't a general network. It can be made as deterministic as one wishes, if deterministic is even a necessary property for this particular network application.

I think we should read that as 'Your colleagues back then would probably have used the term "unreliable", as meaning "non-deterministic"' in the interest of fairness.  That was what the circuit switching people told us IP people back in those days, so I think it fair to assume.

(This text below is not trying to lecture people who know better, I'm just trying to not make too many errors in describing to the audience the problem at hand, as I understand it. And perhaps learn something when I'm told how wrong I am.)

On the second point, determinism is just a matter of how much PDV (that which sales people call "jitter") one tolerates in packet delivery, as long as there is no packet loss. (A switch will be able to drop packets; it probably won't do it, but the frequency if you've done your homework likely will be about as service affecting in terms of frequency as bit errors in a well maintained PDH/SDH circuit.) The core question of course is where the queues are. The synchronous case will not admit packets/frames at edge faster than forwardable end-to-end, whereas the switched Ethernet will accept line rate regardless of provisioning level upstream.  As long as data rates are compatible with system bandwidth the Ethernet will be as deterministic as the synchronous net.

Where Ethernet (or any hop-by-hop technology with varying speeds) has problems, is when line speed is decreasing. If you're feeding your wiring closet switch with 10GE and it's got data hungry clients connected at 1GE, and the server is connected to the core switch in the machine room with 10GE (all-in-all not an unrealistic situation) the quality of the wiring closet switch is going to be much more critical than the core switch. Because the downconversion is going to create a queue of packets waiting to go out on the slow interface. Such queues must use special memory called (TCAM) in order to not slow down the packet forwarding. If the TCAM buffers are full, the switch will drop outgoing packets. Such memory is expensive which means it is a scarce resource, and needs careful management.  This is one of the things that sets switches apart, and why some switches are more expensive than others. 

From this it is trivial to deduce that a single-speed Ethernet is much easier to make almost-deterministic. And, if that was the case, it mostly proves that the original idea would have been feasible, and as has been mentioned, this was already proven by Juniper.

[m k]

I had once an Ethernet to Centronics dongle for a printer and it didn't work with big files.
It didn't know how to wait.

One other case was a 10baseT network of Apple machines.
It didn't work since Macs at the time didn't localize practically anything opened over the network.
Something like sucking the operation through the network like a processing terminal.

But if Ethernet is completely encapsulated inside something deterministic is it still Ethernet?
I'd say it's just a wire with a capacity.

[fourfathom]

From this it is trivial to deduce that a single-speed Ethernet is much easier to make almost-deterministic. And, if that was the case, it mostly proves that the original idea would have been feasible, and as has been mentioned, this was already proven by Juniper.

I think you've missed the essence of my original question.  The control-plane (configuration and management of the various boards in the chassis) doesn't require a deterministic interconnection, and I wasn't looking for one.  My goal, and apparently Juniper's, was to provide a reliable comms path.  We didn't need the equivalent of a timeslotted protocol, or even the fixed-size cell of ATM.  Variable frame-length ethernet and varying (but reasonable) delays would have been just fine.  My approach would have guaranteed no collisions and in-order frame delivery, but the software guys were fixated on "UNRELIABLE", which was not the case here.

I think UNRELIABLE became the mantra during the original thick and thin-wire coaxial ethernet implementations, where we had a shared physical medium and relied on collision-detect, jamming and backoff to make it work.  And of course RELIABLE  comms are carried over UNRELIABLE networks everywhere and everyday, thanks to higher-level protocols.

This control-plane was completely independent of the timesliced data-plane switching system.  These days we have somewhat eliminated the need for a traditionally deterministic data-plane and network.  Instead we rely on speed, statistics, and protocol.  The days of having to use every single bit to within a nanosecond of its life are largely behind us, at least in telecoms (*).

(* More reminiscences:)
Thin-wire ethernet: During my first start-up we pulled the thin-wire ethernet out of the walls to go with 10BASE-T (and of course faster, later).  I ended up with boxes of RG-58 BNC jumpers, TEEs and terminators.  Twenty years (thirty?) later I'm still using these (for RF work) in my home lab.

Multiplexing, Bit-Stuffing, Floating Payloads, Synchronization, Plesiochronous Networks, Metastability, and Clock Synthesis, Prime Numbers:  Well, too much, and off the subject at hand, but lots of fun.  I would love to talk about this and share some of what I learned if anyone cares (hint).

[mansaxel]

From this it is trivial to deduce that a single-speed Ethernet is much easier to make almost-deterministic. And, if that was the case, it mostly proves that the original idea would have been feasible, and as has been mentioned, this was already proven by Juniper.

I think you've missed the essence of my original question.  The control-plane (configuration and management of the various boards in the chassis) doesn't require a deterministic interconnection, and I wasn't looking for one.  My goal, and apparently Juniper's, was to provide a reliable comms path.  We didn't need the equivalent of a timeslotted protocol, or even the fixed-size cell of ATM.  Variable frame-length ethernet and varying (but reasonable) delays would have been just fine.  My approach would have guaranteed no collisions and in-order frame delivery, but the software guys were fixated on "UNRELIABLE", which was not the case here.

I think we agree more than you think ;-)  I fully appreciate your original approach. That is what we've got TCP for!  The gut reaction of the TDM people likely is, as you write in the part I snipped, based on a hubbed Ethernet, on which CSMA/CD was the norm, and that very clearly required every bell and whistle of TCP to deliver traffic.  On a well-behaved modern Ethernet link, these problems are gone. I run audio and video over Ethernet, uncompressed, without retransmissions (no time for them!) at work, and it works out just nicely, if you do your design right.

On timing, as you write; if we can hold packet delay variation well below the timeout value for various control function timeouts, we're good. And the extra energy spent in teaching the control program to accept data when it arrives, and not to crash/hang if it does not arrive exactly when it should (according to the TDM clock), is well invested, because it makes the control system resilient.

[Cerebus]

..., Plesiochronous Networks, Metastability, and Clock Synthesis...

AKA how to cross between two clock domains that are half a country apart. And the guys trying to do it inside the same FPGA think they have problems. 

[fourfathom]

..., Plesiochronous Networks, Metastability, and Clock Synthesis...

AKA how to cross between two clock domains that are half a country apart. And the guys trying to do it inside the same FPGA think they have problems. 

Yep, those FPGA wimps don't have to deal with diurnal phase variation, do they?

[fourfathom]

On timing, as you write; if we can hold packet delay variation well below the timeout value for various control function timeouts, we're good. And the extra energy spent in teaching the control program to accept data when it arrives, and not to crash/hang if it does not arrive exactly when it should (according to the TDM clock), is well invested, because it makes the control system resilient.

Yes.  And again, delay and jitter requirements in the control plane are orders of magnitude more relaxed than in the network data links.

I've been out of that loop for a long time, but I suppose in many cases now the separate "box management" control plane has evolved to being just a custom LAN within the much larger WAN.  At least that's how I would approach it (absent stuff I don't know about).  I like simplicity and uniformity.

Well I am gratified that my hunch about using switched ethernet seems to have been validated.  I've always thought I was right, and I had heard of this method being used elsewhere, but until now have never brought it up among my peers.

[Cerebus]

You might think you're on here to "win points" but I'm most certainly not

<splutter!>

So you're trying to say your participation in this thread is pointless are you? I quite agree. Knock it off will you, that's is a completely pointless interjection, designed to generate heat, not shed light.

[Cerebus]

You might think you're on here to "win points" but I'm most certainly not

<splutter!>

So you're trying to say your participation in this thread is pointless are you? I quite agree. Knock it off will you, that's is a completely pointless interjection, designed to generate heat, not shed light.

Mote, eye. Pot, kettle. Maybe if you hadn't come out trying to put down Benta there would be no heat, just light, and you wouldn't have to resort to faking my words to justify yourself either. Perhaps you should tone it down and, ah, knock it off. Or will you?

No I'll just do what I should have done in the first place, ignore you.

[nctnico]

On timing, as you write; if we can hold packet delay variation well below the timeout value for various control function timeouts, we're good. And the extra energy spent in teaching the control program to accept data when it arrives, and not to crash/hang if it does not arrive exactly when it should (according to the TDM clock), is well invested, because it makes the control system resilient.

Yes.  And again, delay and jitter requirements in the control plane are orders of magnitude more relaxed than in the network data links.

I've been out of that loop for a long time, but I suppose in many cases now the separate "box management" control plane has evolved to being just a custom LAN within the much larger WAN.  At least that's how I would approach it (absent stuff I don't know about).  I like simplicity and uniformity.

Well I am gratified that my hunch about using switched ethernet seems to have been validated.  I've always thought I was right, and I had heard of this method being used elsewhere, but until now have never brought it up among my peers.

There are plenty of realtime ethernet protocols around. Think about modbus-tcpip or (better) Ethercat for example.

With a switch-chip in between you have buffering as well. Recently I had to resort to a bit-banged ethernet MAC (yes, you can do that) which is way way slower than a real ethernet MAC but with a switch-chip on board which does packet buffering, the actual packet loss is extremely low.

All in all, I think your solution would have worked well using simple parts.