Renewing my lab LAN

[SiliconWizard]

I'm going to upgrade my lab LAN. Currently, it's still 1Gbe with about 10 equipments connected to LAN. And then WiFi.

I'd like to upgrade to at least 2.5Gbe, possibly 10Gbe if it's not too expensive (which it looks to be still). So, that would be one router and 10 ports. The router can have fewer ports, and I'd add a switch if needed. 2 sub-lans would be enough, and ideally 2 WAN ports, so a 4-port router, and one or two switche should do.

Also, I'm considering fiber rather than copper. Again if the cost isn't prohibitive.

What kind of equipment would you suggest? (Again, not expensive stuff.)

[Halcyon]

Define "expensive". How much are you looking to spend in total?

To give you some idea, I'm a huge fan of pfSense and Netgate appliances. You can pick up something like the Netgate 2100 for USD$349, which will give you a very decent, commercial router/firewall product. If you have a very fast internet connection (say 1 Gbps or more), you might want to go up to the Netgate 4200. Of course, pfSense is still available for free, which you can run on any x64-based* machine with a few NICs.

As for switches and wireless, it's hard to beat Ubiquiti on price, performance and features, however some people don't like the idea of having to run a seperate network controller. I prefer it (I just run a dedicated VM, but you can host it on your desktop PC. It doesn't have to be running all the time, just when you need to monitor or reconfigure anything). If you're looking to build a 10 Gigabit backbone to your network, look at the Ubiquiti Pro switches. They have 10 Gig SPF+ ports and come in various models depending on how many copper ports you want, and whether you want PoE or not. The Pro 24 is around USD$399.

As for fibre vs copper, that will depend on your requirements. If you're in an area prone to storms, fibre on the WAN side, absolutely, even if it's just a short length with media converters. That will act as an air gap between the outside infrastructure and your network. For everything inside, I just use Cat 6. It's good up to 10 Gigabits. I run DAC cables between the switches and servers (these are cheap).

10 gig NICs can be expensive, but you might find some at a good price on ebay or similar. I picked up a bunch of Intel X520-DA2 NICs for about AUD$75 each a while ago. Being SFP, you can decide whether you want to use fibre or direct attach copper cables. Just watch out for Chinese fakes. If they are "too cheap", they might not be genuine cards.

You can have as many VLANs as you want on the same equipment. You just need to configure your router and switch(es) accordingly and decide whether you want Tagged vs Untagged VLANs. (I use port-based untagged VLANs for my devices.)

If all this is sounding too expensive, some of the "budget" brands I would go for are:
Switches: Netgear business-series; MikroTik; Second-hand Cisco switches.
Wireless: MikroTik; Second-hand Cisco or HP Enterprise.
Routing: pfSense on your own hardware; MikroTik.

Don't buy into this whole wireless "mesh" bullshit the consumer brands love to push. Meshing becomes entirely irrelevant if you're connecting your wireless access points via an ethernet cable. The ability to mesh only becomes useful if you aren't able to run a network cable to a radio, and even then, has its limitations.


* I know pfSense will also currently run on x86 hardware, however talk of dropping 32-bit support has been around for a while. It'll happen eventually.

[Psi]

I wouldn't bother with fiber unless you have a reason to

[Postal2]

Am I right that optical communication gives a high ping?

[Halcyon]

Am I right that optical communication gives a high ping?

With all things being equal, ever so slightly, but realistically not enough for most people to notice or even care about. You'd be talking nanoseconds.

However with copper, other issues could impact overall latency, such as crosstalk (which may lead to delays due to re-transmission). Considering most of the internet is built on fibre, your last little bit of cabling in your house is going to make zero difference.

[Postal2]

.... You'd be talking nanoseconds. ....

A friend of mine could not get less than 10 ms locally on optical fiber. That's why I asked.

[SiliconWizard]

I now have 2Gbps fiber for Internet.

You're right that fiber indoors would probably make little sense for now. One reason would be redoing (or doing from scratch) a complete installation and replacing all cables. Fiber takes significantly less space, so it's easy to fit almost anywhere.

Price range, that would depend on wether I redo all the cabling or not. I was thinking under $1000 if so, and probably about $500 for just a new router/switch and possibly some new cables. Which sounds doable.

For the router, I'd want something running Pfsense or OpenWrt - definitely not one with a proprietary firmware. I've seen some mini-PCs with 4 2.5Gbe ports, but I don't know how reliable these are, nor what the idle power consumption is. My impression is that it's a bit of a gamble with these mini PCs.

[Smokey]

What are you doing now that maxes out gigabit ethernet? 

[Smokey]

...
For the router, I'd want something running Pfsense or OpenWrt - definitely not one with a proprietary firmware. I've seen some mini-PCs with 4 2.5Gbe ports, but I don't know how reliable these are, nor what the idle power consumption is. My impression is that it's a bit of a gamble with these mini PCs.

What about AsusWRT? 

[Halcyon]

.... You'd be talking nanoseconds. ....

A friend of mine could not get less than 10 ms locally on optical fiber. That's why I asked.

That could be due to any number of things. I'm getting sub-10ms ping times out to the internet, which is fibre all the way to the house. On a local network, it should be far less than that (even on WiFi).

[Halcyon]

.... You'd be talking nanoseconds. ....

A friend of mine could not get less than 10 ms locally on optical fiber. That's why I asked.

That could be due to any number of things. I'm getting sub-10ms ping times out to the internet, which is fibre all the way to the house. On a local network, it should be far less than that (even on WiFi).

...
For the router, I'd want something running Pfsense or OpenWrt - definitely not one with a proprietary firmware. I've seen some mini-PCs with 4 2.5Gbe ports, but I don't know how reliable these are, nor what the idle power consumption is. My impression is that it's a bit of a gamble with these mini PCs.

What about AsusWRT?

My view on AsusWRT is that it's fine if you already have an Asus router and want to improve the existing firmware, but I wouldn't recommend it if you're in the market for a new router. It's still very limited in terms of capability compared to something like pfSense or Opnsense.

[Berni]

I now have 2Gbps fiber for Internet.

You're right that fiber indoors would probably make little sense for now. One reason would be redoing (or doing from scratch) a complete installation and replacing all cables. Fiber takes significantly less space, so it's easy to fit almost anywhere.

Price range, that would depend on wether I redo all the cabling or not. I was thinking under $1000 if so, and probably about $500 for just a new router/switch and possibly some new cables. Which sounds doable.

For the router, I'd want something running Pfsense or OpenWrt - definitely not one with a proprietary firmware. I've seen some mini-PCs with 4 2.5Gbe ports, but I don't know how reliable these are, nor what the idle power consumption is. My impression is that it's a bit of a gamble with these mini PCs.

Ah yeah the usual "My internet got faster than my LAN so now i have to upgrade"

Going faster than 1Gbit indeed can get expensive. 2.5G is catching on and you can get routers and network cards for it cheap (some new motherboards even come with 2.5G built in). So id say it is worth getting all 2.5G switches with a bit of 10G mixed in for when speed is actually needed. Going all 10G is expensive and you will not need those sort of speeds for the next decade.

For home use 10G over CAT6 is good enough. Most 10G interfaces are SFP ports anyway so you don't need to choose between fiber and copper. Yes fiber is thinner but only the actual fiber, the connectors on the end are not all that thin, also you can't easily cut it to length and terminate it yourself, the fiber is also a lot more delicate and has a minimum bend radius. So running fiber trough your house might not be as easy. Besides you likely already have a lot of twisted pair copper in your walls, just use that. Sure it might not be cable rated for 10G, but for short runs it usually works fine. If it doesn't work then you can always splice better cable onto it, then use the old cable to pull the new cable trough.

Doing NAT routing on 10Gbit of internet traffic (for future proofing) becomes quite a task, but for home use with a small number of clients it is easier. Good excuse to set up Pfsense on a small PC if you are into it.

Tho for me personally my 1Gbit fiber i have into the internet is less important than having 1Gbit to my NAS server where i keep most of my data. That NAS is due for a upgrade. Currently has dual 1Gbit integrated NICs while the one PCIe slot is taken up by a SAS HBA card. I feel like 2.5G is too small of a step up from 2x1Gbit, so i am mostly waiting for 10G to become more widespread and come down in cost, then rebuild the whole NAS to some more modern hardware.

.... You'd be talking nanoseconds. ....

A friend of mine could not get less than 10 ms locally on optical fiber. That's why I asked.

Typical LANs have ping times way under 1ms so it can't be measured by simply running ping in a console.(unless your "LAN" is connecting buildings many kilometers apart). Most of the cause of ping is due to having to share bandwidth with other traffic. If the common network link that your traffic is destined for is already transmitting someone else's traffic then your packets have to wait for their turn. This is made even worse if the "last mile" network of your ISP also commons connections together, this is very common with coax DSL where an entire street shares one backhaul, so every DSL modem in that street can hear everyone elses upload traffic, so for this to work they need collision avoidance where each modem is assigned a time slot for transmuting so that they don't talk over each other (the time of flight of the coax is also measured and compensated for) hence all upload traffic needs to wait for a time slot. Similar things are also done with some ISPs fiber networks where passive splitters are used to share one big backhaul fiber over multiple subscribers.

I get about 3 to 4ms of ping out into the internet over fiber. But this is of course only to things near my ISPs central office.

[dietert1]

Currently i am trying fiber ethernet for some metrology application. I got a 20m duplex multimode fiber and two tplink MC100CM media converters, all fast ethernet, for about 50 € in total. Of course ping response times are consistenly < 1ms.
Modern gigabit equipment should behave like that or better, yet i don't see a need to upgrade. Avoid waste! Observe energy consumption, which may be higher for gigabit capable stuff.

Regards, Dieter

[RoGeorge]

Wired LAN can do POE, which might be convenient, and waste less electricity than adding a wall adapter for each device.  Depends on what's connected to your LAN.

Another thing, I don't think it will be any noticeable improvement between 1 and 2.5Gbps during day to day usage.  Most probably you'll notice the difference only if you run a benchmark.

[Postal2]

Nevertheless I think copper is better for home.

I now have 2Gbps fiber for Internet. ....

Well, I congratulate, your business is booming. Naturally, you need a lot of traffic.

Providers are usually cunning. They say 2Gbps, meaning 1 down and 1 up.

.... hence all upload traffic needs to wait for a time slot. ....

Not only. Optical transmitter will wait for max payload of packet. Ping sends too small amount.

[bingo600]

Imho - For private usage 10Gb for everything is overkill.
And the "power usage" of 10Gb Copper interfaces are "just ugly".

I have deliberately postponed 2.5Gb to the "desktop", until the prices for 2.5Gb switches comes down to something that resembles the current 1Gb prices.
And it's actually quite rare i max out my existing 1Gb network, plus most of my "daily drivers" are on WiFi anyway ....


That said - I do have a 10Gb 8-port fiberswitch w. SFP's , and a 8-port 2.5Gb w. 10Gb uplink in the drawer.
It's waiting for my new DELL proxmox server to be setup w. an Intel-X520 card.

Until the DELL server is up, i'll just leave my 10Gb in the drawer.

Just my 0.02€ opinion.

.

[SiliconWizard]

You'll "max out" any data link, even 10Gb, if you transfer data from/to SSDs (at least NVMe). So that question is a non-question. Even a HDD from 10 years ago could do better than 100 MBytes/s. So, the LAN becomes a bottleneck. Whether that matters in your own use cases is your call. 1Gb has been default for at least 15 years. Looking for a bit of upgrade is not unreasonable.

2.5Gbe is uh, like 2.5 as fast, so that's a start. But clearly that would be a compromise, as 10Gb is still way expensive. And yes, it's quite power hungry too. Actually, I think 10Gb on copper is very power hungry, and should be a bit less so over fiber.

[Postal2]

You'll "max out" any {very expensive} data link, even 10Gb, if you transfer data from/to {very cheap} SSDs ....

It turns out that cheap water and a very expensive bucket. Dissonance.

[Halcyon]

10 Gig has its uses. I only run it between servers and the network backbone, not to client devices, but it still allows me to max out the performance on my old NAS with spinning rust disks. I can copy data to the NAS as speeds well in excess of what a 1 Gbps network can handle.

It really comes down to what you're using it for. Even if you just want to have a play and have the funds to do it, I say go for it! Everyone has a hobby, for some it's tinkering with networks. Do I need a full blown server rack and room in my house? No, but it's fun!

[Postal2]

...... Do I need a full blown server rack and room in my house? No, but it's fun!

I agree and understand. I have a neighbor who uses a hammer drill as a hobby. Network devices as a hobby is much better.

[Berni]

Yeah power usage of 10G is quite a factor, it can burn a couple of watts per port.

You can get that down using direct attach copper SFP cables, tho running those in walls is not really an option. But is very nice for connecting together switches and servers in the same rack/room.

I guess things got so stagnant in the networking world because going faster presents technical challenges while at the same time people in most cases don't really need faster networking. 1Gbit gets you around 120MB/s and that's about the speed that spinning rust HDDs run at. And it means you can copy 1 DVD worth of data in about half a minute. I find myself rarely moving more data than that at a time (apart from backups, but those run automated in the background anyway). But going from 100Mbit to 1Gbit was an absolute game changer if you ever transfer files over the LAN.

I think that give it 5 to 10 years for the cost of SSD storage to become so cheap that it will make sense to build a fully solid state NAS and then 10G will start making a bit more sense, hopefully with 10G coming down in cost to something reasonable too.

[Postal2]

I would like to remind about powerline adapters, which provide distances that Wi-Fi in the city can't reach.

[Halcyon]

I would like to remind about powerline adapters, which provide distances that Wi-Fi in the city can't reach.

True, but those powerline adapters aren't without their problems and performance is highly variable. I'm not a fan.

[Postal2]

.... and performance is highly variable. ....

I have 6 pcs (2+2+2) of different types. In the worst combination you will always get 20 Mbps. In most cases you will get over than 80 Mbps. One pair of my adapters (modern, rated as gigabit) gives 200 Mbps. All adapters are compatible with each other and can be combined as lan.

[nctnico]

I now have 2Gbps fiber for Internet.

You're right that fiber indoors would probably make little sense for now. One reason would be redoing (or doing from scratch) a complete installation and replacing all cables. Fiber takes significantly less space, so it's easy to fit almost anywhere.

But you'll also need tools like fiber & optical port cleaners. A good pair of those will set you back about 150 euros. Dirty LC connectors will cause problems. I always clean ports & connectors even when the fibers and optical parts are new from their package.

If I where to upgrade, I'd use 10Gbit for the backbone (from internet router to office) and 1Gbit to the devices. 10Gbit all the way would only make sense if  you need to have a lot of traffic between 2 devices.