So yesterday I tried a number of things, and thought I had discovered what was triggering the bug. I still may have, as the scope is no longer locking up, but it's not conclusive. Here are the details:
I swapped out the switch - no change, i.e. the scope still locks up. I also determined that the scope would lock up when connected to a switch along with my Windows desktop computer and nothing else, but would not lock up even if every computer in my house other than my Windows desktop were also connected. So, clearly whatever the issue is, it was being triggered by some traffic originating from my Windows desktop.
At that point, I decided to set up WireShark and get some network captures. Unfortunately, my switch does not support port mirroring, and I was not able to reproduce the lock-up with the scope connected to a 10MB hub (the only hub that I have). So I decided to just put the scope and my desktop computer on the switch, and run WireShark from my desktop. I captured a number of lock-ups this way. The attached capture shows one that was especially interesting. In this capture, the scope booted and locked up almost immediately; the whole process from first network packet from the scope until lock-up took less than 30 seconds.
Now note that this capture is pretty brief, and does not include one type of network packet that the scope may have been receiving, namely broadcast packets. So I did some more captures using broadcast captures. I don't want to post those here, as the likelihood that there may be some security-related or personal info goes up and I don't have the time to review every packet to make sure that's not an issue. However, I can tell you that there were ARP broadcast packets, some broadcasts for that same scanner software, some NETBIOS broadcast packets, and some dropbox local sync broadcast packet. I also see some HTTP exchange between the scope and the PC, specifically it is some LXI identification query. I don't think this is causing any issues, because I also see these when the scope is stable with no lock-ups for 12+ hours. Also note that when the scope locks up, it always right after it sends 3 ARP queries to the desktop, each of which asking what MAC address owns its IP address.
So, looking at this capture, and the others, I decided the only thing that looked unique was these strange scanner software packets. In WireShark they are labelled as protocol "BJNP", but they are actually UDP packets to some particular port. The UDP point-to-point packets are sent to port 8612, and the UDP broadcast packets are sent to port 8612. The BJNP broadcast packets seem to be some mechanism for the Canon printer/scanner drivers to identify Canon printers and scanners on the network. I have no idea why the driver also sends such packets directly to specific IP addresses. But in any case, I no longer own any Canon printers, so I uninstalled this driver software. And then after rebooting everything, including the scope, wonder of wonders, the lock-ups stopped. This is a plausible explanation, as this computer is the only computer on the network that still had the Canon software installed on it, and would explain why the scope only locks up when this computer is on the same LAN with it.
However, that's not the end of story. I decided to write a small "UdpPing" program to allow Rigol to recreate this scenario without having to install old Canon printer drivers. I did that, but so far I have not been able to recreate the lock-up. It's possible that there is some network traffic that I didn't capture (multicast packets maybe?), or perhaps my program isn't faithfully recreating the exact packets that I did capture. I'm going to poke around with it a little bit later today, and will post back here if I find out more.