I think the OP's comparison with consumer routers is forgetting that nearly all of the work had already been done for these groups. Linux already exists, and it can route packets. DD-WRT et al is mostly an exercise in cross porting and UI, the overwhelming portion of functional code in use comes from the mainline Linux distribution underneath.
A scope won't be like that. You get Linux for the core device handling, but you essentially need to start from scratch to deal with all of the functionality of a scope.
Do-able? Sure, if you have a large team and several years. It's a much larger undertaking than a router.
It's even more than what you mention. DD-WRT started after the original code was shared by Linksys, given that Linksys used GPL code and was forced to share it. I'm familiar with AsusWRT Merlin, and as much as that firmware is awesome, it's still mostly based on the GPl code shared by Asus. And AsusWRT Merlin uses binary blobs coming directly from Asus
A lot of the functionality of a modern scope comes from the FPGA, and I doubt that even motivated hackers could make much progress there without the manufacturer sharing a lot of info.
I can see the possibility to improve the UI functionality somewhat using an approach similar to Magic Lantern for the Canon cameras, which in most cases uses clever overlays
https://magiclantern.fm/ and had no help from Canon. But it's falling more and more behind the Canon release train.
The risk for an enthusiast project is that by the time there is enough functionality to be useful, the state of the art will have moved to the point where that effort is obsolete (say, having an open source firmware for a Rigol DS1052a today, when the DS1052e is pretty long in the tooth even for many hobbyists), and very likely the maintainers will have moved on before that point is reached
I can see more value in hacking simpler and cheaper scopes, like the FNIRSI scopes (already mentioned), with simpler HW and much crappier software. But for Rigol/Siglent, I'm not sure it's energy well spent, given the investments made by the manufacturers and the complexity/functionality of the existing firmware