For a low volume, high price device, I don’t think it would be unreasonable to test each unit by hand. Depending on the age of the design and how precise these machines need to be, there could be potentiometers, set screws, etc that need adjusting while the machine is in some sort of calibration test jig. This may need to happen well after assembly because sometimes devices sit in storage and nobody wants to get a cal cert which is 3 years old! Serial port may just be for firmware upgrades.
If you can’t observe any communication, it might be hard to get anywhere. And if you aren’t able to open it, you may find that the 9 pin D connector has non standard wiring (it certainly wouldn’t be the first expensive life sciences instrument to do that).
I think your only real option given the constraints is to send random garbage in the pin you think is RX, listen on TX and hope that you get something useful.