There are RS485 driver ICs from Maxim which have 1/2, 1/4 and even 1/8 unit load which means that you can connect up to 64, 128 or 256 devices (respectively) on a single RS485 bus.
Some examples are: MAX487 (1/4 load, 250Kbps), MAX1483 (1/8 load, 250Kbps) MAX1487 (1/4 load, 2.5Mbps), MAX3061 (1/8 load, 500Kbps, fail-safe, hot-swappable), MAX13485 (1/4 load, 500Kbps, fail-safe, hot-swappable).
Your solution will also work but I would go with the RS-485 driver ICs unless you have long distances between RGB matrix boards.
I'm actually working on a similar system for garden lighting -> one master controller and a bunch of slave units communicating over an RS-485 network using MAX3061 ICs.