There are various combinations of chemicals used to make different color LEDs, which results in LEDs having different forward voltages (minimum voltage from which led starts to light up) and at the same time depending on chemistry and color, the LEDs will have the same brightness at different current amounts.
Human eyes are also more sensitive to some colors, red and green mostly ... hence why we use red on semaphores because it's most easy to notice.
The 10-20mA on leds is mostly a default value, you will find that there's lots of modern leds that will be quite bright even at 5mA. Some "high brightness" red leds will be quite bright even at 1-2 mA.
The amount of brightness you will perceive will also depend on the lens of the led .. there's leds designed to have a big spread (ex 120 degree spread) and there's leds that focus the light coming out in a narrower angle, like a flashlight.
These narrow angle leds will be much brighter at lower currents, but if you lay flat your device with the leds on the desk and you sit on a chair, you may have difficulty viewing which leds are bright because your eyes are at a too wide angle to see the color.
Port expanders are overkill for this. Those would be useful if you have to mix leds with buttons, if you need inputs, lots of things.
There are simpler options like led drivers for example. Some led drivers are super simple, behaving like shift registers... for example you simply shift 8 bits into the chip using a data and a clock wire, and then you enable the chip to switch the 8 leds to the state of those bits. The current of the 8 leds is set using a single resistor and these led drivers can also be chained together, so for example, you could chain 12 such 8 channel led drivers to have maximum 96 leds - you would send 5 0 bits because 6 channels on a chip won't be used, and then you'd shift the 90 bits into the chained led drivers and hit the enable/latch to switch the state of all 96 channels in one shot, then repeat the process to get as much updates per second as you wish.
There are also such led drivers with 16 led channels, here's for example this STP16CPC26 led driver :
https://www.digikey.com/en/products/detail/stmicroelectronics/STP16CPC26MTR/2772228If you look at the datasheet, you will see it has a data in and clock to shift bits in, a black to reset all leds to 0, a latch to update all leds according to the bits shifted in and and even an output enabled pin should you wish to use it.
You can chain multiple such chips by connecting SO pin to the next pins, data pin and connecting the clock pins together.
You can also see that you can control the current down to at least 5mA on this chip through a single resistor.
There's other examples like CAT4016 for example but it's a bit harder to solder :
https://www.digikey.com/en/products/detail/onsemi/CAT4016HV6-GT2/2193932It can however output as little as 2mA on each channel and would use less space on a circuit board.
There's also drivers which can drive more leds and use i2c, like for example this 36 led channel IS31FL3236 :
https://www.digikey.com/en/products/detail/lumissil-microsystems/IS31FL3236A-TQLS4-TR/14308337You can have 3 of these on the same i2c and you set each chips address through resistors because each device on a i2c bus needs a unique address, and then you can simply send 36 bits to each led driver through i2c
You should also consider multiplexing ... you don't necessarily need to have 12 x 8 bit led drivers, or 8 16 bit port expanders.
You could have for example only 2 x 16 bit led drivers and connect 3 leds on each channel. Depending on which led receives power, only that led lights up when the channel is enabled .
You will turn on power to sets of 32 leds through a transistor or mosfet.
So in your code, you could set the 32 bits to the state of the first 32 leds, send power to the first 32 leds and you'll get the first 32 leds working. Wait a few milliseconds, then turn off power going to the first 32 leds, send the 32 bits for the next set of 32 leds and turn on power to 2nd set, wait a few milliseconds and continue with next set of 32, and then loop back to the first set of 32 leds.
Because of retention you will see all 3 x 32 leds working.