Not sure what loads (voltages and currents) you intend to drive with those outputs? There are auto-sensing, bidirectional level converters which you might be able to use, which could avoid the need for a switching matrix entirely. They automatically detect which side is driven, i.e. which pins are input vs. output. The catch is that they can only drive very weak DC loads, a few 100 µA DC, but have much stronger boost circuits to drive level transitions.
When I looked into these about two years ago, I found:
- TXS0108E (Texas Instruments); PI4uls5v108 (Diodes Inc) 2nd source.
- NXB0108 (NXP); FXMA108 (OnSemi/Fairchild) 2nd source
If you have enough I/O pins on your FPGA, you could also think of bi-directional drivers with additional inputs for direction control. I am pretty sure I came across some which allow individual direction selection per pin, but I did not take notes since they were not suitable for my intended use.