ALL coils are 5V. There is no 12V going to any coils. In fact, through all of this testing 12V ISN'T EVEN being used, I am powering the board with 5V and bypassing the 5V regulation circuit for now since I am not actually trying to drive any load through the relays, just make them switch.
I have checked and made sure that every single pin on that chip is connected as drawn in the schematic, nothing is shorted to anything else, I am completely stumped this makes absolutely no sense to me. I double checked and even made sure there was good ground return from all grounded pins, and there is, I tied all grounds to the pad which has a huge open connection to ground plane, I think the chip's logic is working and only based on the fact that DOUT WORKS! If I stick data in, I see it spit back out the other side.
There are 2 5V latching coil relays attached to this chip, No resistors, just direct 5V in on one side, and other side to OUT pin (as this device is a sink device), MAYBE without a resistor on the coil it's asking for too much (over 70mA rated for each pin on max chip, I doubt it), the relay is EE2-5TNU, datasheet says coil has 178 ohm resistance, at 5V that is 28mA, so yeah I don't think it's over current. Only other thing attacked is a 74HC04, I have the inputs pulled up with 10k (It's just acting as an inverting buffer to drive external SSRs)