1) Can I have the powerlines and the digital signal in the same cable like that?
Yes, should be no problem at all.
2) If so, how should I make use of the twisted pairs in the cable.
One pair for the ADAT signal (made into DIFFERENTIAL, of course!)
One pair (BOTH wires together) for ground.
One pair (BOTH wires together) for + voltage
One pair (BOTH wires together) for - voltage.
DO NOT put +V and -V on two wires of the same pair.
you run the risk of blowing up something if somebody plugs the wire into the wrong place.
I have a bricked $700 streaming box that suffered that exact failure mode.
Completely avoidable with proper design.
Is it necessary to use a cable where the pairs are shielded against each other? Like U/STP or U/FTP?
The properly differential data pair should not need shielding any more than the millions of meters of UTP network wiring around the planet.
The power pairs should not need shielding as they can be very trivially filtered for any ambient grunge they may pick up.
3) If I use a any of those cables with shielding, I guess I need a jack that has shielding aswell?
If you use shielded network cable, then the male plugs on the cable AND the female jacks they plug into must all be metal/shielded.
Using an ordinary plastic connector at any of the four points in the chain completely obviates any benefit of using shielded cable.
4) Is it better to have the voltage regulators for the power supply before or after the cable. I guess the voltage would be more stable when the regulators are in the receiving device, but then I'd have power rails with considerable ripple in the cable.
It is better to have the regulators at the far end (destination/load) of the cable.
It is not AT ALL clear how that equates to "considerable ripple in the cable"
5) Do I need some kind of driving stage to transmit the digital signal or can I just go straight out of an optical reciever into that cable and place a buffer opamp on the recieving end? (That might be a really stupid question. )
Or at least unanswerable since we don't know (signal-wise) where you are coming from and where you are going.
As a generic response, I would say that it seems very unlikely that you have a proper differential signal available that can drive a long cable, and equally unlikely that you have a proper differential receiver at the other end. But since you have revealed nothing about either the source or destination circuits, we can't even really guess.
6) CAT3 goes up to 16Mhz. So that should be fine for my 12.288MHz signal, right? But since nobody uses CAT3 network cables anymore, maybe it is cheaper to use a more common cable such as CAT5 anyway?
Not sure where you can even find Cat3 cable anymore. Cat5 is ubiquituous and has perhaps the highest bandwidth/cost ratio on the planet.
Maybe I's a stupid idea and I should use another cable all together. Are there alternatives?
No I think it is an excellent idea. I was thinking of making a cheap TOSLINK to Cat5 adapter set (transmitter and receiver) just for running ADAT over long distances. Like a cheap, budget digital snake with 1~4 Behringer ADA8000 at each end.
But it is not at all clear to us what you are proposing down at the far end of the cable that needs +/- 9V? So we don't know how to answer your power questions.