1.How would you find out what kind of capacitance you need, or if you need it at all. I know It should be a Y type.
It´s long ago I personally worked on this topic and from there I only remember to make it as large as possible to reduce EMI and as small as possible to reduce the longitudinal AC current. Since you are working with DC input you should have no problems there, so I would just try the largest Y1 cap that is available from the catalouge. With DC input this cap has no obvious disadvantages.
2.Is there some rule of thumb or formulas how you should split up the total output capacitance you need in multiple capacitors?
I think there was one EEVBlog video on Youtube about this question. Practically you should look at the impedance curves and you should try to spread the impedance minima over a large frequency range. The electrolytic cap usually has its resonant frequency below 1MHz and e.g. the diode of a flyback transformer easily creates EMI around 30MHz. So you definitely should add some good ceramics from 100pf to 100nf or similar.
3.The Flyback will be powered from my bench power supply. So I thin it should be fine.
Anyhow, if I already spend the money for a custom PCB I´d also spend the money for a small common mode choke in the input line. May some radio amateur say thank you for this.
4.The Transformer has a isolation voltage of 3000Vrms. Is there anything else I need to check for the Transformer?
If the transformer is intended to isolate a SELV-circuit from mains, then there are some more requirements for its construction. Like using triple insulated wire and/or additional insulation sheets. If insulation failure would create a hazard for the operator, I would apply the same standards.