the ideal input current is 3200*0.006/12=1.6A
my Idc is about 2A and Ipeak is about 18A
18A peak? That's a pretty big transformer.. and with 2A average, that means the primary switch spends most of its time off. Also 18A is a lot to switch through a MOSFET. Even with say 0.05ohm on resistance, without switching it'll have to dissipate ~16W..
considering Idc efficiency is 1.6/2=80%
I'm not sure where you got these numbers from. Efficiency is power out/power in. You say you have two charge pump outputs, one positive and one negative, both 1600V with a 260k ohm load.
That means either output is drawing 1600^2 / 260k = 9.85W. For two outputs = 19.7W.
Your input is 12v @ 2A (I doubt this value). Which is 24W. Eff. 19.7/24 = 0.81 which is indeed 80% where did the 1.6 come from?
i use a 3329 ferrite core and my primary winding is 21 turn and secondary winding is 700 turn and pwm frequency 4.8khz and duty cycle 19%
i figure out the core become too hot after a while
A quick google says thast roughly 35mm x 24mm E core? Fine for 20W but if you have 18A on the pimrary of one of those - its saturating, unless it has a monstrous air gap in it.
Ok, the turns ratio is 700/21 = 1:33.3 exactly as you wanted for 400V out from 12V in (for a forward transformer, not a flyback). But the duty cycle... for a step up, normally the duty is >50%, but can be much less for discontinuous mode.
4.8kHz = 208us period. At 19% duty that is an on-time of: 0.19 * 208 = 39.6us. If your peak is 18A, with an input voltage of 12V, then the inductance is:
I(peak) = V/L * T Rearranged for L. L = V/I(peak) *T = 12/18 * 39.6u = 26.4uH. Again this is based on your numbers, so if they are measured, then that's roughly what your primary inductance is.
However yours isn't a flyback converter. I suspect what is happening is you're either your frequency is too low, or because it is a push-pull converter being driven asymmetrically, the core is saturating.
As was mentioned many posts ago, go for a royer push-pull configuration. It will require another primary winding, and a feedback winding, but is much more suited to what you're after - plus its output voltage is pretty much down to the turns ratio. If you insist on driving it with one transistor, then scope the primary waveform, and reduce the ON-time to get the peak current down, and see what happens when the switch turns off.