Like someone else said above, a bridge rectifier will convert your AC voltage to a DC voltage with a peak voltage equal to about 1.414 x Vac - ~ 1v (voltage drop in the diodes of the bridge rectifier)
That 15-17v is peak voltage, it's not all the time that much. After the bridge rectifier you need a capacitor to guarantee the voltage the regulator sees is always at least a minimum amount ( 12v + dropout voltage of the linear regulator)
You can estimate the value of the capacitor with this Capacitor size = Current / [ 2 x Mains AC Frequency x (Vdc peak - Vdc min)]
So for example for 12v 1A , if your linear regulator has a dropout voltage of 1v, it needs to always have at least 13v on the input. If you're in UK where mains frequency is 50 Hz and the Vdc peak is 17v, you'll need C = 1 A / ( 2 x 50 Hz x (17v-13v) = 1 / 100*4 = 1/400 = 0.0025 Farads or 2500 uF ... so at least 2500uF of capacitance are required to always have at least 13v at the input of whatever.
The datasheets of linear regulators assume the input is fairly filtered, so whatever capacitance they recommend at the input doesn't take in account the capacitance required to keep the output of the transformer reasonably smooth and at certain minimum.
It's generally a good idea to at least use a 0.01uF ceramic capacitor for decoupling very close to the in and gnd pins of the regulator, and if the linear regulator is far away from the big capacitor right near the bridge rectifier, they often recommend an additional capacitor, usually around 10uF to 100uF in size.
Some linear regulators also require a tiny amount of capacitance at the output - usually any kind of capacitor 1uF or something like that would work, but there are some designs which require capacitors with a bit higher capacitance value (around 10uF..22uF) and a minimum amount of ESR which makes them work best with electrolytic capacitors (some won't work right with ceramic capacitors at the output). If that's the case, it's usually written in the datasheet.
Your 5v regulator could be connected directly to the output of the bridge rectifier + capacitor, or if the 12v linear regulator can provide more power than what your circuit will use, you can connect the 5v regulator at the 12v regulator output.
Keep in mind though that linear regulators regulate voltage by dissipating what's not needed as heat, so if your product uses 5v at 100mA, the 5v regulator will take the same amount - 100mA - from the 12v linear regulator. Don't forget that when deciding where to connect the regulator.