The peak DC voltage will be approximately Vac x 1.414 minus two times the voltage drop on the diodes that form the bridge rectifier, because at any point in a bridge rectifier, there's two diodes conducting
The voltage drop will vary with the current, the heat of the rectifier... if you oversize the bridge rectifier (for example use a 20-40A bridge rectifier that can be easily found both as new or in some dead atx power supplies) then the voltage drop on these diodes will be around 0.8v... but for simplicity we can just use 1v per diode.
... The max AC current will be approximate 0.62 x Iac .
You have 100VA toroidal, so that's 50VA per secondary or 50VA/12 = 4.16A ... which means you'll have approximately Idc = 0.62 x 4.16 = 2.58 A so your transformer is sized properly to handle 1.5 A, the maximum for your LM317.
So you'll have Vdc peak = 12v AC x 1.414 = ~ 17v - 2 x 1v (drop on diodes) = 15v
Depending on the transformer quality, at very low loads, the transformer may actually output more than the nominal 12v AC so it's best to be aware of that and pick suitable components with this in mind. For example, knowing this I would not assume that just because my estimated maximum DC voltage is 15v it is safe to use a capacitor after the bridge rectifier that's rated for maximum 16v, I'd use one rated for at least 25v, ideally 35v.
Also, the transformer's output may sag a bit below 12v in certain conditions (like mains voltage dropping a bit, combined with high current output)
The linear regulators need to have the input voltage always higher than output voltage by some amount, in order to regulate the output properly. In the case of LM317 regulators, in order for them to regulate well across the whole range of 0.. 1.5A the datasheets say you must have the voltage at around 1.5v above the output voltage.
So after the bridge rectifier, you must use a capacitor that will buffer the voltage and make it so the voltage there will be at least 13.5v as much as possible.
To estimate capacitance you can use this formula which approximates the size :
C = Current / [ 2 x AC frequency x ( Vdc peak - Vdc desired) ]
So let's say you want 1.6A of current (a bit more than maximum 1.5A the LM317 is supposed to output), and let's say you want at least 13.5v all the time, and you assume the peak DC voltage will be 15v DC and let's say you're in US where the AC frequency is 60Hz ... put the numbers in formula and ...
C = 1.6A / [2 x 60 x (15-13.5) ]= 1.6 / 180 = 0.0088888 Farads ... or around 8888 uF ... so you would go with the next higher standard value that's easy to buy ... 2 x 5600 , 2 x 6800 , 8200uF , 2 x 8200 uF , 10000uF ... etc
More capacitance doesn't hurt, up to a point ... way too much capacitance can "stress" the bridge rectifier or the fuse in front of the transformer ... the capacitors basically act a short circuit "sucking" a lot of energy in them, which makes the bridge rectifier and the fuses "see" more current for a very brief period of time, while the capacitors fill up with energy.