http://hyperphysics.phy-astr.gsu.edu/hbase/electronic/rectbr.htmlHave a glance at that. You have a sine wave, then you rectify it, so all your peaks are on one side. Then you smush down the wave form with a smoothing capacitor.
Once you look at the pictures the voltage level differences will be obvious
So, your probe attenuates 10 times in 10:1 mode, theoretically this means you can put 3000 volts into your probe and your oscilloscope will still be ok. However, the problem is that the probe would require special design to withstand these voltages. The insulation in your probe and the capacitors must be able to withstand 3000 volts, if you tried this with an ordinary probe it is likely that something would fry (I would guess the capacitor would short out).
You can get something that can work at a few thousand volts cheaply on ebay.
http://www.ebay.com/sch/i.html?_odkw=high+voltage+probe&_osacat=0&_from=R40&_trksid=p2045573.m570.l1313.TR6.TRC1.A0.Xhigh+voltage+probe+oscilloscope&_nkw=high+voltage+probe+oscilloscope&_sacat=0However the nice really high voltage stuff will be built very weird, not just thick insulators. There is one tektronix probe that features a gas insulator, there is some kind of freon in the probe that works well as a high voltage insulator and does not effect the bandwidth of the probe much.
As for your question, your multimeter in AC mode will measure RMS. As to how accurate it is, this depends on frequency and crest factor. There are different techniques for determining RMS measurements,
1) make the assumption that the input waveform is a sine wave, do a Vpp measurement and then convert this to RMS using the formula using a microprocessor, cheap, fast, but it is inaccurate with anything but a sine wave
2) various analog conversion methods (squaring it and averaging it, there are a few variations on this method)
3) record it on an oscilloscope and do it mathematically
4) thermal
Your multimeter if its mega cheap will do #1, your oscilloscope will do #3, your medium to high end multimeter will do some variation of 2. Different methods have different trade offs (some might give a faster response at the cost of accuracy, others may break down at higher frequencies due to nonlinearities in the nonlinear analog circuits, and they will have different accuracies at different crest factors (how fucked up the wave form is = crest factor).
The ultimate judge of RMS voltage is a thermal meter, this essentially uses your signal to heat a DC to high frequency resistor and measures the temperature change in the resistor. This method will not lie to you and it will work on really screwed up wave forms, including noise, accurately. It is not fast to respond however as you need to actually heat something up. Furthermore the bandwidth will be much wider with a thermal meter. You will notice attenuation MUCH MUCH sooner in an analog RMS circuit rater then a thermal one (your meter will sometimes have 3 db rating, the frequency at which a sine wave will by off its real RMS value by 30%, just like your oscilloscope does).
As for converting RMS to Vpp, sometimes its really tricky. Consider the VPP of this waveform and its RMS reading:
You need to hook up an oscillsocope :p