Well, porcelain is glass, mostly SiO2, with enough Ca and K (or other fluxes) to make it goop, and just enough Al2O3 to hold it together (recrystallizes into a mullite skeleton on firing). Hence the translucent quality, it's as glassy as you can get without the stuff slumping into a puddle in the kiln. Likewise the high density / low porosity, low and consistent expansion (no phase change due to free quartz/cristobalite), value as an insulator, etc.
Agate is quartz, but so finely crystalline that it's basically glass anyway. Quartz doesn't have cleavage planes either, but a conchoidal fracture, so the crystallinity wouldn't make much difference in terms of hardness. (Grain boundaries can still be a point of weakness though, hence why granite for example is weaker to certain forces.)
Both have a Mohs hardness of 7, or very close.
Hardness is the ability to resist deformation (scratching, abrasion) by another material; it isn't bulk strength, cleavage, intergranular strength, etc. You can quite easily chip and break these materials with a steel tool (Mohs 4-5?), and I think it's pretty obvious if you have a go at one of these with a hammer and chisel, you'll be able to chip, dent or fragment them even without much difficulty.
Likewise, media doesn't need to be especially hard to do the job. Steel balls in a ball mill, can be pretty hard, but they're also regularly used to break up rock and ores of quite high hardness. It's the impact that does it: say two balls clack together (giggity?) and there's some grains of rock inbetween, maybe quartz, maybe corundum, whatever. To some extent, the grains will push into the surfaces of the two balls, deforming them, maybe chipping out a bit of metal, but the deformation can also serve to simply grab onto the grains, and then the whole impact force is transmitted through those tiny grains, which quickly fail under compression. The balls aren't likely to hit straight-on either, rotation or a glancing blow turns into a shear force tearing the grains apart.
A ball mill might end up contaminating the charge with a lot of iron (or whatever the balls are made of), but if that can be acid-washed out, or just doesn't matter, well, that's fine.
You're working a mortar and pestle the same way, whether by impact, smashing up larger bits with the pestle, or with a thrust-and-gyrate motion, or a continuous orbital motion, either way applying large forces to sharp fragments as they stir around and catch randomly between the grinding faces. It's just a hell of a lot slower because it's your hand doing it, lol.
Inevitably, some of the grinding material will come loose and contaminate the sample; if this is undesirable, harder materials should be chosen, which is where a zirconia or alumina set might be desirable.
You also want to avoid reactions with the grinding material. It was, I believe, mercury compounds can react with copper on fine milling, seemingly a reaction like cold-welding but involving the transfer of compounds,
https://en.wikipedia.org/wiki/Mechanochemistry . That makes silica attractive, it's pretty inert to most things. This might make steel undesirable for grinding something like tin compounds, if that's something you need to do.
Tim