Altium just has really shitty snap behavior. The only way I've found to get it to stop fighting me like this is to turn off snapping entirely, which is effectively what holding the Ctrl key does. You can also turn off hotspot snapping like RiZsho mentioned using Shift+E (which actually cycles through three different modes). There are a bunch of options in the snap section of the PCB properties panel (when you have no objects selected). The conflict resolution mode also matters, you can cycle through the options using Shift+R, and setting it to Ignore Obstacles will give you complete freedom to put the things wherever you want--and of course create DRC violations in the process. Reducing the snap distance also helps, this requires you to get closer to a snap point before snapping to it, but this of course makes it harder to snap to the things you WANT to snap to.
In a previous job I did a lot of AutoCAD work, and I desperately wish someone at Altium would take a look at their snap system, which is truly the best I've seen in any software ever. Great granularity in what you can snap to, which angles snaps are tracked in, and the snap tracking is extremely well done, which makes it very easy to line objects up based on snap points without having to dodge a bunch of snap points you don't want. You can even create temporary snap tracking points on-the-fly in the middle of other commands, so for example you can track off the ends of two non-intersecting line segments to extend them to where they would intersect, then create a temporary track point at that location, and then track off of that temporary point in whatever directions you've configured.
But mostly I just want Altium's snap system to not be an active hindrance like it is now.