If you want to use SAM-BA over UART, then you don't need a crystal. If you want to use USB, then you do need a crystal and load capacitors.
You probably won't need all the bypass caps just for programming, but you will absolutely need all the capacitors on the VDDCORE. They are a part of the internal LDO circuit.
If you follow all the guidelines in the schematic checklist of the datasheet, then I see no reason for it to not work.