Lars' 1ns resolution TIC is designed around the 1.1V ADC voltage reference available in the 328P and a few other AVR MCUs, but unfortunately not in the 32u4. To use a 2.56V AREF you would have to change a number of values in the calculation() routine, and you would probably lose some resolution and linearity. Also you can change the resistor that charges the 1nF capacitor to get a voltage swing of approximately 2.5V instead of 1.1V in Lars' original schematic.
Another solution is to use an external 1.1V voltage reference, but that requires a PCB redesign to add the cicuitry for the external 1.1V voltage reference circuit.
In the worst case, abandon the 32u4 and switch back to the original 328P that Lars used.
I did mention the AREF pin when you asked for comments early on, but I didn't check the 32u4 datasheet for its internal voltage references. My bad.
Edit: Lars' GPSDO schematic is quite simple but the complexity lies in the source code and how everything is tied together. If you have an oscilloscope do check what happens at the 1nF capacitor in the TIC, triggered by the PPS. It's very interesting!