Configurable peripherals in a small microcontroller would be nice. Suppose I want to have a processor read a value from an ADC at exactly regular intervals.
One option would be to set up a timer. Maybe it has a couple of outputs; one which can toggle a GPIO to start acquisition, and then another output a short time later which calls an ISR. The ISR wiggles a few more GPIOs, or maybe kicks off a DMA transfer which in turn uses an SPI peripheral to suck the data out of the ADC. Then, when the DMA transfer is complete, another ISR is called which wiggles another GPIO, sticks the ADC result in a buffer, and signals to the foreground task that it's available.
All this is a PITA to set up, as well as taking up potentially valuable CPU time.
How much better would it be to simply create a peripheral in programmable logic which does exactly this? Kick off a timer at start up, wiggle pins as necessary, then only signal the host CPU when the final data is already in a FIFO?
Better yet, signal the host CPU when that ADC data has already been filtered, analysed, averaged, interpolated, checked for outliers and generally passed as valid and meaningful to the top level application?
FPGA fabric in a CPU would be a great thing, but not to make the CPU faster.