What are those protocols usually, something like command followed by N data? So, there'll be configuration registers, an address register, commands to mutate them, and commands to read/write bursts of data?
So a random R/W, is a small fraction of total throughput: if you're transmitting 3 bytes of address for 1 byte of data (R/W) that's 1/4 raw, but also the overhead of two command bytes, so 1/6th or about 8MB/s!
If you were doing a display that has exclusive RAM access during scan, and maybe allows one other access during horizontal retrace and some more during vertical (or if this is for an LCD panel, then not necessarily retrace, but the corresponding "porch" timings typical of unbuffered display controllers), that might work; especially if data can be streamed out continuously without being restricted to fixed length burst modes.
I suppose performance would be comparable to old school EGA or VGA, where the display controller takes priority and refuses bus access (forcing WAIT states until available). The redraw time is usually abysmal.
But yeah, by far the better solution is to just get some SD or DDR RAM that the FPGA is happy with. Could even start with a dev board that comes with both; they're very affordable these days.
AFAIK, the RAM controller is just a drop-in IP block, and resolves all the timing and bus access you need. Maybe add some buffering for the priority output stream (display read), since it's a hard real-time output. Could even control refresh cycles, since you probably don't need to waste time refreshing currently-unused RAM? Or, you can use it for everything else too; if this is for a vintage computer add-on, for example, you could add a direct RAM window (as EMS or etc.) and toss a few megs at the CPU for no cost.
Tim