Google Coral M.2 on non-x86 GNU/Linux SBC

[DiTBho]

My point was: MSI(-X) is the only way to do interrupts on PCIe.  There are PCI interrupt emulation messages, but it's all MSI based.  So either that means it has no interrupts at all, or it means something different than what I understand.  It's hard for me to see how a PCIe bus that doesn't support interrupts either meaningfully implements PCIe or supports much of anything.

so, according to your reasoning then every single bunch that offers miniPCIe PCIe should also offer MSI 

Message Signalled Interrupts are an alternative in-band method of signalling an interrupt, using special in-band messages to replace traditional out-of-band assertion of dedicated interrupt lines.

Supported

• in PCI bus since its v2.2
• later available PCI Express bus, but not always present in every SoC PCIe
• in some non-PCI architectures

Just to re-check again, I contacted the MediaTek' technical support

>The Google Corale M.2 card needs MSI-X support.
> Your datasheet doesn't mention it: is MSI-X supported?

Not supported, MT7621 SoC is PCIe 1.1 host.

[nctnico]

This works just fine on Arm64 platforms.

64bit arm 

which SBC? with m.2? with miniPCIe?

With my self designed SBCs using iMX8 SoC or Jetson TX2 module. Both have M.2 slot and PCIe.

[DiTBho]

my self designed SBCs using iMX8 SoC or Jetson TX2 module. Both have M.2 slot and PCIe.

impressed 

what about the software side in the userland?
standard SDK from Google (so, python-based) and standard kernel drivers? as described by them?
or something hacked/customized?

and, may I ask for what is the TPU used for?

[ejeffrey]

My point was: MSI(-X) is the only way to do interrupts on PCIe.  There are PCI interrupt emulation messages, but it's all MSI based.  So either that means it has no interrupts at all, or it means something different than what I understand.  It's hard for me to see how a PCIe bus that doesn't support interrupts either meaningfully implements PCIe or supports much of anything.

so, according to your reasoning then every single bunch that offers miniPCIe PCIe should also offer MSI 

Message Signalled Interrupts are an alternative in-band method of signalling an interrupt, using special in-band messages to replace traditional out-of-band assertion of dedicated interrupt lines.

No, this is wrong.  MSI is optional in parallel PCI but mandatory in PCIe.   PCIe has no out-of-band interrupt signalling mechanism at all.  Go ahead and check: the connector pinout has no interrupt pins.  The conventional PCI out of band interrupts (INTA, INTB, INTC, and INTD) are implemented as dedicated messages in PCIe.  This is mostly necessary to allow PCI to PCIe bridges, native PCIe devices would not normally use them.

Note that MSI and MSI-X are different.  MSI-X allowed more interrupts and multiple interrupt destinations to allow more flexible interrupt routing in multi-processor systems.  I don't know when MSI-X became mandatory, it's possible that early PCIe versions did not have mandatory support for MSI-X.

It's also possible that there is some confusion between the interrupt signalling mechanism (which is always MSI in PCie) and how the host interrupt controller lets you configure things.  I'm not familiar with the host and OS level configuration.

[DiTBho]

MSI is optional in parallel PCI but mandatory in PCIe.   PCIe has no out-of-band interrupt signalling mechanism at all.  Go ahead and check: the connector pinout has no interrupt pins.  The conventional PCI out of band interrupts (INTA, INTB, INTC, and INTD) are implemented as dedicated messages in PCIe.  This is mostly necessary to allow PCI to PCIe bridges, native PCIe devices would not normally use them.

Note that MSI and MSI-X are different.  MSI-X allowed more interrupts and multiple interrupt destinations to allow more flexible interrupt routing in multi-processor systems.  I don't know when MSI-X became mandatory, it's possible that early PCIe versions did not have mandatory support for MSI-X.

It's also possible that there is some confusion between the interrupt signalling mechanism (which is always MSI in PCie) and how the host interrupt controller lets you configure things.  I'm not familiar with the host and OS level configuration.

ah, ok, now I got it, thank you for your clarification 

That's probably what they meant when they replied:
MSI yes by default
MSI-X not supported in ePCI v1.1

Now I have to understand why and if the Google Coral strictly really needs MSI-X instead of MSI.
Then I will have to re-check the Linux kernel support for Mediatek MIPS SoCs

[ejeffrey]

My understanding of the way this works:

MSI: each endpoint (bus/device/function) gets a single unique memory address.  It writes to that address to signal an interrupt.  The interrupt number (up to 32) is part of the message payload.

MSI-X: each endpoint is given multiple unique memory addresses, one per requested interrupt.  Also, more interrupts are supported (up to 2048)

Message routing is only based on the address not the data.  So a given device using MSI can only send interrupt messages to a single group of cores.  MSI-X uses distinct addresses for each interrupt source so they can be more easily routed to a particular processor/core.

This is all at the PCIe transaction level.  You also need to consider the capabilities of your platform interrupt controller.  I guess (I'm not sure) it's also possible that a given platform interrupt controller doesn't support that many interrupts and just combines them into one per device.  If a platform does this but your device driver isn't written to gracefully handle this situation then it won't work even if the interrupt signalling is MSI-X.  Again, I could be wrong about this, I don't know much about how platform interrupt controllers or the device driver architecture works, I'm more familiar with the PCIe transaction layer.