Linus Torvalds says RISC-V will make the same mistakes as Arm and x86

[DiTBho]

kernel 6.10
Linus Torvalds said
RISC-V will make the same mistakes as Arm and x86

[..] Even when you do hardware design in a more open manner, hardware people are different enough from software people [that] there’s a fairly big gulf between the Verilog and even the kernel, much less higher up the stack where you are working in what [is] so far away from the hardware that you really have no idea how the hardware works [..]

[..]So, it’s really hard to kind of work across this very wide gulf of things and I suspect the hardware designers, some of them have some overlap, but they will learn by doing mistakes — all the same mistakes that have been done before [..]

[..]They’ll have all the same issues we have on the Arm side and that x86 had before them.
It will take a few generations for them to say, ‘Oh, we didn’t think about that,’ because they have new people involved[..]

[..]But even if RISC-V development is still expected to make many mistakes. It will be much easier to develop the hardware now. It took a few decades to really get to the point where Arm and x86 are competing on fairly equal ground because there was al this software that was fairly PC-centric and that has passed. That will make it easier for new architectures like RISC-V to then come in[..]

umm ... 

[DiTBho]

not sure what the "mistakes" are 

[brucehoult]

He doesn't know any, he's just hypothesising.

Most RISC-V people are not "new" but are coming from long experience with x86 and/or Arm, and all major decisions are made with very wide consultation with domain experts in both industry and academia, not by half a dozen "new" people sitting around a table in a back room with the boss constantly asking them if they are done yet.

If anything, the problem with the RISC-V ISA process is that it is too careful, too slow, for companies impatiently wanting vector/SIMD processing, hypervisor, crypto acceleration, etc etc.

Individual companies are of course making mistakes, in designing their cores, in building an SoC around the cores, etc.

The weird thing is that Arm is the opposite, pushing out all kinds of ISA extensions with almost no one picking them up.  E.g. they published the SVE spec in January 2016 ... 8 1/2 years ago ... as an optional part of ARMv8.2-A but it's seen nearly zero uptake until it was forced on people in ARMv9. Virtually all commercial cores and chips are still on ARMv8.2-A ... without SVE ... and that includes things such as the RK3588 and whatever the chip is in the Pi 5 (both A76 cores). Meanwhile they published ARMv8.3-A (Oct 2016), ARMv8.4-A (Nov 2017), ARMv8.5-A (Sep 2018), ARMv8.6-A (Sep 2019), ARMv8.7-A (Sep 2020), ARMv8.8-A (sep 2021), ARMv8.9-A (Sep 2022). And starting in Sep 2019 the ARMv9-A series.  As far as I know most of those have zero implementations, except by Apple (A12: ARMv8.3-A, A13: ARM8.4-A, A14/M1: ARM8.5-A, A15/A16/A17/M2/M3: ARMv8.6-A, M4 ARMv9.2-A). Qualcomm I believe was on ARMv8.2-A right up until a jump to ARMv9 with Snapdragon 7 Gen 1 and Snapdragon 8 Gen 1.

[coppice]

Is he talking about poor instruction set and memory model types of choices, or the subtle stuff the RISC/V people are not going to be any more immune to, like information leaking/injection flaws?

[madires]

I think it's about not seeing the big picture. Hardware and software people have different points of views. They need someone in between, i.e. someone who understands both worlds and helps to explain or manage the big picture.

[SiliconWizard]

If it's about security holes due to speculative execution, for instance, sure. No reason chip designers would not make the same "mistakes". RISC-V is an ISA, not a microarchitecture.

The kind of "mistakes" RISC-V will avoid is instruction sets that become absolute monsters with thousands of instructions and thus the need for excessively complex decoders and execution units.

But one issue that is rampant is that due to the heavily modular ISA (which is also one of its strong points), RISC-V runs the risk of becoming very fragmented. So that over the years, any particular implementation may still be much leaner than a typical x86, but all existing extensions combined, the number of effective available instructions is not unlikely to largely exceed what ever went into an x86. After that it's only a question of marketing. Because to be used effectively, this modular approach also implies more reasonable designs with relatively specific uses, instead of making CPUs that will just contain as many features as is physically possible, which is what Intel/AMD are pretty much doing. Will RISC-V CPUs be immune to this feature inflation if they become as widespread as x86 ones for similar applications? I don't know. Marketing is a bitch.

[coppice]

But one issue that is rampant is that due to the heavily modular ISA (which is also one of its strong points), RISC-V runs the risk of becoming very fragmented. So that over the years, any particular implementation may still be much leaner than a typical x86, but all existing extensions combined, the number of effective available instructions is not unlikely to largely exceed what ever went into an x86.

So, you are worried that RISC/V will be just as fragmented as x86 currently is, with a dozen subsets of AVX512.

[Nominal Animal]

One interesting facet of the Linux kernel development is how use directs the development.  Someone working in the userspace needs a facility, and works with others to get such a facility included in the Linux kernel.  Features are not developed first and then wait for users to find out about it; it works the other way around.  In fact, when such facilities are developed, they often get modified until they are both useful (and not easily replicated in pure-userspace software) and maintainable within the kernel: any design will go through that wringer before getting included (except for drivers for new hardware that fit in to the existing driver framework).

This means that Linus and other kernel developers are used to seeing interfaces used in novel ways, and also getting bit by hardware features that make some approaches impossible or overly complicated or annoyingly slow, simply because the hardware designers were not aware of those practical use cases and their patterns beforehand.

I personally don't think that will ever change, because we already have reached the complexity levels where a single human being cannot master everything on the hardware side, nor everything on the kernel side, nor everything on the userspace side.  I do believe this is what Linus was referring to, plus the tendency of people designing competing products or "reinventing the wheel" so to speak to fall into the same potholes their predecessors did, especially because they believe they know better than those predecessors (hubris, really).  (It is especially funny in archaeology, where everything that does not have an immediate purpose to a modern-day human is a "religious artefact".)

RISC-V is interesting, because although it is designed by "lot of people", it does not seem to suffer from "designed by committee".@brucehoult might be able to enlighten us how they avoided that, but I suspect most of it is common purpose, hashing out a solution the slow way instead of giving in just to avoid a difficult discussion, and modularity.

[DiTBho]

the hardware designers were not aware of those practical use cases and their patterns beforehand

I don't know, it's very complex

In my small way though... I can think about how things went in writing the myC machine level for the Arise-v2 architecture

Usually you think about the ISA first - the hw-guys do it, who then have to implement it in hw -
then you write the machine level of the compiler - the sw-guys who have to implement the toolchain

In my case I did exactly the opposite. That is, I modified the ISA several times to simplify my life as much as possible with monads &C.
And the result? It's a "strange" thing, that probably nobody likes, except me.

And this is also the real reason why these things are better done in large teams, with people who have different and much more specific skills.
I don't know if it's possible to organize and manage, but this seems like a case where things converge toward a higher level of "suboptimal" the more brainstorming you do.

[Nominal Animal]

In my case I did exactly the opposite. That is, I modified the ISA several times to simplify my life as much as possible with monads &C.

Yep, and the result probably included constructs you could not have imagined beforehand needing/being useful.

Something as simple as choosing between LL/SC and CAS for atomics makes a huge implementation on the underlying hardware architecture.  I believe (but have no real experience in ISA design, so based only on my experience in coding for various architectures and approaches, and the bugs they have had) that LL/SC is simpler.  The first multiprocessor/multicore environments I could really delve into were all x86-based, so my initial experience was with CAS and bus locking; it took me quite a while to see how LL/SC solves the same problems with fewer instructions in the instruction set.

Any previous experience definitely causes some bias, until one gets enough experience to get a proper perspective.  It does to me, at least; I've had to learn to compensate no matter how much it might 'sting'.  I think it is like being in a valley, or on a mountain surrounded by other mountains, and exploring that biome in detail; then one morning waking up in a jungle or a sandy desert or salt flats.  All together, there is way too much there to really delve into in a single lifetime.  However, it is a difficult job to create and manage a large team of focused creative people; it is not an accident they say that managing software developers is like herding cats.

And this is also the real reason why these things are better done in large teams, with people who have different and much more specific skills.

For sure; even I have done my best work in a team, not alone.  It's just that you don't just slap together a team and expect excellent results of it; often it just collapses and key members leave.  It takes something more.  A benevolent dictator is one possibility.  I don't know what RISC-V folks did, but as I see it, they avoided the "design by committee" mismash/discontiguous results with little extra bits sprinkled here and there without any overall design.  It isn't perfect, but what I've seen thus far, I like.  How well it performs in practice, we'll see as the devices become more and more widely used.  (I sometimes check marc.info mailing list archives for messages with risc-v in the subject.)

[nctnico]

not sure what the "mistakes" are 

That would be good to find out. Otherwise it is just FUD

[David Hess]

He doesn't know any, he's just hypothesising.

Torvalds has direct experience with these issues from his work at Transmeta, and from porting Linux between ISAs, especially newer ones, where all of the same mistakes were made.

Is he talking about poor instruction set and memory model types of choices, or the subtle stuff the RISC/V people are not going to be any more immune to, like information leaking/injection flaws?

He is referring to both, but mostly the former which impacts operation.  Torvands has participated in many public discussions about this problem going back more than a decade.

Going by memory, examples of problems include:

Weak Memory Ordering - The strong memory ordering of X86 became one of its advantages.
Page Size - Notice that Apple backed off from 64K pages, compromising on 16K pages. (1)
Lack of Support for Unaligned Access
Lack of Support for Shorter Datatypes - Alpha did this.

(1) Page size relates to cache addressing, so a larger page size allows faster access of larger caches, but larger pages can cause other performance problems.

[DiTBho]

LL/SC and CAS

LL/SC depends on the cache subsystem, and on MIPS R10K it's problematic, especially on non-coherent system.
I remember serious problems supporting the Linux kernel on some R10K systems.
We had to patch gcc, resulting in poor performance and problems.

So, not only the architectures, but also the cache design are important.

[David Hess]

LL/SC depends on the cache subsystem, and on MIPS R10K it's problematic, especially on non-coherent system.
I remember serious problems supporting the Linux kernel on some R10K systems.
We had to patch gcc, resulting in poor performance and problems.

Linus had software assisted cache coherency and software assisted page translation on his list of bad ideas also.

[rhodges]

Linus had software assisted cache coherency and software assisted page translation on his list of bad ideas also.

But that made embedded MIPS so much FUN!
And being able to use cached pointers or uncached just made the day so happy!

[Nominal Animal]

So, not only the architectures, but also the cache design are important.

Absolutely.

I was thinking more along the lines of inter-core bus locks (that x86 does), compared to a coherent cache subsystem with a relatively simple cache-level signal whenever a cacheline is transferred between two cores' or processors' caches, and handling the reservation on that cache line sensibly in hardware, if I was creating say an FPGA core.  Adding DMA and priorities makes it a bit more funky, but I guess that is the reason DMA bypasses caches on so many Cortex-M cores.  The bus lock sounds like a simple solution, but I've dealt with my share of complicated mutex deadlock scenarios with multiple concurrent cores to know it is a deceptively complex approach, with the same kind of sneaky butt-biting gotchas speculative execution has.

[SiliconWizard]

All this is not defined by any RISC-V spec, at least that I know of. And, yet. Maybe it will happen over time. Although you stilll want implementation flexibility as RISC-V is made for all kinds of targets and not just "big systems" with a Linux-like kernel.

Memory ordering at this point is left to implementation, the ISA just providing fence instructions. Not unreasonable. Possibly a bit slippery, or unsufficiently fine-grained.

[DiTBho]

coherent cache subsystem with a relatively simple cache-level signal whenever a cacheline is transferred between two cores' or processors' caches, and handling the reservation on that cache line sensibly in hardware, if I was creating say an FPGA core

On a typical MIPS implementation of LL/SC

Code: [Select]

Load  linked     : LL rt, offset(rs)
Store conditional: SC rt, offset(rs)

To implement a memory semaphore, each LL/SC sequence has three sections:

• section1: The LL loads a word from memory
• section2: A short sequence of instructions checks or modifies this word
• section3: The SC stores a new value into the memory word, unless the new value has been modified.
  If the word has not been modified, the store succeeds and a 1 is stored in the destination register.
  Otherwise the Store Conditional fails, memory is not modified, and a 0 is loaded into the destination register

In section3, to understand if a word has been modified or not, the hw uses the cache dirty mechanism, and this creates quite a few problems
Indeed, section2 must not contain any of the events listed below, or the SC will fail:

• exception
• execution of ERET
• load instruction
• store instruction
• SYNC instruction
• CACHE instruction
• PREF instruction
• external intervention exclusive or invalidate to the secondary cache block containing the linked address

I recently implemented two Arise-v2 cores and wrote a very simple scheduler that works in SMP.
I didn't implement LL/SC in the traditional MIPS way, also because I didn't implement the cache.

Cache makes everything more complex, mutex and semaphore mechanisms should not rely on cache.