Why most people say "vendor supplied libraries are bad"?

[Siwastaja]

Well, the "library" approach is for people who simply cannot write even a few lines of code, but still somehow manage to get something done by massive code copy-pasting and library call operations. I truly wonder how it's possible to build anything like that.

Case in point, for motor control, when you need to calculate exactly these, and only these, operations:

x = (1/sqrt(3)) * (y + 2*z);
d = Ia*cos(phi) + x*sin(phi);
q = x*cos(phi) - Ia*sin(phi);

- invariably, absolute in 100% of cases I have seen or read about, this set of operations performed maybe every 0.1ms, is called "computationally intensive" "math wizardry" "typically requiring DSP or FPGA" and will be hidden in a massive library behind multiple function calls.

Learning how to use the library easily takes days, though. In reality, it would be 1 table lookup, 6 multiplications, maybe 4 additions. But no, no can do. Need massive DSP and preferably a closed-source library just because they are even more wizard...ious?

Any appnote or tutorial discussing this makes extra sure not to reveal this math, only talk about its fanciness and complexity, because the library user could otherwise make difficult questions, such as "why?" or "oh but isn't that trivial?"

[nctnico]

Come to think of it... I already have my own API with underlying functions which is cross-platform across many microcontrollers. And it isn't bloated at all. In the end an I2C interface from manufacturer A doesn't do anything different compared to the one from manufacturer B.

Why would you waste your time making it compatible across manufacturers when you (by your own words) always use LPC?

I have not used LPC controllers always. There was a world before ARM microcontrollers existed. I've run my library on various microcontrollers including the 8051.

There are different situations. You may need to read I2C only at the beginning (such as memory SPD, or some sort of boot EEPROM), or you may have a dozen devices on the same I2C bus which you car read relatively fast, so that you need to master some kind of arbiter or queue to make sure your tasks don't interfere to each other. You may decide that you need to use interrupts or not. You may use DMA or not.  You may need to deal with 7-bit addresses vs 10-bit addresses. By the time you write the "driver" which covers all the situations, it will be

1. relatively big, representing substantial amount of bloat
2. difficult to use, because it is designed to cover huge number of situations
3. complex, which means potentially more buggy
4. expensive, because you spent time making sure it works for all situations, even the situations you will never encounter
5. useless, because all you need to do without it is just write few registers, which cannot really be simplified by much

You are way overcomplicating things. Why would a driver need to cover all cases at once? Just as long as you have a consistent API everything is OK.

[ehughes]

x = (1/sqrt(3)) * (y + 2*z);
d = Ia*cos(phi) + x*sin(phi);
q = x*cos(phi) - Ia*sin(phi);

Ha,   The first time I saw Field Orientied Control,  I wondered the same thing.    Why exactly do I need a large vendor library for this?


I used to do talks on DSP in embedded systems.  The 1st slide was that I can do DSP on anything.  An 8051 or an Abbacus.   

[SiliconWizard]

I do agree that quite often, software bloat and *excessive* use of third-party libraries is a sign of both laziness and lack of programming skills.

I'm all, as I often mention, for code reuse though, as much as reasonably possible. So every time that allows code reuse, using libraries is a benefit. Don't underestimate the time you may sometimes waste though, rather than save: as others have said, it may take you significant time to figure out how to use some library properly, and then some more if you ever run into bugs - and bugs are everywhere. And it's usually much easier to fix your own bugs than bugs in third-party code (assuming source code is provided - if not, don't expect most vendors to react quickly to bug reports). The reusability factor may also be hindered by nasty license issues, so beware of that as well.

General code reuse is a very vague idea. It makes sense within a company, sometimes even within some BUs in a company, but across every possible kind of projects/products/companies? Not necessarily so much. So, making your OWN code reusable (as much as possible, but not unreasonably either) is a good thing, but fully relying on third-party code for that doesn't always work, and is often a good excuse for not writing reusable/portable code yourself. Which I for one don't think is a good approach.

A "common" and universal API may sound like a great thing, but the road to hell is paved with good intentions. Ultimately it comes down to the typical idea of "the right tool for the job". If you want a relatively universal API and good low-level abstraction, use some OS. But that's not justified for small projects IMHO. nctnico mentioned network connectivity as an example of the benefits of a common API (or an OS as I said), but IMO it completely depends again on the project. If your product, apart from the network connectivity/internet access, wouldn't require any kind of OS or any high level of abstraction, unless you're going for very high volumes and very low cost, using third-party chips for the connectivity part is usually a better solution, with much faster TTM, fewer bugs and and a lot less frustration overall.


Anyway, to get back on topic, given the complexity of many modern MCUs, vendors have no choice but provide support libraries. I think they would lose a lot of potential customers if they didn't.
Now whether you or your team uses them for any particular project is entirely up to you, the known shortcomings and your company's policies.

[SilverSolder]

In practice, "code reuse" sometimes boils down to using some hardware (with embedded code) for some new purpose!

[SiliconWizard]

x = (1/sqrt(3)) * (y + 2*z);
d = Ia*cos(phi) + x*sin(phi);
q = x*cos(phi) - Ia*sin(phi);

Ha,   The first time I saw Field Orientied Control,  I wondered the same thing.    Why exactly do I need a large vendor library for this?

Yup.

But don't overestimate the math skills of a typical software developer. Absolutely not willing to start a flame war here, but just the result of my observations over the years. I have run into only a very few that would actually be able to fully understand the above.

[Siwastaja]

x = (1/sqrt(3)) * (y + 2*z);
d = Ia*cos(phi) + x*sin(phi);
q = x*cos(phi) - Ia*sin(phi);

Ha,   The first time I saw Field Orientied Control,  I wondered the same thing.    Why exactly do I need a large vendor library for this?

Yup.

But don't overestimate the math skills of a typical software developer. Absolutely not willing to start a flame war here, but just the result of my observations over the years. I have run into only a very few that would actually be able to fully understand the above.

Oh yes, for example I tend to be quite sucky with math, although I'm brave enough to at least partially blame the obfuscation built-in in both mathematics itself (300-year old notations do not always play well, and for someone who likes to talk in one's head, having ten different symbols pronounced the same and stylized in five different fonts to denote completely different things is nothing but obfuscation, when you could just use multi-character variable names like in programming), and especially in teaching material made by those who do not truly understand what they are teaching well enough to be able to teach it to a beginner. FOC is a great example of such obfuscation actually.

Still, as an embedded programmer, you absolutely must know how to implement a multiplication, summation, division, or sine function lookup. This should be basics. Dealing with numerical ranges in your code is something you can't avoid, except by buying the complete solution so that you do nothing.

So in any case, understand that math or not, the only sensible choice is to perform a few elementary school level math operations in your own code, even if you don't fully understand why they are there; compared to linking to a massive library and learning to use it to perform the same operations, which you still won't understand.

Can I get my name in history books for renaming bog-standard 2D vector rotation formula in my name whenever I come up with any application needing bog-standard 2D vector rotation? Some do have that privilege

I'm actually thinking it would be a nice joke to come up with a "Siwastaja transform" which would finally perform (x+y) through some large matrix operations. Then write it as a C++ library using every possible modern C++ feature available (and as many exceptions as possible); and a large bunch of dependencies to other libraries, obviously a matrix processing library at least. Make it a week-long process to install the specific versions of unstable library development packages and get it compile. Then add a ROS (Robot Operating System, the one that needs its own rcd, rls and so on command because cd, ls and so on are NIH) wrapper layer, again with similar complexity, so that users can easily perform said transform operations through the message bus. That would be great. I'm sure I could get people to actually use and promote it. If I could do it, that is. My C++ skills are greatly lacking.

[SiliconWizard]

Still, as an embedded programmer, you absolutely must know how to implement a multiplication, summation, division, or sine function lookup. This should be basics. Dealing with numerical ranges in your code is something you can't avoid, except by buying the complete solution so that you do nothing.

So in any case, understand that math or not, the only sensible choice is to perform a few elementary school level math operations in your own code, even if you don't fully understand why they are there; compared to linking to a massive library and learning to use it to perform the same operations, which you still won't understand.

I agree with that to some extent, but using some math you don't understand is not always the wisest thing to do. If you can't understand it, you may have a hard time judging whether this is the right thing to do in a particular context, to begin with. Or you may have found some resource that contains errors.

I'm actually thinking it would be a nice joke to come up with a "Siwastaja transform" which would finally perform (x+y) through some large matrix operations. Then write it as a C++ library using every possible modern C++ feature available (and as many exceptions as possible); and a large bunch of dependencies to other libraries, obviously a matrix processing library at least. Make it a week-long process to install the specific versions of unstable library development packages and get it compile. Then add a ROS (Robot Operating System, the one that needs its own rcd, rls and so on command because cd, ls and so on are NIH) wrapper layer, again with similar complexity, so that users can easily perform said transform operations through the message bus. That would be great. I'm sure I could get people to actually use and promote it.

Good one!

[IDEngineer]

I'm actually thinking it would be a nice joke to come up with a "Siwastaja transform" which would finally perform (x+y) through some large matrix operations. Then write it as a C++ library using every possible modern C++ feature available (and as many exceptions as possible); and a large bunch of dependencies to other libraries, obviously a matrix processing library at least. Make it a week-long process to install the specific versions of unstable library development packages and get it compile. Then add a ROS (Robot Operating System, the one that needs its own rcd, rls and so on command because cd, ls and so on are NIH) wrapper layer, again with similar complexity, so that users can easily perform said transform operations through the message bus. That would be great. I'm sure I could get people to actually use and promote it.

That sounds sadly familiar. Pretty sure I've already used that library! 

[NorthGuy]

You are way overcomplicating things. Why would a driver need to cover all cases at once? Just as long as you have a consistent API everything is OK.

If you only have one use case for I2C, you don't. If you have 20 different use cases, your driver must cover them all. Otherwise, you would be doing the same as I do - write a separate code for each use case.

If you a hardware vendor though, you may have millions of users, each of them with their own use cases, you you simply have to cover them all. Otherwise, your users won't be able to use your library and will complain - your library cannot do this, your library cannot do that. In practice, they create libraries which are both complicated and at the same time do not cover the case which you need at the moment - the library doesn't work with interrupts, the library requires RTOS etc.

[SilverSolder]

You are way overcomplicating things. Why would a driver need to cover all cases at once? Just as long as you have a consistent API everything is OK.

If you only have one use case for I2C, you don't. If you have 20 different use cases, your driver must cover them all. Otherwise, you would be doing the same as I do - write a separate code for each use case.

If you a hardware vendor though, you may have millions of users, each of them with their own use cases, you you simply have to cover them all. Otherwise, your users won't be able to use your library and will complain - your library cannot do this, your library cannot do that. In practice, they create libraries which are both complicated and at the same time do not cover the case which you need at the moment - the library doesn't work with interrupts, the library requires RTOS etc.

There are good reasons not to use a library in some cases.  E.g. I see Oracle developers rewrite queries with small variations all the time (rather than put them in a library in a more generic form) for performance reasons (i.e. using "tricks" that they know Oracle "likes") on large (read: slow!) databases.   In the embedded world, there are likely situations where you are looking for the last ounce of performance too.

[Siwastaja]

In my experience, I2C implementations in microcontrollers vary quite a bit, the level of their usefulness and brokenness, especially.

Some require so much register configuration and polling during a simple transfer that you are wasting almost as much CPU time as bit-banging the whole bus.

Some are so buggy and full of corner cases that you fight for days getting them work.

Couple that with some I2C devices having a buggy interface.

A generic driver for the most common microcontrollers, with stable API, sidestepping all these issues sounds great on paper, but I guess it's going to be almost impossible to do properly, and I doubt we'll ever see it.

I2C, though, is most often used to just set some configuration registers up or read some super-slow sensor at whatever rate, doesn't matter. A simple write-once "library" you can reuse yourself, that simply bitbangs the whole I2C without even trying to use the MCU peripherals, configurable for the specific GPIO access patterns with a few #defines, would likely hit the nail 95% of the time.

[bson]

The real problem is the libraries don't relieve you of the need to thoroughly read user guides and datasheets.  For example, if you want to set up a DMA channel to repeat a pattern on a port, you need to make sure this can be done, how it's done in the hardware, and for example if there are channel limitations, data size limitations, if it needs to be in a specific SRAM bank, whether it can be in NOR flash, etc etc.  Then timing, which clock base to use, how to configure the clock tree and timer channel, to trigger the DMA.  And, now, having read the hardware docs and having a clear notion of how to use it, you're stuck figuring out how to make a library that attempts to hide all these details behind an abstract interface do it.  You almost invariable end up having to reverse engineer it from the source code.  In short, you do about 5x as much work for no benefit, with tons of hairy debugging because the library abstractions are all intertwined in complicated ways.  That would be fine if it actually got you something in terms of portability and performance, but portability is STILL completely tied to the actual hardware and isn't just going to work on anything with the right functionality - it has to be EXACTLY the same.  And performance often suffers as the abstractions add tons of vectoring and sometimes totally bizarre stuff that looks like a first-year student's programming project.

[thm_w]

Well, the "library" approach is for people who simply cannot write even a few lines of code, but still somehow manage to get something done by massive code copy-pasting and library call operations. I truly wonder how it's possible to build anything like that.

Case in point, for motor control, when you need to calculate exactly these, and only these, operations:

x = (1/sqrt(3)) * (y + 2*z);
d = Ia*cos(phi) + x*sin(phi);
q = x*cos(phi) - Ia*sin(phi);

You are using an equation which forms a portion of the functionality of the library.
The library is massive because it supports other features that your three lines of code do not cover.

Closed source libs sure agreed, don't use those. I'm not aware of any STM32 libs that are closed, but, I'm sure they might exist.

The real problem is the libraries don't relieve you of the need to thoroughly read user guides and datasheets. 

Well, they do if you're just working on a basic project.
Call some "AnalogRead" function every 1s, and the chance that you would run into any issues is very low. All this without reading 65 pages on the ADC in the ref manual.

[Nominal Animal]

Most library interfaces are pretty much shit.

It is not because their writers are bad programmers, it is because when you design a library, you almost never know how it ends up being used.  It's like creating a tool, and then hoping it will be useful for something.

As an example, consider that analogRead() function.  Almost all ADC hardware nowadays works independently of the microcontroller processor itself, so actually there are three separate stages: setup (source/mux, gain, sampling options), trigger, and sometime later, a state change in some hardware register indicating the conversion is completed. analogRead() combines all three, with a busy-wait loop waiting for the conversion to complete; during which time the microcontroller processor could do other useful work, like scale/sum/square (for statistical details) the previous converted sample.

This is excarberated by vendor libraries having partial design goals opposite to user-developer needs.  One of them is the need to retain development effort within the vendor product: you don't want to make it easier for your clients to switch to your competitors products.  Another is that the vendor has little experience in the variance of how different use cases their products may be applied to.  (Sure, they'll sometimes work closely with their largest clients, but for every large client you have a hundred small clients, in todays world.  If you only work for companies who have that sort of ties to vendors, your experience is probably different.)
A yet another is that the vendor libraries are usually seen as a cost, and thus created and maintained with strict development resource limits, making them quite poor quality overall.  As the vendor portfolio grows, the complexity in the library increases, making them worse as time goes on, not better.

So, if you want code that works best for your own use cases, you'll have to develop or help develop the libraries yourself, in my experience.

None of this is unique or specific to embedded environments.  High-performance computing suffers from the exact same kind of issues.  If you think about it, even simple read()/write() interfaces could use splitting into submit/completed stages, allowing the thread to do useful work in between.  But for humans, this kind of async thinking seems to be too hard.

[SilverSolder]

[...]
  If you think about it, even simple read()/write() interfaces could use splitting into submit/completed stages, allowing the thread to do useful work in between.  But for humans, this kind of async thinking seems to be too hard.

In fairness, many Read/Write paradigm interfaces do come with Open(), Flush(), etc. commands...   which means the programmer has to think a little more, in return for an efficiency gain.

[Nominal Animal]

[...]
  If you think about it, even simple read()/write() interfaces could use splitting into submit/completed stages, allowing the thread to do useful work in between.  But for humans, this kind of async thinking seems to be too hard.

In fairness, many Read/Write paradigm interfaces do come with Open(), Flush(), etc. commands...   which means the programmer has to think a little more, in return for an efficiency gain.

No, I meant that when DMA is used or hardware uses latches/queues etc. to transfer blocks of data, it is silly to not have a native asynchronous interface where the call only triggers the transfer, and another call can be used to check or wait for completion.  Having library calls do busy-wait loops or loop until a hardware flag changes is a significant waste of energy.

This is particularly useful with networked/distributed computation (e.g. MPI_Isend() and MPI_Irecv()), but rarely used in HPC because async=hard, and it is easier to put more money in hardware and use blocking calls instead.

[SilverSolder]

[...]
  If you think about it, even simple read()/write() interfaces could use splitting into submit/completed stages, allowing the thread to do useful work in between.  But for humans, this kind of async thinking seems to be too hard.

In fairness, many Read/Write paradigm interfaces do come with Open(), Flush(), etc. commands...   which means the programmer has to think a little more, in return for an efficiency gain.

No, I meant that when DMA is used or hardware uses latches/queues etc. to transfer blocks of data, it is silly to not have a native asynchronous interface where the call only triggers the transfer, and another call can be used to check or wait for completion.  Having library calls do busy-wait loops or loop until a hardware flag changes is a significant waste of energy.

This is particularly useful with networked/distributed computation (e.g. MPI_Isend() and MPI_Irecv()), but rarely used in HPC because async=hard, and it is easier to put more money in hardware and use blocking calls instead.

Ah, I see what you mean.   -  yes, non-blocking calls are a total pain in the *rse for simple program flows.  The extra work needed to use a non blocking call is only worth it if you have to have the efficiency...  so we end up with these "lazy programmer" implementations that include the waiting, for the convenience and simplicity.

[station240]

I've had 'fun' with one library for TI's DSP micros.
One communications library has a whole set of filters for incoming traffic bitmasks and stuff.
While it's good at filtering out errors and data you elected not the receive, when things go wrong, there is no way to read any of the errors as they have already been wiped from the buffer by the library.
It's as if someone wrote the library as an example of how to do everything you need, then hid it inside a private class so you cannot access anything.

One day I should take that library, and separate the protocol code and the filters/error handler code into separate functions.

[Kasper]

Years ago I took a course on making portable device drivers in C.  I enjoyed the idea of separating the MCU specific details from the rest of the code and dreamt of the day I had a large repo of clean and solid code modules of my own to be dispersed as needed.

Fast forward to last week, I make a little IOT prototype for a software / networking guy who has his own business and seems to know what he's doing.  He requests I just make half of the hardware and write minimal FW just for testing.  He buys the other half of the HW (4-20mA to I2C converter) from a website for $40 usd.  I get 2 boards running, show him the button and LEDs work and leave it with him.  Later he invites me over to help troubleshoot, my LED 'doesnt work' and he has other errors.

I expected he would add my LED code to the arduino library supplied by the vendor.  He didn't, he just used their library and complained my LED was acting weird.  Back to the topic: the libraries supplied by the vendor was just a link to some random github example for the 4 channel ADC on their board.  The board was built using 2 channels single ended. But the supplied library printed all 4 channels and had different versions for single or differential.  My client was scratching his head, trying to figure out what code to use, what channel to look at and expecting something more refined than a raw ADC value.  I don't think he even glanced at the libraries.  The maker site didn't even bother to cut out the 2 channels they aren't using or the differential code.  They didn't supply schematic either so I ended up reverse engineering the board to figure out how the ADC was connected and what mods to do to make it work for our application.  It was not compatible with the 4-20 sensor he wanted to use.

The best part, their hardware sucked too: resistor networks very close to PCB edge despite having lots of spare room. One resistor was not thermally balanced and ended up skewed and not soldered. 

In this case, the vendor supplied library and the vendor supplied hardware both sucked and I enjoyed telling my client he should just have me do everything next time.

[Bassman59]

I've had 'fun' with one library for TI's DSP micros.
One communications library has a whole set of filters for incoming traffic bitmasks and stuff.
While it's good at filtering out errors and data you elected not the receive, when things go wrong, there is no way to read any of the errors as they have already been wiped from the buffer by the library.
It's as if someone wrote the library as an example of how to do everything you need, then hid it inside a private class so you cannot access anything.

One day I should take that library, and separate the protocol code and the filters/error handler code into separate functions.

I'm having "fun" with TI's USB device library for one of their ARM families. It is a mess of abstractions and pointers pointing to pointers to structures, and structures of pointers pointing to ... well, who the fuck knows. Their code "works" for the device classes their support. But writing support for a class they don't is mind bending, because following what was done on a "similar" class is painful. Their "goal" as such was to provide a "driver" your "Application code" can hook into, but I'm trying to write the "driver" and I don't need the extra shit they piled into their code.

I got the thing to enumerate, with an LED that lights up in response to the set-configuration request and goes out with a disconnect. And I made a change -- dunno what, exactly -- to add some more of the design functionality and now the interrupt for device disconnect no longer gets asserted. It's arsed, as they say across the pond.

Otherwise, their TivaWare library is straightforward enough and usable. And it's also burned into ROM in the chip, saving flash space.

Hilariously, the NXP USB library for LPC55xx devices is worse. It's opaque. And their support people say, "Start with an existing class design and modify it to suit."

I will say that Silicon Labs USB stack is a lot easier to get working. It's just that they don't have any plans to do High Speed USB, so ... blecccch.

[Sal Ammoniac]

It is a mess of abstractions and pointers pointing to pointers to structures, and structures of pointers pointing to ... well, who the fuck knows.

That's the major issue I have with vendor libraries. Ever try to debug that crap? You spend all of your time following pointer after pointer until you finally get to code that actually does something. Keeping track of all that abstraction is distracting and annoying. 

[Bassman59]

It is a mess of abstractions and pointers pointing to pointers to structures, and structures of pointers pointing to ... well, who the fuck knows.

That's the major issue I have with vendor libraries. Ever try to debug that crap? You spend all of your time following pointer after pointer until you finally get to code that actually does something. Keeping track of all that abstraction is distracting and annoying. 

This is worse, because it's an unholy mess of globals, static globals and then those things are passed as arguments in functions. Object disoriented programming.

One of the TI support people who deals with this parts says, "I wouldn't have written it that way," which is one way of admitting that the library is a disaster without saying such.

[dietert1]

Yesterday i unpacked a ATSAMC21 explained pro kit to start development of a simple adapter board that is to conver/transmit I2C messages from a Sensirion SHT35 sensor and SPI data to a Can bus. So i spent about 20 hours on that and the application layer still needs to be refactored.
Without using Atmel Software Framework that wouldn't have been possible. I can tell you that i got furious yesterday when i lost about four or five hours trying to import the generated project into IAR. For the time being i gave up, because i got hard faults.

I mean this is what it is. Software work isn't fun unless you are free to follow your fantasies of self written HAL layers and the like. It's a lot of searching, trying to understand to some level what others did before.

Regards, Dieter

[Sal Ammoniac]

I mean this is what it is. Software work isn't fun unless you are free to follow your fantasies of self written HAL layers and the like.

Sure, if your goal is a quick and dirty project (or, heaven forbid, product), and you don't have time to do it right, then using vendor libraries may be a time saver. In my opinion, a well-engineered product practically demands the use of custom-written peripheral drivers.