Linux Dependency Black Hole

[mag_therm]

I used  linux in engineering for 15 years , now I am hobby user in ham radio.
When installing applications, I recall only a few issues in prior years, but after about 2020, the dependency issues started to escalate.

This year I have been installing Software Defined Radio applications with lots of dependency problems, and this week I had to abandon attempts to install
the NXP MCUX IDE on Fedora due to inability to resolve wrong/missing dependencies. And over on the HamRadio section, Metrologist has just documented the output while tryng to install a ham radio application on a debian Pi.
Engineering software for industrial use needs a lifetime of over 15 years, Similarly ham radio sdr software followed the surge of hardware development back around 2014, still current.

I would like to ask a few questions to improve  understanding of how and why dependencies became a problem for users trying to install applications.

1A). When a dependency itself is updated,is it normal/allowable for the original/prior functionality af that utility to be removed?
   That is if a function was previously called with a set of args, and then gives an output, why can't that functionality be retained "forever" along
    with the new additions which come out with a new version number appended to the dependency filename. ?
   Examples that do work "forever" are like the bash utilities, and sqlite stty etc. Why can't the libs be more like that?
   If back compatibility could be made a rule, it would take away 80% of dependency failures, in my experience.

1B). Conversely why does the app installer not throw a warning and proceed anyway to use the updated lib that it finds ?
   In my experience on Fedora , rolling back a dependency is difficult, and also can disable applications already installed.

2).Sometimes when installing a missing dependency, I see it is just a few kB or MB. Why would the App developer not include such small functions into the app?

3). Python3. I am fairly sure that Python 3 versions have always retained ability to run deprecated functions. So why would an app want a specific Python3?

4). Paths: I saw on Fedora that system paths do change over the years. The simple way for a user to fix a "can't find" would be to find and export the PATH or lib LD..  path.
Why can't the app installer be doing a check of the system paths before throwing a "can't find"?
   An example was that NXP app MCUXpresso say just "Can't find Python3.8" doesn't say the path it used. When I installed it in ~/.penv and exported that       path, still ""Can't find Python3.8"

 The way this is going, I can see that soon, we will have to install apps on their dedicated complete o/.s
 And that is what I am now trying to be able to solve present problem.

[ataradov]

1A). When a dependency itself is updated,is it normal/allowable for the original/prior functionality af that utility to be removed?

If the major version of the library changes, then entire API may be different.

A good example here is libusb. Version 1 and version 2 of the library use entirely different approach to the API. They did not change it "just because", they changed it to support higher performance.
Maintaining APIs forever is a lot of work and nobody wants to do it for free.

   If back compatibility could be made a rule, it would take away 80% of dependency failures, in my experience.

Are you willing to pay for it?

1B). Conversely why does the app installer not throw a warning and proceed anyway to use the updated lib that it finds ?
   In my experience on Fedora , rolling back a dependency is difficult, and also can disable applications already installed.

Major versions of the libraries are usually not installed automatically. And multiple major versions can usually coexist. Minor versions rarely break APIs. And when they do, it is not on purpose.

Sometimes things break because application authors used undocumented APIs. And this is on the application authors.

2).Sometimes when installing a missing dependency, I see it is just a few kB or MB. Why would the App developer not include such small functions into the app?

Because then you will have to maintain the code. Plus sometimes there are license restrictions.

3). Python3. I am fairly sure that Python 3 versions have always retained ability to run deprecated functions. So why would an app want a specific Python3?

Likely because the app is written poorly.

   An example was that NXP app MCUXpresso say just "Can't find Python3.8"

MCUXpresso is a cost center for NXP, not a profit center. It gets corresponding amount of attention. All vendor tools from all IC vendors are bad. They cost money and don't directly contribute to profits. As long as they mostly work, it is fine. FPGA IDEs are some of the worst software ever created. Yet there are no alternatives and what are you going to do? Not use that FPGA?

Your questions are too abstract and actual details would not be universal and would depend on the specific cases.

[coppice]

The current version of the ITU G.1050 spec is at https://www.itu.int/rec/T-REC-G.1050-201607-I/en . Its from 2016. Not exactly new, but not exactly ancient. Try using the software provided there. It requires numerous things, mostly open source, but it requires specific revisions of many of those things. It looks like within 2 years of the publication of that spec you couldn't find some of the necessary versions on the internet, and now you can find almost none. So much for the notion that with open source you just publish and somewhere on the internet it will be effectively archived.

Demanding too much backwards compatibility is a bad thing. You learn a lot about how to design something well by getting a first weak pass out the door. If you can't be incompatible you are locking in a lousy design forever. You do need to keep the old stuff relevant for a very long time, though. ataradov referenced libusb totally breaking compatibility. gstreamer and other software has been through the same thing. Some of those things struggled to move people on to the better solution, but that did discipline them to keep the old version usable and relevant. Many people are too keen to let the old stuff rot when they are able to carry a good number of people forward to the new version.

[ataradov]

I find it hard to complain and demand people keep the old stuff up to date given that all those things are developed for free. Bigger projects like Gstreamer have some monetary support, but smaller libraries don't. Even though they are often at the foundation of everything. And even when there is monetary support, people allocating the money usually have some influence on how it is spent, and rarely there is incentive to spend it on keeping the old stuff up to date.

If anything, a more reasonable demand is to have the applications that use those libraries be kept up to date. It is strange to ask authors of the libraries to do more work because you don't want to do the work yourself.

[mag_therm]

Thanks ataradov and coppice
Yes I understand this is all FOSS. I support linux and FOSS and believe the concept has been of significant benefit over the years.
DSP is an outstanding example in my area of interest, another is image processing.

I am pointing out that the problems with linux dependencies is getting worse, to the level of "reasonable" users not being able to install,
and asking why.

[Zucca]

The current version of the ITU G.1050 spec is at https://www.itu.int/rec/T-REC-G.1050-201607-I/en . Its from 2016

would be worth to try it with some old Linux distro from 2016 in a VM?

[ataradov]

It depends on what you are installing. Most libraries and software in the distribution repositories are usually well matched, since package maintainers do that work.

If you are installing some random software you download in *.tar.gz, then you are on your own. If you are installing some proprietary software, you need to make sure your system matches the requirements.  Usually those things target recent version of mainstream distributions.

I would not say it is getting worse, if anything, it is getting better. But for things to get better, incompatibilities are introduced, so all the old software that was not updated in decades is less compatible. The new software benefits from improvements though.

Having things improve while staying the same is the perfect example of incompatible requirements.

[coppice]

If anything, a more reasonable demand is to have the applications that use those libraries be kept up to date. It is strange to ask authors of the libraries to do more work because you don't want to do the work yourself.

I'd be happy if they just kept the old releases available, even if various bit rot issues stop them compiling out of the box with other current stuff. I would probably have got the G.1050 code going if they had. As it was, the number of issues I would have needed to sort out looked overwhelming.

I recently had reason to want to build a 20 year old version of GCC. That was easily available, but various other things of the same vintage were not, and the effort to build it was just too much. So, I got a 20 year old installer for Fedora that contained the version of GCC I wanted (Fedora 3 I think), and tried to install that on a spare machine. It seems that was the changeover from IDE to SATA, and I couldn't find either old IDE based hardware or old SATA based hardware where the disk drivers in that version of Fedora were happy. I gave up on that too.

[coppice]

The current version of the ITU G.1050 spec is at https://www.itu.int/rec/T-REC-G.1050-201607-I/en . Its from 2016

would be worth to try it with some old Linux distro from 2016 in a VM?

They require a bunch of things not in the distros, and the relevant old revisions of some of those things are no longer available on line.

[mag_therm]

It depends on what you are installing. Most libraries and software in the distribution repositories are usually well matched, since package maintainers do that work.

If you are installing some random software you download in *.tar.gz, then you are on your own. If you are installing some proprietary software, you need to make sure your system matches the requirements.  Usually those things target recent version of mainstream distributions.

I would not say it is getting worse, if anything, it is getting better. But for things to get better, incompatibilities are introduced, so all the old software that was not updated in decades is less compatible. The new software benefits from improvements though.

Having things improve while staying the same is the perfect example of incompatible requirements.

My experience over 18 years is that the Fedora dependency problem has become not better, a lot worse since about 2020, but that could be partly  in my usage changing from day job to ham radio
And yes,the packages installed from repositories by dnf usually had commendably few or no dependency problems

[xvr]

More and more developers use Docker for distribution of these products. This could be a solution for dependency problem.

[mag_therm]

More and more developers use Docker for distribution of these products. This could be a solution for dependency problem.

I have not had an app so far  which could have used Docker. I have installed two of .AppImage. These were with  no problems, also I read that AppImage will work across all linux variants too.
maybe that is the way forward, but not many available yet.

[DavidAlfa]

Ahh linux. Yep. FOSS heaven (Fu**ed Open Source Software).
Last day I removed OpenOCD to manually install a newer one. Got asked to remove no longer necessary packages. Sure!

It completely uninstalled my desktop, leaving me dead in the shell.
Sadly (Or luckily) this already happened few years back, so I quickly checked if gnome had been eradicated and got it back within 10 minutes.

Sorry I can't answer your questions. So many "whys" on my head too. So freaking no sense in so many ways.
Basics - nice. Anything else - pain and time wasting.
Maybe some people love browsing forums for 2 hours everytime they need to fix or make something, I don't.
I guess that's a golden-egg-laying chicken in IT, it's a maintenance hell, hours and hours, $$$.

[DimitriP]

More and more developers use Docker for distribution of these products. This could be a solution for dependency problem.

Depending on docker to avoid depenency issues might be an issue depending on ...docker dependencies...

[mag_therm]

I ran a grep on an application tar.gz to see how the system dependencies are defined.
here is an snippet from the valgrind file  in the tar.gz :

QQ-valgrind.linux.supp:148:   obj:/usr/lib/*/libpulse*
QQ-valgrind.linux.supp:169:   obj:/usr/lib/*/libQt5Gui*
QQ-valgrind.linux.supp:182:   obj:/usr/lib/*/libfontconfig*.so.1.8.0
QQ-valgrind.linux.supp:265:   obj:/usr/lib/*/libfreetype*
QQ-valgrind.linux.supp:272:   obj:*/lib/*/libdbus*
QQ-valgrind.linux.supp:279:   obj:/usr/lib/*/libcairo*

So it seems in the tar.gz, dependency versions are not explicit, and it seems they would be from the whatever system that the tar.gz was made on.
The user who built from tar.gz would not have dependency version problems.

Then opened the .rpm (for Fedora )
Here the files are mostly .bin  and there is a /bin/valgrind on linux.
I don't know if valgrind is run again on user machine  during installation.
I suspect not because we have the dependency version problem.

So we may be better, if having a big dependency mismatch,  to build from tar.gz rather than trying to use the .rpm .deb etc packages.
Can anybody familiar with this comment?
Thanks !

[ataradov]

The user who built from tar.gz would not have dependency version problems.

This is not correct in general. There is no common way to define dependencies in tar.gz outside of autotools stuff, which rarely covers everything. It is mostly just basic checks.

You will have all sorts of issue if you build stuff from the source and let it be installed in your system outside of the package manager.

So we may be better, if having a big dependency mismatch,  to build from tar.gz rather than trying to use the .rpm .deb etc packages.

This is the worst idea ever. If I absolutely need to install something from the source, I at the very least specify --prefix=/opt, so it does not mess up the main file system. And then you still rely on package author not being an idiot, which is not a given.

[mag_therm]

Well I have built stuff from tar.gz, in fact recently too on two for the SDR.
Always in the ~ as I recall.
I presume many others around the world have too.

And please don't take all this so emotionally, its only a hobby for me.

[ataradov]

I don't take it emotionally. I don't really care what you do to your system. But what bothers me is when people go on the forums and complain that "Linux sux" when they mangled their system by installing random stuff from tar.gz.

Installing into "~" is fine. Installing into "/" is just asking for trouble.

But it also depends on the type of a system. If this is some random RPi that can be re-imaged when necessary, then who cares. If this is your primary system, not destroying it is a good call.

[linux-works]

appImage is a large image but it WORKS and that's great. for many products, I do just take the easy way out and do the app image thing.  cura is one of them and open-scad is another.  both too large and hard to build/install so I just take the appimage route.  its not ideal but it works.

[mag_therm]

I don't take it emotionally. I don't really care what you do to your system. But what bothers me is when people go on the forums and complain that "Linux sux" when they mangled their system by installing random stuff from tar.gz.

Installing into "~" is fine. Installing into "/" is just asking for trouble.

But it also depends on the type of a system. If this is some random RPi that can be re-imaged when necessary, then who cares. If this is your primary system, not destroying it is a good call.

As I wrote earlier, I am staunch linux user of 18 years, having earned my living from it and I like messing about with it now in ham radio.
from my o/p, see, I am not complaining about linux; I am asking questions about the increasing problem of dependencies.

I am retired with a few left over computers .
For risky trial stuff like trying this MCUXpresso I use a standby computer on Fedora 37 now.
On that I don't care if reloading Fedora as I like to keep it up to date

Linux is fairly fault tolerant in my experience, but when clients would ask to fix problems, I would always recommend to just reload the whole thing.
Because it is difficult or impossible to find out what somebody else did especially if they had root.

[ataradov]

I am asking questions about the increasing problem of dependencies.

You have not actually clarified what "problem of dependencies" you have in mind. I've been using Linux since 2002, and as the only OS since 2007, and in that time I only see things improving over time (discounting whatever Canonical is doing).

So, if you can provide specific examples, there may be something to discuss. Otherwise it is just way too abstract.

[tatel]

If you don't want to get into dependency hell, just stick to apt and what is in the repositories for the release you installed. Package maintainers did that work for you. To make a package for Debian implies you have full knowledge about all the Debian policies, etc, and to learn it means a lot of work.

I also think things are getting worse. The reason are all those pesky new/better/wonderful ways to install software... that are full of shit. They don't want to learn how to create debian packages, or think they will make some money going that way, or fame/glory/whatever, so they have to invent the wheel again and again. But, while apt is almost the perfect wheel, those new wheels, well, not so much.

AppImage can work. I don't like it however. Docker could be useful if you have servers and want/need to migrate not just the software, but also the contents, from one server to another. What I can't swallow is pip. Use pip to install any python thing, and odds are things will be messed up sooner rather than later, because you have now python things installed, unknown to apt. Pip would like to be half the good apt is. And you'll probably will forget you installed that thing with pip. Then a rough ride will begin.

tar.gz packages? To me, that means source code to be built. If you do so, you'll need to learn previously about the dependencies, and build that dependencies first. And the dependencies of the dependencies. And so on an on... until you have built all what's needed. All should go into a separate partition. There used to be a /local directory just for that partition to be mounted there.  That way you can have different versions of libraries, etc, needed to do the work, and those binaries can be easily preserved if you want to upgrade the rest of the system. I used to do that only for versions of software with new features/fixes that I needed, and I get it back to be installed with apt as soon as the corresponding debian package for "stable" is released.

But, often, the main reason is just user ignorance and unwillingness to read any documentation. Debian has quite good guides, and anyone that did read about the sources.list file, would know that metrologist, in that other thread, is getting messages about hits from two different repositories, one of them for bookworm, the other one for jessie. So, no wonder he got a dependency hell. He just messed things, first, choosing to do an upgrade when the easiest way, not having anything functional on that system, would have been to go for a clean install. Then he modified /etc/apt/sources.list without knowing what he was doing (because he couldn't be bothered to read the guide). Then he complains "Linux sucks". LOL.

Easiest solution: clean install with just the defaults. After that, use just APT to install any packages. If some software couldn't be installed with APT, it just means that software isn't mature enough, and it should be avoided like the plague, until you know what you are doing.

Edit: if you are going to do a Debian install downloading the packages from Internet, just use an ethernet cable. Wifi won't work... because the firmware for the wifi hardware isn't free. You'll have to install that firmware with apt after install is done. It's just the Debian way, and Canonical profiteered from that greatly. Again, anyone that did read the guide would know it... he would even know that an image including non-free firmware, with kernel modules for wifi hardware, is also available, but very few people seems to be willing to read anything nowadays

[mag_therm]

Thanks atardov and tatel
Here is what I propose to do for my problem which is NXP MCUXpresso:

The usb installer from linuxshoponline for Debian 12.4 is due to arrive soon.
I will buy another ssd sata today , then dd /dev/zero it using fedora, then  set it up in rack as the only media
install debian per installer instruction on the usb
install pyenv which I understand is only available from github as  ~./pyenv
Add LD_LIBRARY_PATH pointing to ~./pyenv...python3.8

then start install on debian 12.4  of the MCUXpressoide....deb.bin   from NXP . I will try to find any NXP instructions for that
If significant problems I will start new thread on eevblog with scrots etc  requesting assistance.

[xvr]

> I will try to find any NXP instructions for that

This is not a problem with bad Linux compatibility design, but problem with bad customer support in NXP

I remember some old story with some kind of software thing from Siemence for programming their hardware (now I don’t remember what kind of hardware it was)
Some company bought a lot of hardware, and when they started programming, a problem arose - no one in the company could install the software package provided by Siemence (it was Windows, by the way).
After several days of unsuccessful battles with installers, a field support engineer from Siemens is called. It comes with a laptop with the software installed, just to show how easy it is to do
They ask him to install software on their computer. He started and give up after 5 hours of unsuccessful attempts.
Then the team lead (after consultation with Siemence headquarters) makes a decision - he confiscates the field engineer’s laptop with the software installed and says that he can take it back after the software will works on their server.

[mag_therm]

Hi xvr,
Yes I intend  contact NXP if I have any problems with Debian 12.4 installation as that is its intended o/s.
Customer ? well I'm retired ham who bought a $40 Dev Board, that will carry some weight!

I used to program Siemens PLCs a long time ago. I regarded those PLC and the "IDE" as better, more advanced than the USA & Japan PLCs.

Any alternative for IDE on LPCX4367 if I have no luck with the MCUXpresso ?