Replacement for C standard library: your wishlist?

[SiliconWizard]

Apart from replacing unsafe functions (even though a number of them have already been deprecated), I'd like to see standard hash tables/dictionaries. And, as you mentioned, pool allocation functions.

[Nominal Animal]

I take it that you are fully aware of the challenges and the problems. So ... that would actually entail that you go the extra mile and don't use C. If you have good ideas, look into the future, not the past.

Why do you insist D, Rust, and Go are "the future", and C is "the past"?  No, I am not interested in how convinced you are, or who else says so; I am interested in logical arguments for or against, or personal experiences.

Like I said, I don't want more abstractions.  D++ is object-oriented, and I don't need that abstraction.  I don't want a runtime like Go has, either; if you have ever dealt with C++ runtime from other languages, you know why.  Perhaps Go avoids that particular trap, but that is yet to be seen.  I want an efficient low-level systems programming language, and that is what C is.  No, C is not particularly good.  A big part of not-particularly-good is the standard library, easily shown by comparing programs written with POSIX C to those doing the same thing in standard C, so I want to find out how much better we could do.  Rust is the closest, but I'm not convinced it is better than C yet, and I don't see how I could contribute anything meaningful there.

As to "better", I intend to have real-world compilable code that everyone can look at and compare to, and tell me how and why I'm wrong.  That will be useful.

I know that actually a different programming language would be the real painkiller.

Yes, that I do agree with.  Perhaps, one day, I will take that step, too.

But for now, freestanding C provides an easy starting point.  I intend to create the library, then implement a few typical service daemons, so that one can compare them to similar daemons written in C and other languages.  Instead of abstract examples or handwaving, I want tangible, real-world code to examine and compare to.  (And not just compare the human-readable parts, but also the binaries generated, and definitely the run-time resource use.)

I think I've missed something. What is a hosted or freestanding C environment, or, rather, the difference?

In a hosted environment, the standard C library provides you a number of functions, from malloc() to exit().  In a freestanding environment, you do not; you only have some macros (from specific header files you can still include) and variadic parameter support (from <stdarg.h>).

I'd like to see standard hash tables/dictionaries.

Environment variables are an example of a dictionary every process has access to, and I have an idea on those.

Do you have any examples of interfaces you've found useful?  Function prototypes would give a good idea, with example real-world use cases.
Compare to e.g. qsort_r() instead of qsort() for sorting: the comparison operation often needs external information, like offset or column within the string to start the comparison at, and passing an untyped user pointer to the comparison function makes that easy in a thread-safe manner, as one does not need to use global variables.

And everyone else, if you have found a particularly useful function you have used when writing low-level code – a service daemon, a command-line utility, etc. – please let me know; the most interesting bit is how/why you found it useful.

[PlainName]

In a hosted environment, the standard C library provides you a number of functions, from malloc() to exit().  In a freestanding environment, you do not; you only have some macros (from specific header files you can still include) and variadic parameter support (from <stdarg.h>).

OK, thanks. First time I've come across that distinction. Well, those names for it anyway.

[Fixpoint]

I take it that you are fully aware of the challenges and the problems. So ... that would actually entail that you go the extra mile and don't use C. If you have good ideas, look into the future, not the past.

Why do you insist D, Rust, and Go are "the future", and C is "the past"?

Please stick precisely to what the other one says, otherwise we will make no progress. I never said that D, Rust or Go are the future. I gave those languages as examples for new languages that have been developed and that fix severe flaws of C. (And I repeat what I said earlier: I explicitly do not recommend Go.) I do not say that those languages are the future. I even hope that they ARE NOT.

No, I am not interested in how convinced you are, or who else says so; I am interested in logical arguments for or against, or personal experiences.

What kind of anti-intellectual stance is that? There is actual RESEARCH available, research that provides all those "logical arguments", research that has been done for DECADES by many people in the mathematics and CS community, some of them even clever. And you tell me "no, I am not interested what others have to say". Well -- what do you expect from me? How am I supposed to react to this? Clap my hands? Do you think saying something like that makes you a free thinker? Honestly, I don't know what to make of this.

Don't get me wrong: If you are SERIOUSLY interested in this stuff, I am definitely willing to talk about it. But currently I am under the impression that that's not really the case.

By the way, object-orientation has nothing to do with all this. D (the name is D, not D++) SUPPORTS object orientation, but that does not mean that--

No, I'm sorry. I am already beginnnig to give a lecture on the 101. I shouldn't do that.

[Nominal Animal]

I take it that you are fully aware of the challenges and the problems. So ... that would actually entail that you go the extra mile and don't use C. If you have good ideas, look into the future, not the past.

Why do you insist D, Rust, and Go are "the future", and C is "the past"?

Please stick precisely to what the other one says, otherwise we will make no progress. I never said that D, Rust or Go are the future. I gave those languages as examples for new languages that have been developed and that fix severe flaws of C. (And I repeat what I said earlier: I explicitly do not recommend Go.) I do not say that those languages are the future. I even hope that they ARE NOT.

You wrote,

During the last decade, much better options than C have been developed (including the embedded use case), for example D, Rust, or Go (I don't recommend Go, but of course I have to include it here).

and later,

I take it that you are fully aware of the challenges and the problems. So ... that would actually entail that you go the extra mile and don't use C. If you have good ideas, look into the future, not the past.

I'm sorry, but I cannot understand how else I should interpret these except as "don't use C, because it would be looking into the past", and by inference, that for example D, Rust, or Go are "the future" that I should be looking into.

What kind of anti-intellectual stance is that? There is actual RESEARCH available, research that provides all those "logical arguments", research that has been done for DECADES by many people in the mathematics and CS community, some of them even clever. And you tell me "no, I am not interested what others have to say".

Ahem. "Please stick precisely to what the other one says, otherwise we will make no progress."

I did not say anything remotely like that.  I said, "Frankly, the CS articles I've read in the last decade or so, have concentrated more on constructing abstractions that help unskilled developers write code, and automatic detection and correction of the programming errors they create, than looking at robust, efficient, long-term maintainable code bases.  Hot air and gushing about favourites, without any true invention or progress."  I later said, "No, I am not interested in how convinced you are, or who else says so; I am interested in logical arguments for or against, or personal experiences."

In other words, most of the CS articles about this kind of stuff that I have looked at (at ACM, including in TOPLAS), is either irrelevant in practice, or of poor quality; usually both.  Consider A large scale study of programming languages and code quality in github: cited 126 times, but a recent reproduction study showed the same data does not actually support the original conclusions claimed.  Perhaps you consider that a single bad example among the good ones, but in my experience, this is typical of current CS articles.  Sure, there are good papers in there describing say an algorithm, perhaps even something that in some selected cases produces a few percent increase in efficiency for some workloads, but I haven't seen anything that measurably affects software engineering in years.  Nothing that makes a significant practical difference.

Maybe I haven't read enough; certainly possible!  If you disagree, just point me to a paper you think is relevant to low-level systems programming and if/when implemented, would make a difference to software projects' code quality.  Right now, I believe you are seeing serious progress where I see only abstract, occasionally incremental development; I see nothing that really affects real-world practical software engineering.

My stance on this is simple: C is a language proven in practice, but its standard library has significant easily demonstrated issues.  C compilers do not force us to link against the standard library, so it is a very straightforward matter to create a replacement library, one that does not implement standard C functions, but provides a different set.  I believe I have at least a partial idea of what kind of a set/API would be "better", but I would highly appreciate others experience and findings, because those surely help: I do not know everything, and I err often.  Opinions are less useful than the reasons for those opinions, because with the reasons, I can compare to my own reasons, and decide whether my own opinion is on a weaker basis, and should change.
Aside from garbage collection (and automatic methods of detecting programmer errors), I know very few good CS articles relevant to low-level systems programming published in the last decade or so.  Definitely none that describe anything fundamentally better than I believe is achievable with freestanding C and a "better" base library.

[MarginallyStable]

The #1 thing is to ban free() and base a C library on garbage collection. I disagree with Apple and prefer tracing over reference counting. The Boehm library work very well, with a few modifications -- I vastly reduced the amount of static memory needed and optimized the object size buckets and a number of other things.

I disagree here.  I think improvements to C should be made by making the language simpler and having less gotchas, especially in terms of simpler to understand & simpler to predict.  Garbage collection implementations always have corner cases and harder to predict behaviours (eg runtime slowdowns).

It's bad enough that dynamic memory allocation is already unpredictable, I don't want more of this 

Garbage collection is faster than malloc/free. I can (and have) proved that by using LD_PRELOAD to replace the C library malloc with Boehm GC for a large range os programs, with the result that they finish sooner and/or use less CPU time.

That's even on programs that are completely ignorant of the fact they are now running under GC.

A properly-written GC can easily limit the maximum pause time to an imperceptible level.

That would depend on if the program ever garbage collected. You might see the same performance increase by simply not calling free and hoping you do not run out of memory before the program finishes.

[Nominal Animal]

Garbage collection is faster than malloc/free. I can (and have) proved that by using LD_PRELOAD to replace the C library malloc with Boehm GC for a large range os programs, with the result that they finish sooner and/or use less CPU time.

That would depend on if the program ever garbage collected. You might see the same performance increase by simply not calling free and hoping you do not run out of memory before the program finishes.

There are also cache effects.  Meaning, if instead of accessing different cache lines the GC causes the process to access the same cache lines (by reusing the same memory region allocated for the process) it can actually run in less wall clock time than the never-free()ing code, because of fewer cache misses.

[brucehoult]

The #1 thing is to ban free() and base a C library on garbage collection. I disagree with Apple and prefer tracing over reference counting. The Boehm library work very well, with a few modifications -- I vastly reduced the amount of static memory needed and optimized the object size buckets and a number of other things.

I disagree here.  I think improvements to C should be made by making the language simpler and having less gotchas, especially in terms of simpler to understand & simpler to predict.  Garbage collection implementations always have corner cases and harder to predict behaviours (eg runtime slowdowns).

It's bad enough that dynamic memory allocation is already unpredictable, I don't want more of this 

Garbage collection is faster than malloc/free. I can (and have) proved that by using LD_PRELOAD to replace the C library malloc with Boehm GC for a large range os programs, with the result that they finish sooner and/or use less CPU time.

That's even on programs that are completely ignorant of the fact they are now running under GC.

A properly-written GC can easily limit the maximum pause time to an imperceptible level.

That would depend on if the program ever garbage collected. You might see the same performance increase by simply not calling free and hoping you do not run out of memory before the program finishes.

Yep, you can try that too. And I have.  For example with gcc compiles (a relatively short-running program).

What you'll find is that even relatively small compiles quickly use GB of RAM and it's faster to do a few GCs. But not too many.

Maybe I'll put together a page on how to try this experiment for yourself.

[magic]

Fix the size of int, long, char, etc; don't let them be platform specific.  In practice this means having to use different type names like stdint.h does (otherwise you break compat with other code & compilers),  I normally go for more sensible names like   u8, u16, u32, u64,   s8, s16, s32, s64,   f32, f64  (rather than things like uint32_t, uint64_t, etc)

Found the x86 user

C was designed to run on platforms where 32 bits is slower than 16 bits and on platforms where the contrary is the case. It will even support a 36 bit CPU for that matter, though probably fewer people care about that nowadays.

I agree with the latter part, the standard uint_t names are annoying.

[Whales]

Fix the size of int, long, char, etc; don't let them be platform specific. [...]

Found the x86 user

I have spent a good 10 minutes now trying to come up with convincing defence.  All it has done is remind me of is that I miss my snow laptop (edit 7 years now?!?!)

[Nominal Animal]

C was designed to run on platforms where 32 bits is slower than 16 bits and on platforms where the contrary is the case. It will even support a 36 bit CPU for that matter, though probably fewer people care about that nowadays.

That is completely true, but all our interchange formats use these base integer and IEEE-754 floating-point types; and that is what we really need them for.

In real world applications, size_t (and to a lesser degree, uintptr_t/intptr_t) are the "common" integer type, being able to describe the size of any in-memory structure (in units of char, "byte").

The distinction between "int" and "long" has become annoying, because on LP64 x86-64 ABIs, longs tend to be "cheaper" than ints, as the native general purpose register size is 64 bits.  On ILP64, there is no distinction.  What we need, is a (set of) integer types that can describe at least a specific range of values.  We could look at Fortran (its kind= notation), but make it simpler, explicit; so that to declare an unsigned integer type variable foo, we'd use
    INT(0, max) foo;
and to declare a signed integer type variable bar,
    INT(min, max) bar;
with the preprocessor macros evaluating to a scalar unsigned or signed type that can represent all integers from min up to and including max – but possibly a much larger range –, using the fastest representation and size for the current hardware; or emitting a compile error if no suitable type exists at preprocess time.  And yes, this implies that e.g.
    typedef  INT(0,256*256*256-1)  rgb_color;
    typedef  INT(-1, 1024*1024*1024-1)  my_size;
would be perfectly acceptable ways of declaring integer types.

Unfortunately, the C preprocessor does not really make that possible; it would require a ternary macro expression evaluator that is evaluated at preprocess time, with numerical values evaluated (with basic +-*/ integer arithmetic) at preprocess stage.  (That would sure be useful for standard C code too, though.)

The best we have, is the badly-named uintfastN_t and intfastN_t types, with <stdint.h> defining them for N = 8, 16, 32, and 64.  We could define them for all N between 8 and 64, but since there is no hardware that GCC and clang compile to with integer types having any other values of N, there hasn't been any practical need.  Maybe we should, though, at least for N multiples of 8/16/32?

I still have to come up with a good, short descriptive name for these types, that are easy to type too.

[Nominal Animal]

One idea does keep bugging me:  What if we used uNe and sNe for the exact-width types, and uN and sN for the fast types that may have bigger range?

In other words, that u32 == uintfast32_t, and s64e == int64_t?

I know that is a big break from existing practice, and would feel weird to many, but it just seems to me it would match real-world use cases better.
If you have a function or a local variable, you want to use the fast types.  You should only use the exact-width types in data structures (and in arithmetic expressions via casts).

[brucehoult]

Fix the size of int, long, char, etc; don't let them be platform specific. [...]

Found the x86 user

I have spent a good 10 minutes now trying to come up with convincing defence.  All it has done is remind me of is that I miss my snow laptop (edit 7 years now?!?!)

Geez, that thing's in far worse shape than my 2011 11" MacBook Air (dual core i7 1.8 GHz, 2.9 GHz turbo)

A dual 1.7 GHz A15 is nothing to be sneezed at. I've got a quad core A15 2.0 in an Odroid XU4 and it rocks if 32 bit is good enough. I'm surprised it can run on batteries though. Well, I mean ... anything *can* ... just not for long.

[magic]

I have spent a good 10 minutes now trying to come up with convincing defence.  All it has done is remind me of is that I miss my snow laptop (edit 7 years now?!?!)

Okay, so you might know that ARM has no 16 bit registers and no 16 bit operations besides load/store IIRC. So perhaps 32 bit would be a good fit for int on ARM, but then try to use it on AVR and see what happens

The distinction between "int" and "long" has become annoying, because on LP64 x86-64 ABIs, longs tend to be "cheaper" than ints, as the native general purpose register size is 64 bits.  On ILP64, there is no distinction.

There is nothing cheaper about 64 bit on x86-64 until you actually need 64 bits. There is a full set of 32 bit registers and a full set of operations on them implemented efficiently. In fact, to get 64 bit arithmetic you add a special prefix byte to the corresponding 32 bit instruction. Ditto when accessing the R8-R15 registers, by the way. In all honesty, x86-64 is an LP64 architecture - an i386 with long pointers and ability to do 64 bit arithmetic if you really insist. If you think about it, its primary design objective was to run 32 bit Windows natively unlike Intel Itanic

[DiTBho]

Yesterday I tried to compile a C++ project with a lot of big templates on a i686 32bit Linux box, and g++ crashed because unable to manage more than 2Gbyte of stack, or something similar. The CPU is already 64bit, but I have two kernels and two rootfs,  32bit and 64bit, so I rebooted the computer into i686 64bit, and everything went fine.

gcc is Mbyte-hungry
g++ looks Gbyte-hungry

[DiTBho]

I agree with the latter part, the standard uint_t names are annoying.

I disagree. I don't find it annoying, but rather elegant, practical, and coherent

[SiliconWizard]

I agree with the latter part, the standard uint_t names are annoying.

I disagree. I don't find it annoying, but rather elegant, practical, and coherent

Ditto. This is actually a consistent naming convention for std C types, that I use in my own code as well.
It certainly shouldn't be changed. Shorter and less consistent identifiers are not just less consistent and possibly ambiguous, but they drastically increase the possibility of identifier clash, which the std comittee is well inspired to avoid at all costs.

[magic]

There is nothing inconsistent or confusing about the set proposed by Whales, which is also used in many actual C projects.

At the very least, drop the damn _t. As if I didn't know that intNN is a type, as if everybody weren't using an editor which highlights it anyway

[Nominal Animal]

Does anyone have any particular function interfaces they really like?

As I've already mentioned, I really like getline(), because it avoids line length limits.  (Perhaps there should be an optional length limit, or rather a maximum memory use limit.)  I'd love a getdelim() that takes a set of possible delimiters instead of a single character, and the inverse (that only reads input as long as they are within a specified set), similar to strspn() and strcspn(); but without consuming the delimiter (like getdelim() does).  It would allow lexical analysis directly on file-like streams, as well as universal newline support, with very little overhead.  Instead of a first-level stream type, I think it'd be better to have buffer types (readonly, read-write, writeonly) exposing this functionality, attachable to a file descriptor.  I think having the stream abstraction explicitly described as an abstraction could also help new programmers get a more realistic overview of Linux/BSD/etc. OS operations.

I also find iconv_open(), iconv(), iconv_close() very powerful for character set conversions; POSIX regex (regcomp(), regexec(), regfree()) for regular expressions; nftw() for walking filesystem trees (although how the number of file descriptors it uses needs work, and needs atfile support); scandirat() for listing directory contents; glob() for finding files matching a pattern (except it too needs atfile support).

Instead of open(), I intend to expose open(dirfd, pathname, what, how [, mode]), with how containing flags affecting the file descriptor, and what affecting filename resolution (O_DIRECTORY, O_BENEATH, openat2() RESOLVE_ flags).  (The number of flags needed exceeds 32, so instead of requiring a larger type, I think splitting them into two by purpose works better.) 

Since Unix domain sockets and named pipes (FIFOs) are visible in the filesystem, I think we should expose them via what flags explicitly.  A synthetic dirfd constant would expose abstract Unix domain sockets (not visible in the filesystem), with how determining whether to connect or to listen, and what determining the socket type (stream, datagram, seqpacket).  The hope is to encourage use of Unix domain sockets for IPC.

(I do think that having to use mkfifo(), bind(), and connect() to access filesystem-visible FIFOs and sockets is a silly complication due to development history, and that the above is about fixing that, not providing an abstraction.)

[PlainName]

As I've already mentioned, I really like getline(), because it avoids line length limits.

But it's inconsistent. You sometimes have to free() and sometimes not, and sometimes have to free() even if you say you don't want anything from it! I think it's trying to cover too many bases and would be better split into two functions: one that allocates memory and one that uses pre-allocated memory.

edit: also it's a blocking call. Sure, we all use a RTOS nowadays but sometimes we don't (and this is specifically for small embedded systems, don't forget). If it doesn't block you can add blocking easily enough, not so easy to do it the other way.

[Nominal Animal]

As I've already mentioned, I really like getline(), because it avoids line length limits.

But it's inconsistent. You sometimes have to free() and sometimes not, and sometimes have to free() even if you say you don't want anything from it! I think it's trying to cover too many bases and would be better split into two functions: one that allocates memory and one that uses pre-allocated memory.

I don't understand what you mean.  You start with

Code: [Select]

char   *buffer_ptr = NULL;
size_t  buffer_max = 0;
or with

Code: [Select]

size_t  buffer_max = somenumber;
char   *buffer_ptr = malloc(buffer_max); /* verified to have succeeded */
The buffer_ptr must always be either NULL, or a pointer to dynamically allocated memory as returned by malloc(), calloc(), realloc(), or aligned_alloc().  It cannot and must not point to statically allocated memory.

To read an input line, you do

Code: [Select]

ssize_t  len = getline(&buffer_ptr, &buffer_max, file_handle);
If len==-1, then either there was no more data to read, or there was an error.

Regardless of the result, it is always safe to discard the buffer via

Code: [Select]

free(buffer_ptr);
buffer_ptr = NULL;
buffer_max = 0;
Because free(NULL) is safe, this is always safe to do, even when buffer_ptr==NULL.

You do have to be aware that after every getline() call, the function may have modified buffer_ptr and buffer_max; you must not assume they are unchanged.

TL;DR: You start with a NULL pointer and a zero size, or an already dynamically allocated buffer, and can discard (or steal the dynamically allocated buffer) at any time you wish.  After you're done with it, you always discard the buffer.  Very consistent, very simple, very robust.

So, what is the inconsistency?

[Siwastaja]

The base C language is quite fine actually, most of the footguns are related to the stupidly designed standard library. Similarly, most things that are more complex than they need to be, are related to the library. People think they prefer "higher level" languages because of the core language features, but I think they actually like simple-to-use, powerful libraries that come with said languages.

With C, there is always some extra housekeeping to do, but there is no reason why searching, concatenating, parsing, generating strings for example must be a delicate 100-line endeavor vs. a two-liner in a "modern" language. With a decent, modern-day library, it would be, say, a 10-line job in C.

So I think this project is exactly on the right track. The freestanding C plus a few core library features (those that are likely provided by compiler built-ins instead of actually linking to stdlib) like memcpy or malloc are fine, but the mess caused by all the complex, unsafe, and unhelpful library "helpers" makes full-blown standard C more difficult than it needs to be.

[PlainName]

Does this buffer the line until it's completely read, then? If the stream is, say, a serial port, does it return after each call with -1, meantime buffering the input, until it sees a delimiter, and then return the buffer length? If so, I misinterpreted the description.

[Nominal Animal]

Does this buffer the line until it's completely read, then? If the stream is, say, a serial port, does it return after each call with -1, meantime buffering the input, until it sees a delimiter, and then return the buffer length? If so, I misinterpreted the description.

Plain reading of the IEEE Std 1003.1-2017 says that if an error occurs, getline() and getdelim() should return -1 with errno set, but most current implementations instead return the currently buffered data.  Most of them also assume that when read() returns a short count, it is either because of end-of-input or an error, and that just isn't true in practice.

The behaviour you and POSIX describe – that when the underlying descriptor is nonblocking, the call will return an error indicating it would block if the buffer does not contain a delimiter and end-of-input hasn't been received yet (because the buffer-filling read() call reports an error, EAGAIN/EWOULDBLOCK) – is what makes sense, obviously.  This also applies when the underlying read() is interrupted by signal delivery.  Instead of returning whatever contents there may be in the buffer, it should return an error (EAGAIN/EWOULDBLOCK, or EINTR).

Like I said, I like the interface; not necessarily the implementation details (which I consider buggy anyway).

[PlainName]

OK. I retract my objection, then.