Is ST Cube IDE a piece of buggy crap?

[nctnico]

Quoting nctnico, the truth is somewhere in the middle.

If integrating a complete existing TCP/IP stack and fixing/working around all the bugs took over a year for a team of four, clearly this process is broken and doesn't provide what it's supposed to do. You should be writing rest of the application and enjoying the library.

OTOH, that doesn't mean it could have been done from scratch in a few weeks, this is just way too optimistic. It sounds like the requirements for this stack were high, this was "the real thing". It's well known how colossally complex full TCP/IP solutions are.

As I wrote before: due diligence is key when using a library. Believing the sales pitch is a huge mistake. More often than not I evaluate several libraries before choosing one.

[Kjelt]

Still, NorthGuy is completely right about the bias. People are ready to accept a year of work of just using a library without blinking an eye, as a normal, expected cost. In Orwellian way, this may be even called "saving time". But the mere idea of having to spend the same year of work doing something from scratch makes people outright crazy. People fear writing code, and management fears large blank sheets.

There must be more to that story.

Yes there is. The stack was evaluated for IPv4 and IPv6 was "promised" to be implemented in the future.
More stacks were evaluated but this one was choosen by the wise system architects.
The whole idea was to not write from scratch but to stick to core business so implement the companies product intellectual property and buy what was common stuff.
AND most important the guarantee for full stack support by that company.

Then after you bought the stack and went from prospect to customer and product testing begins even with some pentesting from other companies all hell broke loose.
Only then could you see the holes and things that went wrong, so the tickets to the company to support the stack were flowing but the support was bad.
Tickets would be solved not in days or a few weeks but rather months and communication was cumbersome.

Anyway to make a long story short, we ended up with most of the source code (ofcourse under NDA) so we could fix things ourselves more quickly instead of a compiled library.
And lessons learned do not only evaluate what you think you are going to buy, but also really invest time to see who and how many engineers are behind a product, to support and fix bugs.
Still my guestimate for developing a decent fast and reliable IP stack with IPv4+6 and some rudimentary services would still be at least a manyear, but probably more. 

[DavidAlfa]

Check the include search path under project options.
You can use relative and absolute paths, and that might be the issue

[NorthGuy]

... I do not want to have to write from scratch: an Ethernet stack for TCP/IP and web server

Me neither. Nor do I suggest to write their own TCP/IP stack to anyone. My point is that people grossly exaggerate  efforts necessary to do so. And the responses clearly show that this is indeed so. Some people even think it may take years.

But then I guess you have to do it to believe it. Try it and we talk again ...

Unfortunately, I don't have time or need. If I ever implement TCP/IP, I'll try to remember to time it and post here. Until then this is all hypothetical.

... perhaps you are a genius wizzard and myself and the programmers from my previous team are morons 

That's not what I'm saying. Any programmer with few years of experience in low-level programming and good understanding of data structures can do that. If you wish, I can break it down into small tasks with time estimates, starting from an unprogrammed board to working TCP transmissions ready to be used by the application layer.

Look at the long list of bug fixes in Lwip for example. Many of them are easy to overlook in the first pass; it is hard to write a good protocol stack.

Programming is programming. It is not different whether you implement TCP or your application code. There will be bugs whether you like it or not. You will have to test and fix them.

[nctnico]

... I do not want to have to write from scratch: an Ethernet stack for TCP/IP and web server

Me neither. Nor do I suggest to write their own TCP/IP stack to anyone. My point is that people grossly exaggerate  efforts necessary to do so. And the responses clearly show that this is indeed so. Some people even think it may take years.

Those 'some people' have actual hands-on experience with large projects like that 

If you've been in software engineering for a while you ought to know that development on a project (features added / bugs fixed versus time spend) slows down exponentially due to complexity (and if you are unlucky: poor architectural choices).

[Bassman59]

... I do not want to have to write from scratch: an Ethernet stack for TCP/IP and web server

Me neither. Nor do I suggest to write their own TCP/IP stack to anyone. My point is that people grossly exaggerate  efforts necessary to do so. And the responses clearly show that this is indeed so. Some people even think it may take years.

I know that I do not know enough about networking and TCP/IP to even give an estimate about how long it would take me to write a stack. You have to understand the problem before you can work on the solution.

Other people have invested the time to fully understand IP, and they've written code to implement it. I'm happy to use lwIP, which is the result of a developer who took the time to understand IP and continues to maintain it as IP evolves. I'm glad it exists. I can put it in my micro and have a nice web server that lets me do what my projects need to do.

[peter-h]

LWIP seems to work well.

However it seems to be the nature of "libraries" that most are buggy. And they remain buggy for ever - because most of these bugs are fixed by people who are paid for their time and who are not going to contribute their fixes to the wider community (confidentiality, an explicit policy to not assist the competition, and in many cases you could work out from forum posts what Company X is working on). So, almost the only bug fixes which are posted openly are fixes done by hobbyists, and they naturally have fewer resources.

Regarding ST libs, you can try reporting stuff on the ST forum but it is pretty obvious that if any ST staff do read it, they keep an exceedingly low profile I have seen a few things fixed between Cube today and Cube 1 year ago, but only a few.

Check the include search path under project options.
You can use relative and absolute paths, and that might be the issue

Close; it was actually fixed under project / properties / settings / c compiler / include paths.

[NorthGuy]

Those 'some people' have actual hands-on experience with large projects like that 

  What hands-on experience? Nobody here ever said that they ever wrote their own TCP/IP implementation.

If you've been in software engineering for a while you ought to know that development on a project (features added / bugs fixed versus time spend) slows down exponentially due to complexity (and if you are unlucky: poor architectural choices).

Sure. IMHO the number of bugs grows proportional to the squared size of the code. That is exactly why, if you build your code base from dubious over-complicated libraries, it is likely to have many bugs, as Kjelt brilliantly demonstrated. And you will inevitably waste lots of your time working with such a code base. Smaller, simpler, to-the-point code will give you much less problem, and that's the kind of code you can write by yourself if you wish.

Why would you use lwIP if you can write your own? Because lwIP is decent and already does what you want. You probably won't encounter many bugs in lwIP, most likely none. So you expect to spend less time incorporating lwIP than you would if you implemented the usable portion of it by itself. But that doesn't mean that writing your own code is impossible or take lifetime to write. If the library doesn't do what you need, you don't go around buying unusable crap, you write your own.

[westfw]

Nobody here ever said that they ever wrote their own TCP/IP implementation.

We had our own TCP/IP.

It started out simple enough, but in the end (or "so far") there were hundreds of man-years invested.
Lots of special requirements, over the years.

It was glorious!

My favorite personal memory was when I implemented IP options, back before anyone else had done so.
We took the code to InterOp and sent broadcast Pings with IP options, just to see what would happen (it was long enough ago that you could get away with stuff like that.)  Things crashed right and left.  Other things sent horribly malformed responses.  WE crashed when we tried to interpret the malformed responses!  Lots of fun!

[Siwastaja]

In real world, TCP/IP tries to put massive bandwidth over unreliable links in unreliable environment, this tuning takes 99% of the effort. Byte stream over unreliable packets isn't really the optimal thing to do, especially if the next layer parses the byte stream back to delimited packets. But this is what we have.

Of course if you just assume that packet loss is small and malformed headers do not occur, then it's easy enough to write it to the specification and just use a buffer of latest packets for resending purposes.

Until you see that results in unsatisfactory performance, buffer bloat, awkward delays whenever a packet is lost, the cat video doesn't play properly.

Open source implementations have the best chances of evolving in the right direction. Heck, even Microsoft used BSD code for their TCP/IP stack!

There is one particular option which is surprisingly widely used: if you need a very limited kind of TCP, just don't use TCP, create your simple code over UDP!

But NorthGuy is right that there are special cases where custom solutions are not that hard to do. The risk is in misidentification; what if the use case grows? Well, nothing prevents you from turning back at that point. But the more time you spend in the "simple" implementation, the more attached you come to it. This is the danger of NIH.

But yes, dangers of NIH, while real, are often exaggerated, while dangers of code reuse and libraries, while equally real, are often belittled. You just need to understand the particular case you are working with really well to make an informed choice.

[peter-h]

There is one particular option which is surprisingly widely used: if you need a very limited kind of TCP, just don't use TCP, create your simple code over UDP!

Yes this is used where you control the other end. It is used for closed systems e.g. distribution of weather images, where one line can be lost and nobody cares. And a lot of the time you can send each packet a few times, just in case. Just got to make sure it is within any possible MTU, so say 1200 bytes max.

[westfw]

Just got to make sure it is within any possible MTU, so say 1200 bytes max.

Why would you have to do that?  Fragmentation and reassembly all happen at the IP layer underneath UDP, so max udp payload should be up near 64k.  IIRC, EGP was throwing around 18k UDP packets before it was replaced by BGP…

Although I have to disagree with your assertion that tcp is meant to deal with error-prone networks.  For the last couple of decades the only errors it has worried about have been congestion.  There are some other  protocols to use if your network is actually lossy.

[peter-h]

Fragmentation and reassembly all happen at the IP layer underneath UDP

I got that backwards then

UDP is used on e.g. low quality / huge latency satcomm connections (Thuraya/Iridium) where you can have latencies of the order of 1-3 secs which tend to just "destroy" TCP/IP.

I thought there was no guarantee of delivery whatsoever.

with your assertion that tcp is meant to deal with error-prone networks

Was that me?

There still are connections out there which drop a lot of packets, like the ones above.

Anyway, well off topic re the Cube IDE libraries

[NorthGuy]

Just got to make sure it is within any possible MTU, so say 1200 bytes max.

Why would you have to do that?

For example, if you expect very high packet loss, say 20%. If a packet is broken down to 20 fragments, all 20 must go through to reassemble the original packet. With 20% loss, the probability of this happening is very low - around 1%. So, instead of 20% loss, you will have 99% loss. Moreover, the recipient's buffers will get clogged with lots of fragments of different packets which never will be fully assembled. You can actually mount a DoD attack by sending fragmented packets but withholding some fragments so that the packets can never be assembled.

[Siwastaja]

Although I have to disagree with your assertion that tcp is meant to deal with error-prone networks.  For the last couple of decades the only errors it has worried about have been congestion.  There are some other  protocols to use if your network is actually lossy.

But that's exactly what it needs to do in real world, outside test lab, you agreeing or disagreeing doesn't matter. Networks are what they are, broken devices also exist, and if this causes complete disconnects, or very long delays, or worst, wrong behavior, people start complaining. It's in the best interest of the "consumers" to make networks work as flawlessly as possible, and this is exactly what has gone into the typical TCP/IP implementations during the last many decades.

Buffer bloat alone is a huge issue when some links add delay (and jitter!), yet huge bandwidths are passed through those links (4K cat videos). It's a miracle how well modern TCP/IP works. Or, a lot of engineering. The difference can be actually seen, a modest amount of packet loss won't feel as problematic as it did maybe 20 years ago. The web page seems to open either in 100ms or then it takes a few seconds. Not that "need to wait for 2 minutes or manually refresh page" bullcrap anymore.

The question whether you need to replicate all that when rolling your own is the key question in this context.

[nctnico]

Just got to make sure it is within any possible MTU, so say 1200 bytes max.

Why would you have to do that?  Fragmentation and reassembly all happen at the IP layer underneath UDP, so max udp payload should be up near 64k.  IIRC, EGP was throwing around 18k UDP packets before it was replaced by BGP…

I think you are mistaken TCP for UDP here. UDP is limited to whatever the maximum packet size is of the network layer (typically 1500 bytes for ethernet so the maximum UDP size is little over 1400 bytes). UDP has no fragmentation / re-assembly which makes it suitable for transmitting / streaming non critical data. For some embedded platforms I'm using much smaller maximum packet sizes; like 500 bytes or so which is enough for basic DHCP and DNS.

[NorthGuy]

... Fragmentation and reassembly all happen at the IP layer underneath UDP ...

I think you are mistaken TCP for UDP here.

Both UDP and TCP run on top of the packet-based IP protocol (RFC 791), which provides its own mechanisms for fragmenting and defragmenting packets.

UDP simply uses these packets adding practically nothing on top of IP.

TCP provides byte stream interface on top of IP. It breaks the byte stream into fragments which are transmitted with IP packets then re-assembled back into the byte stream on the other end.

[peter-h]

I think ntnico is right - because the MTU negotiation itself uses UDP packets. There is no more fragmentation support underneath UDP.

I have seen this negotiation broken a number of times, sometimes at "big" ops e.g. Vodafone. We have a server at work which was not visible from a Vodafone 3G/4G client, until the server MTU was reduced to 1300 They fixed it after a few months. I think it was a config issue between Voda and our ISP, so fairly obscure.

[NorthGuy]

I think ntnico is right - because the MTU negotiation itself uses UDP packets. There is no more fragmentation support underneath UDP.

There's no need to guess, just read: https://datatracker.ietf.org/doc/html/rfc791

MTU is a characteristic of a network interface (for example, Ethernet frame may only carry 1500 bytes of payload). A packet goes through a number of hops from the source to the destination, all with their own MTUs. If a packet doesn't fit MTU ant any of these hops, the IP protocol (the one underneath UDP) will fragment the packet.

[nctnico]

Unfortunately a lot of devices don't support UDP fragmentation (or any IP packet fragmentation). On Linux you'll get an error when you try to send a UDP packet which doesn't fit the MTU. You have to turn on IP fragmentation explicitly but it is highly discouraged because lots of devices don't even support it (well enough).

[NorthGuy]

On Linux you'll get an error when you try to send a UDP packet which doesn't fit the MTU. You have to turn on IP fragmentation explicitly but it is highly discouraged because lots of devices don't even support it (well enough).

I don't get any errors. Try:

Code: [Select]

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <netinet/in.h>

int main() {
  int sockfd;
  int bytes;
  char buffer[0x10000];
  struct sockaddr_in addr;
 
  memset(&addr, 0, sizeof(addr));
   
  sockfd = socket(AF_INET, SOCK_DGRAM, 0);
  printf("sock fd %d\r\n", sockfd);
 
  addr.sin_family = AF_INET;
  addr.sin_addr.s_addr = htonl(0xc0a80a04);
  addr.sin_port = htons(6000);
   
  bytes = sendto(sockfd, buffer, 65000, MSG_CONFIRM, (const struct sockaddr *) &addr, sizeof(addr));
  printf("sent %d\r\n", bytes);

  close(sockfd);
}
It prints:

Code: [Select]

sock fd 3
sent 65000

[dkonigs]

I love seeing threads full of high-horse armchair experts like this one.

Early in my career, before I meaningfully entered the professional world, this was pretty much my only real exposure to those who worked within it. Everyone seemed to have such high expertise, and such a high opinion of what the standard skill level should be, that I couldn't help but feel grossly inferior and inadequate. Heck, by the impressions I was getting, I wasn't sure if I was even worthy of owning a keyboard, let alone using it to interface with a computer, or *gasp* using it to actually write software.

Then I got out there into the workforce, and was shocked and amazed at just how far this was from the truth. The average level of skill in the industry was far lower than I could have imagined. The true experts seemed more like an outlier than the norm. Its just that the riffraff didn't have as much online technical presence outside of their own workplaces.

[nctnico]

On Linux you'll get an error when you try to send a UDP packet which doesn't fit the MTU. You have to turn on IP fragmentation explicitly but it is highly discouraged because lots of devices don't even support it (well enough).

I don't get any errors. Try:

Interesting. This must have been changed then at some point; documentation I have been able to find suggested that Linux wouldn't allow it without explicitly turning it on.

[Sal Ammoniac]

I love seeing threads full of high-horse armchair experts like this one.

Early in my career, before I meaningfully entered the professional world, this was pretty much my only real exposure to those who worked within it. Everyone seemed to have such high expertise, and such a high opinion of what the standard skill level should be, that I couldn't help but feel grossly inferior and inadequate. Heck, by the impressions I was getting, I wasn't sure if I was even worthy of owning a keyboard, let alone using it to interface with a computer, or *gasp* using it to actually write software.

Then I got out there into the workforce, and was shocked and amazed at just how far this was from the truth. The average level of skill in the industry was far lower than I could have imagined. The true experts seemed more like an outlier than the norm. Its just that the riffraff didn't have as much online technical presence outside of their own workplaces.

I've interviewed a lot of people for embedded developer positions and I can confirm that quite a few folks have substandard knowledge of fundamental embedded concepts. In many cases, their knowledge level is so low I can't see how they've held a job in the field.

I've found that the best people in the field also do it as a hobby--in fact, that's a very good indicator of how competent someone is, so I always ask that in an interview.

[peter-h]

It has always been the case that to get good at anything you have to take a keen personal interest in it.