STM32F7 H7 LWIP webserver problem

[cal]

Hi,
Can anyone point me in the right direction ?
I designed a complicated project for my company consisting of an STM32H743 webserver using C, FreeRTOS,SSI,CGI and LWIP based on STM32cubeide compiler. I need technical help on the upload from browser straight to  serial port on STM32 via the STM32. It is partially working, but there are problems with intermittent corruption of the binary file during upload.
Basically I have SSI CGI etc working OK but the binary file upload for large files sent from the browser has never worked consistently.

In desperation, I used an STM32F746 Nucleo board without RTOS/SSi/CGI etc. - just a simple cubeide project using LWIP configured only for HTTPD.
Exactly the same problem.
I realise that I could strip out the content-disposition /type fields etc., but why are there no examples around that give a fault free performance without corruption, especially during large binary or text file transfers?
Any ideas? Anyone?
Thanks
Chris

I use this index.shtml:

  <form action ="/index.s html" enctype="multipart/form-data" method="post">
  <p>Simple upload. Please specify a  file:
   

<input type="file" name="datafilemnu" size="40">
   <input type="submit" value="Upload ppf">
 </br>

The reception of file data is as follows: (probably hacked about a bit now)
#define ser  // for serial port on usart3

err_t httpd_post_begin(void *connection, const char *uri, const char *http_request,
                       uint16_t http_request_len, int content_len, char *response_uri,
                       uint16_t response_uri_len, uint8_t *post_auto_wnd)
{

   return ERR_OK;
}


err_t httpd_post_receive_data(void *connection, struct pbuf *p)
{ //multipart/form-data is used
   int32_t i,len=0;
   uint32_t DataOffset, FilenameOffset;
   char *data, *ptr, filename[20];
   char readval[4800];
    DataOffset =0;

    data = p->payload;

    len = p->tot_len;

   // HAL_UART_Transmit(&huart3,data,len , 0xFFFFFF);

    if (DataFlag ==0) //has not entered before - must be first pbuf
    {  m1++;
        /* parse packet for Content-length field */
              //   size = Parse_Content_Length(data, (uint32_t)(p->tot_len));
      // parse packet for the octet-stream field
      for (i=0;i<len;i++)
      {
         if (strncmp ((char*)(data+i), octet_stream, 13)==0)
         {
           DataOffset = i+16;
           DataFlag++;
           break;
         }
      }

      // case of MSIE8 : we do not receive data in the POST packet
      if (DataOffset==0)
      {
         DataFlag++;
         BrowserFlag = 1;
         pbuf_free(p);
         return ERR_OK; // returns here but this is firefox

      }
      // case of Mozilla Firefox v3.6 : we receive data in the POST packet

       for (i=0;i<len;i++)
       {
         if (strncmp ((char*)(data+i), "filename=", 9)==0)
         {
            FilenameOffset = i+10;
            break;
         }
       }
       i =0;
       if (FilenameOffset)
       {
         while((*(data+FilenameOffset + i)!=0x22 )&&(i<20))
         {
           filename = *(data+FilenameOffset + i);
           i++;
         }
         filename = 0x0;

       }
       if (i==0)
       {

         pbuf_free(p);

         DataFlag=0; // RESET HERE
         return ERR_OK;

       }


     ptr = (char*)(data + DataOffset);
     len-= DataOffset;

  //   memcpy(str, p -> payload, p -> len);//added

m2++;
#ifdef ser
HAL_UART_Transmit(&huart3, ptr,len , 0xFFFFFF);
#endif

     pbuf_free(p);
     return ERR_OK;
    }//if dataflag==0


    else
    {

       // if last packet need to remove the http boundary tag
       // parse packet for "\r\n--" starting from end of data

          i=0; //??? 4?
       while ((strncmp ((char*)(data+ i),http_crnl_2 , 4))&&
             (strncmp ((char*)(data+ i),http_crnl_3 , 4))  && (i<len))
       {
         i++;
       }
    //   char *ip;
     //  ip=p->payload;
     //  removeSubstr(ip, "dndata=");
    //   memcpy(str, p -> payload, p -> len);//added
       m3++;
#ifdef ser
       HAL_UART_Transmit(&huart3, p->payload,i , 0xFFFFFF);
       #endif
           if(i!=len)
              DataFlag =0;
    }
    // not last data packet

    pbuf_free(p);

    return ERR_OK;
}

void httpd_post_finished(void *connection, char *response_uri, u16_t response_uri_len)
{
     connection = NULL;
}

   

[tellurium]

If commercial solution is OK, you can take a look at https://mongoose.ws/tutorials/stm32/nucleo-f746zg-cube-freertos-lwip/
File uploads are covered in https://mongoose.ws/tutorials/file-uploads/

[cal]

Thanks but tutorials are 'blocked due to Trojan' according to Malwarebytes - so I'm a bit apprehensive!
Chris

[peter-h]

I wrote an HTTP server for a 32F417 based product. Took a while but now is rock solid, file up and downloads via HTTP. Had to pay someone on freelancer.com to produce javascript for the file transfers - can't be done without that if you want a progress count of some sort. Also much easier to use PUT and you need JS for that.

A contractor started on that project using the ST supplied HTTPD source and after a few k GBP I killed that project (and paid him for time spent, obviously). That source is crap.

Most people develop fixes in company time so they don't publish them. They just leach from the web, github, everywhere

Basically none of that ST-supplied code actually works for serious use. LWIP itself is OK but you spend a lot of time doing the ETH hardware interface, and setting up the opts files with sensible values. FreeRTOS is good.

My HTTP server is a bare cut down job, using the netconn API instead of the socket API (that wasn't such a great idea, in case I want to use TLS one day) and I avoided the multipart stuff because it is messy to parse on a small micro.

Check out
https://www.eevblog.com/forum/microcontrollers/anyone-here-familiar-with-lwip/msg4306798/#msg4306798
https://www.eevblog.com/forum/microcontrollers/any-experts-on-lwip-and-the-netconn-api-(not-looking-for-free-can-pay)/msg4336729/#msg4336729
https://www.eevblog.com/forum/programming/what-is-the-http-server-client-interaction-to-do-file-transfers/msg4303780/#msg4303780

[betocool]

I wrote and audio analyzer based on the STM32H7 and use it from time to time. Lately I've used it to record music directly into a PC. Works a treat.

https://github.com/betocool-prog/Audio_Analyser_FW_OS

I have not updated the documentation, and it requires a daughter card to work. However, if you look at "tasks/Src/controller.c" you'll find a TCP Server on line 188 "controller TCP task". What it does is reading data from a streambuffer (the audio from the ADC) and streaming it to a TCP client on the PC. You could use netcat to connect to the TCP server and it would send everything without a hitch.

Of course you'll need to modify the code so that your send buffer is not ADC data but whatever it is you're sending. But the controller task itself should work well I think.

Hope it helps.

Cheers,

Alberto

[ttt]

      for (i=0;i<len;i++)
      {
         if (strncmp ((char*)(data+i), octet_stream, 13)==0)
         {
           DataOffset = i+16;
           DataFlag++;
           break;
         }
      }

Please don't take this the wrong way but have an experienced C developer review your code. The above code will read past allocated buffers resulting in undefined behavior.

At the very least do something like this:

Code: [Select]

      const size_t octet_stream_len = strlen(octet_stream); // Assuming octet_stream is a const char * string somewhere.
      if (len >= octet_stream_len) { // ensure we don't read past the length of the buffer
        for (i=0;i<(len - octet_stream_len);i++) // '- octet_stream_len' to prevent comparing past the buffer
        {
           if (strncmp ((char*)(data+i), octet_stream, octet_stream_len)==0)
           {
             DataOffset = i+16; // Where does the 16 come from?!
             DataFlag++;
             break;
           }
        }
      }

There are issues like this all over in your code. I highly recommend to at least run static analyzers on your code to catch these kind of problems.

C is a footgun.

[zilp]

If you want people to actually look at your code, please put in some minimal effort to make it readable. Like, format it as a code block, remove unused code, make sure the declarations for all variables are included, ...

ttt has already pointed out one problem - but also, more fundamentally, you seem to assume there is such a thing as a "POST packet", which there simply isn't, so you probably need to restructure your code. TCP doesn't transport records, it's just a stream of bytes, which is chopped up into packets for transmission over IP, and that you see those packet boundaries at all is simply an artefact of how low-level TCP implementations tend to be constructed, but how the stream is chopped up is completely abitrary, so your parsing state machine must be built such that it can handle packet boundaries between any two bytes in the stream.

Also, you really should read the HTTP RFC. You can't build interoperable software by looking at examples and guessing, only by reading, understanding, and following the relevant standards - and internet standards luckily are freely available, so no reason to not read them.

[tellurium]

The web server / uploads / multipart encoding is something that is not trivial.
A robust hand-crafted ad-hoc solution can be made only by an experienced engineer who understands the area well, and have done it before.
LWIP examples are not something that are easy to follow and modify.

OP, if you're not experienced in this area, then the following choices are reasonable:

a) educate yourself, which will take quite a bit of effort and time that far exceed asking a question in this forum (pointers are already given), or
b) seek for help from someone more experienced, or
c) use a ready-made code written by those experienced in the area (pointers are already given)

[peter-h]

There is no need for multipart for file upload and download via HTTP.

[tellurium]

There is no need for multipart for file upload and download via HTTP.

That's right!

Also, a client code that uploads a file (which is often a piece of JS code) can split the file into small-ish chunks, and send them sequentially - next chunk when a response from the previous one arrived. That makes the whole thing under control in terms of RAM usage, and also quite simple on the server side.

The same tactics can be done over Websocket, which can speed up uploads significantly if e.g. TLS is used - because it avoids multiple expensive TLS handshakes and does everything inside a single WS connection.

[peter-h]

a client code that uploads a file (which is often a piece of JS code) can split the file into small-ish chunks, and send them sequentially - next chunk when a response from the previous one arrived. That makes the whole thing under control in terms of RAM usage, and also quite simple on the server side.

No need for that even because tcp/ip does end to end flow control. So no large buffers are needed on the server side either. There is a thread here on how to do that. It was much simpler than I expected.

Lots of people tell you that you have to send in chunks and ack each chunk. Even some experienced web coders have told me that. You absolutely do not need to do that. You can just send the data - in either direction - and it "just works". What is not possible without JS is file transfer client -> server with an indication of progress (a bar graph, a % reading, whatever). Going server -> client, most browsers provide some sort of progress indication, even if primitive, and no JS is needed. I am doing file transfers with 512 byte blocks and I am getting about a megabyte per second down (limited by lwip buffer config, ETH low level compromises, and the filesystem, FatFS, being in SPI flash with a 21MHz clock), and the up speed is limited by the flash write speed to about 30kbytes/sec (actually I do a pre-read and if the 512 byte block is same, then I skip the write, which speeds up file uploads about 50x).

The thing I can't do much about is browser caching, which does weird stuff in some browsers, so a 512k file may appear to upload in 1 second, but you have to wait until the transfer is completed, which is not exactly dumb-user-friendly. If you want to break browser caching, in a browser-independent way, then you do need to handshake each block. I suspect image upload libs like Dropzone do this, but good luck getting that to run on an embedded target.

However I am not doing it with TLS. No idea how I would do that. I would need to change from netconn to bsd sockets for a start, I think. But a server open to the outside is a very bad idea for IOT, and if you aren't opening it, then HTTPS is pointless. TLS uses a huge amount of code - roughly 200k of my 450k total codespace - and a huge amount of RAM - I am allocating a 48k block for MbedTLS's private heap, and this seems not too generous - plus the setup crypto takes ~3 secs on a 168MHz 32F417. So maybe 1 second at 480MHz.

[tellurium]

I am pretty aware of the TCP flow control mechanism. As server slows down reading from the socket/connection, then the TCP stack shrinks the TCP windows size, signalling the client to slow down too. That's a common knowledge.

Yet even taking that into consideration, there are cases when chunking is the way to go.
Why? Because when one uses TCP flow control, the buffering / TCP window is determined by the network stack config. To put it simply, the socket/connection buffer size drives it. With chunking, the control is on the application layer.

I don't see any point arguing further. My remark was that the most controllable way is chunking, and I stand on that point of view.

[peter-h]

That's a common knowledge.

Obviously, I knew that you know all this, but not everybody does

Because when one uses TCP flow control, the buffering / TCP window is determined by the network stack config. To put it simply, the socket/connection buffer size drives it. With chunking, the control is on the application layer.

What is the benefit? It's a lot more work all around, including setting up buffers in client JS and implementing the handshaking there.

I don't see any point arguing further. My remark was that the most controllable way is chunking, and I stand on that point of view.

Nobody is arguing with you.

I was trying to help out the other guy, who is trying to do this with that ST-ported junk code.

I've had a huge amount of help on eevblog, working alone on pretty complicated stuff, so I try to put some back by helping out when I can (which is not often).

[tellurium]

The client side is not complex. Here is the example, under 50 lines of code:
https://github.com/cesanta/mongoose/blob/master/examples/file-upload-multiple-posts/web_root/app.js

Since the client code knows the total length and already sent length, it knows the progress %, too.

[dietert1]

Yesterday i tried the ST supplied "junk code" and used their "LwIP HTTP Server Netconn RTOS" example from the STM32H7 repository V1.10 to run on a Nucleo-144 STM32H743. IAR converted the project to a newer version and i reduced optimization to "low" to ease debugging. I compiled and installed the example app via USB.
Used a patch cable to connect the nucleo to the notebook and got the home page in the browser after looking up the hard coded IP address of the ST example. The dynamic page with its RTOS task info also worked.
Then i compared their LwIP to the current release 2.2 and there are differences (including corrected typos or renames). Will try to update the STM32 project to the current version.
Will also try to get the STLink CDC connection to support debugging. We got one of these nucleo boards wired with a GPIB interface and i could try to make it an ethernet GPIB controller, maybe with a fiber PHY later on..

Regards, Dieter

[dietert1]

Meanwhile i got the LED blinking and integrated the GPIB driver, added a working telnet connection and Usart3 = STLink CDCs, too. All working reliably under the RTOS. "wrt" and "rcv" are telnet threads and "Start" is kept alive in order to produce the enumerated test messages. I had to increase heap from 30 to 40K.
Now i am trying to come up with a fiber interface, maybe similar to the 1x9 transceiver module inside our tp-link media converter. It is an Allray ATR-01105EM for 155Mbps, duplex SC, MM, 1310nm, 2km, 3.3V.
What does it take to connect such a fiber transceiver to the RMII of the STM32H7?
The nucleo evaluation board i am currently using has a copper interface built with a LAN8742A chip plus magnetics and socket for 10/100BASE_T. Inside the tp-link media converter there is a IC+ IP102GA, once more with magnetics and socket.
Just found TI TIDA-00928 where they show how to connect a DP83822 phy to either copper or fiber transceiver. Need to study that.

Regards, Dieter

[peter-h]

The STM32 RMII interface is specifically designed for the LAN8742 or similar PHY chips, but obviously you knew that. It is not totally dedicated; there are alternatives
https://www.eevblog.com/forum/microcontrollers/eth-phy-ksz8091rna-versus-lan8742/

I have played with fibre and been selling RSxxx-fibre (multimode only so too slow for ETH speeds) boxes for many years. The key difference from ETH is of course the simplex nature of fibre (unless you go for more exotic multi wavelength stuff) but there must be an easy hack because there are ETH-fibre converters which cost peanuts and probably contain almost nothing

[dietert1]

Meanwhile i found that others wired DP83822 to a STM32H7, too. They wanted to use fiber in an environment with explosives, while i target metrology setups (EMI reduction). The chip version DP83822HF with lower Vdda supply voltage requirement matches those fiber transceivers, e.g. AFBR-5813. I already ordered some parts. The phy driver needs some mods, too. But very little to make it work under lab conditions.
Yes, i considered ripping apart one of those china made media converters, but decided to buy from a distributor instead as it is faster and saves some work. The complete converter can be had for about € 10 while the AFBR-5813 from rs-components will be € 40 a piece.
A bit annoying is the long startup time of the LwIP stack inside my code example, about 10 to 20 seconds until the http and telnet servers are responding. And of course breakpoints break those connections (same as USB). So debugging needs trace buffers and the like for checking flow of execution. And combining "lost and found" components causes overhead. At some point i need to refactor the tasks for better integration.
For the time being it's reading a HP 3478A meter by text commands and bringing back the results on the telnet connection. A satisfying result after some days of work. Need to study yet whether i will have a second 16 bit port on the nucleo board pins for sniffing the GPIB bus, like my STM32L433 controllers do.

Regards, Dieter

[dietert1]

These 1x9 transceiver modules are cheap at aliexpress, but they offer the single fiber variety with SC connector and 1310/1550 nm wavelength. It's 1000FX over one fiber for $ 15 a pair (fiber extra). It needs to be a pair due to the different wavelength for each direction. This can't work with 2x the same device. With single mode fiber this works up to 20 km!

Regards, Dieter

[dietert1]

Meanwhile i made a protoboard with a fiber ethernet interface. Like the Nucleo144 i was using before it has a STM32H743, this time in a 100-pin variant. Then the DP83822 PHY i mentioned above and a fiber transceiver. I also added the RTC crystal and battery. The comparator converts the signal detect output of the transceiver from ECL to CMOS level.
I managed to pin-strap the PHY for 100 MBit full duplex FX mode and RMII interface, so it works without any register programming. The ST supplied example firmware uses only standard PHY registers to execute a software reset and to determine link status and parameters.
Everything appears to be working. Today i got the ethernet part up and running, including the http server. Total supply current is 350 mA at 3.3 V, with the CM7 core running at 400 MHz with 1.2 V supply.

Regards, Dieter

[dietert1]

Today i learned that the STM32H743 1.2 V core supply comes from Vdd = 3.3 V by a LDO regulator inside the MCU. So i added a ISL85415 based step-down converter for the 1.2 V and two 5A silicon diodes as overvoltage protection.
In CubeMX there is a RCC parameter "SupplySource" that became "Power_External_Source_Supply". Then after flashing and the next POR reset the total supply current went down from 350 mA to 220 mA and the MCU temperature dropped to 40°C. Now the PHY is the hot part at about 50 °C.

Regards, Dieter

[julian1]

Interesting idea, I didn't realize one could drive a fibre transceiver from an ethernet PHY.
It is easy to imagine regular ethernet magnetics are spewing EMI all over the place - not good for metrology applications.

I have a board with regular stm32f4 + lan8742 + integrated RJ45 + mongoose test code.
It was a bit surprising to see the lan8742 use > 100mA with nothing configured/ not even the RMII port on the stm32 side.
And >200mA, with ethernet link up.
So maybe about the same as DP83822.

[dietert1]

Yes, i measured 160 pF of common mode coupling by the ethernet magnetics on a STM32H743 board. Of course one could use ferrites to suppress high frequencies. An optical fiber is a good alternative. A fiber media converter is about 10 to 30 €. I got an A/B pair of 1310/1550 nm single fiber converters at aliexpress for 15 € (without fiber and without mains adapters), except they sent 2x the same converter, so that didn't work yet.
I think in my test setup the MCU now takes 30 to 40 mA, the PHY takes 50 to 60 mA and the 100 MHz fiber transceiver takes about 130 mA. It's pretty amazing that the protoboard and the SMD adapters handle the 50 MHz RMII transfers. Maybe i can design a "fiber blackpill".

Regards, Dieter

[peter-h]

I posted a lot a while ago about the LAN8742 current draw, in the context of fast ETH shutdown, to give the CPU the max time to write 512 bytes into an SPI FLASH, after a drop in a +5V rail (feeding a 3.3V LDO for the CPU VCC) is detected from within a 1kHz interrupt.

The 8742 draws a lot of current indeed.

Do people still use fibre a lot? Many years ago I used to make various fibre "modems" (for 232 422 485 etc) and the sales pretty much died when the world went ETH, including factory automation everywhere. ETH is isolated which solves most of the issues. Fibre should still be useful for IS (intrinsic safety) applications but these are rare and "money no object" so that business goes elsewhere. And fibre termination is a PITA, with the 4 digit priced crimp etc tools.

[dietert1]

My personal target is a metrology application. A fiber link provides high bandwidth and better isolation than the usual optocoupler or ethernet magnetics. For my tests i got some ready made 20 m fibers with SC plugs for € 8 to 15. I don't need 20 km long fibers with perfect coupling.
I also got some digital isolator chips with low common mode coupling across the barrier. They use RF transmission inside, so one needs to check EMI.
Currently i am designing a small pcb and doing a code-revision of the ST demo application - maybe due to your continuous rants. Yes, there are lots of nasty details to resolve. Yesterday i implemented some error spotting and found two more anomalies:
- FreeRTOS implements an infinite loop in prvTaskExitError() that may go unnoticed.
- Errors occur in HAL_ETH_ReadData() when trying to fetch incomplete packets or packet chunks. The RxPktSemaphore may not work as intended in ethernetif_input().

Regards, Dieter