ELF to binary for boot loader

[peter-h]

There is a lot of stuff online for ELF to BIN etc. For example arm-none-eabi-objcopy can generate the binary.

But not much on how this actually works.

Let's say you are programming a 32F4 within Cube IDE, using STLINK V2/V3. Does the IDE just feed a binary block to the debugger, to program the FLASH with? The debugger has very little storage it in. It must be just programming the stuff straight in, 0x08000000 onwards.

But there must be a way to specify gaps otherwise if you generated some text for bottom 100k and then top 40k, you would end up with a 1MB binary block, mostly empty. I don't think this is happening. So whatever is managing this process must be generating multiple binary blocks and programming these individually.

Also, AIUI, for single stepping, you need to have a symbol table "somewhere" and it can't be in the CPU FLASH because a 1MB binary could be several MB with symbols.

So the stuff loaded into the CPU must be just the binary block comprising of the sections 'text' and 'data'.

IOW, same as what you would need to generate for a boot loader which just takes a binary block and programs the CPU FLASH with it.

I am trying to work out how to generate a single binary block. The program will never have discontinuous text or data sections, and AFAIK data always follows text.

[langwadt]

There is a lot of stuff online for ELF to BIN etc. For example arm-none-eabi-objcopy can generate the binary.

But not much on how this actually works.

Let's say you are programming a 32F4 within Cube IDE, using STLINK V2/V3. Does the IDE just feed a binary block to the debugger, to program the FLASH with? The debugger has very little storage it in. It must be just programming the stuff straight in, 0x08000000 onwards.

But there must be a way to specify gaps otherwise if you generated some text for bottom 100k and then top 40k, you would end up with a 1MB binary block, mostly empty. I don't think this is happening. So whatever is managing this process must be generating multiple binary blocks and programming these individually.

Also, AIUI, for single stepping, you need to have a symbol table "somewhere" and it can't be in the CPU FLASH because a 1MB binary could be several MB with symbols.

So the stuff loaded into the CPU must be just the binary block comprising of the sections 'text' and 'data'.

IOW, same as what you would need to generate for a boot loader which just takes a binary block and programs the CPU FLASH with it.

I am trying to work out how to generate a single binary block. The program will never have discontinuous text or data sections, and AFAIK data always follows text.

a bin file doesn't have gaps and you need to specify where to put it in memory
an intelhex file can have gaps and specify the address for each line

[SiliconWizard]

And... you can have a look here: https://stackoverflow.com/questions/19458031/how-to-create-a-executable-hex-from-elf-file-format

[peter-h]

I saw that post and thought he was trying to generate a an .exe (a DOS, or windoze command line executable). These have a complicated format, with a load-time fixed-up table of jumps etc.

I am also very familiar with Intel hex. But I don't want to use that, due to the bulk. For a 1MB CPU you would need ~2MB intel hex.

So I guess the objective here is just a string of bytes representing the CPU FLASH, 0x08000000 onwards, as long as there is text+data. This must incidentally be what gets fed to the debugger too.

[SiliconWizard]

I saw that post and thought he was trying to generate a an .exe (a DOS, or windoze command line executable). These have a complicated format, with a load-time fixed-up table of jumps etc.

I am also very familiar with Intel hex. But I don't want to use that, due to the bulk. For a 1MB CPU you would need ~2MB intel hex.

So I guess the objective here is just a string of bytes representing the CPU FLASH, 0x08000000 onwards, as long as there is text+data. This must incidentally be what gets fed to the debugger too.

hex is a very simple format to deal with, much simpler than elf. So point is, you can use objcopy to generate a hex file from a elf file, and then do whatever you find convenient from the hex file. If a hex file is too big, you could either compress it, or convert it to some custom format.

[ataradov]

I think you are confused. IDE issues very simple programming commands for the flash controller to the debugger/programmer, it does not  store anything.

Normally ELF files will be contiguous, unless you go out of your way to make them otherwise. And if you do, you need to come up with your own binary format that includes separate sections.

[ajb]

You can certainly use objcopy to create a binary of the application and hand that off to a bootloader for installation.  You'll want to provide at least a length for it and maybe some other metadata so it's probably a good idea to define some sort of header the bootloader can inspect, and a checksum or cryptographic signature would be a really good idea.  Easy enough to create a script to build all of that into an update package.  There are even tools to script the parsing of source files, so you can do things like extract version information or other metadata from the application files and put that into the update package for easy automated deployment.  Of course if you don't have enough memory available to hold an entire binary image+header in addition to the installed software--whether in some sort of on-chip/soldered-on storage or a removable memory card or USB drive--then you'll need to drip-feed the binary to the bootloader from some other device.  What that other device is would be up to you/your application.

[westfw]

You can certainly use objcopy to create a binary of the application and hand that off to a bootloader for installation.

And you probably should, because the elf format itself is relatively complex and probably subject to changing.

There is also readelf (a program) and libelf (a library for parsing elf files.)

[ataradov]

Parsing ELF just to extract the programming data is trivial. I actually prefer doing that over parsing HEX files, since there is no need to deal with a ton of formatting options and parsing text files.

But for the bootloaders, it makes no sense to directly accept either of them, you really want some dedicated format, possibly with check sums and stuff like that.

[brucehoult]

I am also very familiar with Intel hex. But I don't want to use that, due to the bulk. For a 1MB CPU you would need ~2MB intel hex.

What is the problem with that?

[brucehoult]

Parsing ELF just to extract the programming data is trivial. I actually prefer doing that over parsing HEX files, since there is no need to deal with a ton of formatting options and parsing text files.

Parsing Intel hex is easy. I do it in 50 lines for a simple RV32I emulator here:

https://github.com/brucehoult/trv/blob/main/trv.c

I'm sure I couldn't load an elf file with as little code -- certainly using my self-imposed no headers, no libraries constraint.

I'd be happy to be proven wrong.

[ataradov]

Here is 50 lines for parsing ELF files. And most of this is just structure definitions. The actual parsing is 20 lines.

Code: [Select]

typedef struct
{
  uint8_t      e_ident[16];
  uint8_t      e_type;
  uint16_t     e_machine;
  uint16_t     e_version;
  uint32_t     e_entry;
  uint32_t     e_phoff;
  uint32_t     e_shoff;
  uint32_t     e_flags;
  uint16_t     e_ehsize;
  uint16_t     e_phentsize;
  uint16_t     e_phnum;
  uint16_t     e_shentsize;
  uint16_t     e_shnum;
  uint16_t     e_shstrndx;
} elf_ehdr_t;

typedef struct
{
  uint32_t     sh_name;
  uint32_t     sh_type;
  uint32_t     sh_flags;
  uint32_t     sh_addr;
  uint32_t     sh_offset;
  uint32_t     sh_size;
  uint32_t     sh_link;
  uint32_t     sh_info;
  uint32_t     sh_addralign;
  uint32_t     sh_entsize;
} elf_shdr_t;

bool read_elf(uint8_t *data)
{
  elf_ehdr_t *hdr = (elf_ehdr_t *)data;
  elf_shdr_t *shdr = (elf_shdr_t *)&data[hdr->e_shoff];

  if (hdr->e_type != 2/*ET_EXEC*/ || hdr->e_version != 1/*EV_CURRENT*/ || hdr->e_machine != 40/*EM_ARM*/)
    return false;

  for (int i = 0; i < hdr->e_shnum; i++)
  {
    if (shdr[i].sh_type == 1/*SHT_PROGBITS*/ && (shdr[i].sh_flags & 2/*SHF_ALLOC*/))
    {
      printf("program %d bytes from file offset %d into memory address %d (0x%08x)\n",
          shdr[i].sh_size, shdr[i].sh_offset, shdr[i].sh_addr, shdr[i].sh_addr);
    }
  }

  return true;
}

Obviously replace printf() with the actual data extraction and programming functions.

This does not verify input data and valid ELF file is assumed.

And of course replace 40 (EM_ARM) in e_machine check with an actual target machine type. Or just remove the check and assume that ELF file is valid for the used processor.

[peter-h]

You can certainly use objcopy to create a binary of the application and hand that off to a bootloader for installation.  You'll want to provide at least a length for it and maybe some other metadata so it's probably a good idea to define some sort of header the bootloader can inspect, and a checksum or cryptographic signature would be a really good idea.  Easy enough to create a script to build all of that into an update package.  There are even tools to script the parsing of source files, so you can do things like extract version information or other metadata from the application files and put that into the update package for easy automated deployment.  Of course if you don't have enough memory available to hold an entire binary image+header in addition to the installed software--whether in some sort of on-chip/soldered-on storage or a removable memory card or USB drive--then you'll need to drip-feed the binary to the bootloader from some other device.  What that other device is would be up to you/your application.

That's exactly what I need to do - many thanks. Just a straight binary image of the CPU FLASH which can be programmed into it directly.

Sounds like objcopy -O binary.dat will do this.

Then I will need to get a little executable written which packages that binary block with a length value, a CRC and maybe some crypto hash. Has anyone done this within Cube? AIUI it needs to be written in Java, unless you are happy for a Windows-only setup.

[gf]

But there must be a way to specify gaps otherwise if you generated some text for bottom 100k and then top 40k, you would end up with a 1MB binary block, mostly empty. I don't think this is happening.

Exactly this is happening. My understanding is that objcopy copies all loadable sections in the .elf file to the .bin file. Gaps betwen sections are filled. Non-loadable sections (like debug, etc.) are not copied.

       objcopy can be used to generate a raw binary file by using an output
       target of binary (e.g., use -O binary).  When objcopy generates a raw
       binary file, it will essentially produce a memory dump of the contents
       of the input object file. All symbols and relocation information will
       be discarded. The memory dump will start at the load address of the
       lowest section copied into the output file.

[brucehoult]

Here is 50 lines for parsing ELF files. And most of this is just structure definitions. The actual parsing is 20 lines.

Thanks, I'll give that a try.

I assume there's supposed to be padding after e_type and e_version? Otherwise all the 16 bit and 32 bit fields are unaligned. It woul dbe better if padding had explicit fields.

This can be handy for elf-in-ROM on small machines, or memory-mapped on big machines. Seems like potential for trouble if it wants to unpack over the top of the place you read it into...

[peter-h]

"Exactly this is happening. My understanding is that objcopy copies all loadable sections in the .elf file to the .bin file. Gaps betwen sections are filled. Non-loadable sections (like debug, etc.) are not copied."

Fortunately this isn't a problem in my project.

I do have a bit of code which is linked to execute in RAM, halfway up ie 0x20010000, but it should not be loading that at that address.

It was discussed here, and is working fine
https://www.eevblog.com/forum/microcontrollers/how-to-create-elf-file-which-contains-the-normal-prog-plus-a-relocatable-block/new/#new

"IDE issues very simple programming commands for the flash controller to the debugger/programmer, it does not  store anything."

OK; that makes sense. That's why the more expensive debuggers claim to run faster. The debugger is probably programming every byte as a separate transaction. Although one should still be limited by the FLASH programming speed. STLINK V3 does appear to program a bit faster than V2. I think the only real advantage of a fast debugger is with stuff like breakpoints with an ignore count; these trigger a breakpoint on every hit and a fast debugger will be able to process this faster so slowing down the target less.

Is there any way to incorporate objcopy -O binary.dat into the Cube script, so that along the .elf (which is used by the debugger) it pops out the binary?

[ataradov]

I assume there's supposed to be padding after e_type and e_version? Otherwise all the 16 bit and 32 bit fields are unaligned.

Newer versions of the standard define e_type as uint16_t and e_version as uint32_t. Makes sense, I guess. This code is pretty old, I don't even remember where I've got this layout from.

This can be handy for elf-in-ROM on small machines, or memory-mapped on big machines.

Even stripped ELF files contain a lot more stuff, so if you are optimizing for size, it may not be the best idea to use ELF for anything.

Another issue is that for RAM code you don't really get a good info. ELF is not designed to be converted into binary, it was designed to be loaded as is. So even with this code you get output like this if you have RAM functions:

program 6884 bytes from file offset 65536 into memory address 4194304 (0x00400000)
program 4 bytes from file offset 72420 into memory address 4201188 (0x00401ae4)
program 4 bytes from file offset 72424 into memory address 4201192 (0x00401ae8)
program 112 bytes from file offset 131072 into memory address 541065216 (0x20400000)

0x00400000 in this case is the flash offset and 0x20400000 is SRAM. As you can see, the last allocated section is in the RAM, but the actual location of the data is in the flash.

The way objcopy handles this is it takes all the parts within the section (as defined by SECTIONS part of the linker script) and concatenates them together without filling the gaps. So in this case to simulate objcopy you would ignore the address and copy the data sequentially.

The only time objcopy fills the gaps is when there are multiple sections that contain loadable data. Those are the sections defined by the MEMORY part of the linker script. To extract that info add this:

Code: [Select]

typedef struct
{
  uint32_t     p_type;
  uint32_t     p_offset;
  uint32_t     p_vaddr;
  uint32_t     p_paddr;
  uint32_t     p_filesz;
  uint32_t     p_memsz;
  uint32_t     p_flags;
  uint32_t     p_align;
} elf_phdr_t;
..........

  elf_phdr_t *phdr = (elf_phdr_t *)&data[hdr->e_phoff];

  for (int i = 0; i < hdr->e_phnum; i++)
  {
    printf("--- %d\n", i);
    printf("p_type       : 0x%08x\n", phdr[i].p_type);
    printf("p_offset     : 0x%08x\n", phdr[i].p_offset);
    printf("p_vaddr      : 0x%08x\n", phdr[i].p_vaddr);
    printf("p_paddr      : 0x%08x\n", phdr[i].p_paddr);
    printf("p_filesz     : 0x%08x\n", phdr[i].p_filesz);
    printf("p_memsz      : 0x%08x\n", phdr[i].p_memsz);
    printf("p_flags      : 0x%08x\n", phdr[i].p_flags);
    printf("p_align      : 0x%08x\n", phdr[i].p_align);
  }

Although it looks like it may be relying on the physical address from the program headers as well to convert things:

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  LOAD           0x010000 0x00400000 0x00400000 0x01aec 0x01aec R E 0x10000
  LOAD           0x020000 0x20400000 0x00401aec 0x00070 0x081ac RWE 0x10000

 Section to Segment mapping:
  Segment Sections...
   00     .text .init .fini
   01     .data .bss

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 1] .text             PROGBITS        00400000 010000 001ae4 00  AX  0   0 64
  [ 2] .init             PROGBITS        00401ae4 011ae4 000004 00  AX  0   0  4
  [ 3] .fini             PROGBITS        00401ae8 011ae8 000004 00  AX  0   0  4
  [ 4] .data             PROGBITS        20400000 020000 000070 00 WAX  0   0  4

Here 0x00401aec is the address of the RAM functions (part of the .data segment) in the flash. If you just go by "Section Headers", then you will have a huge gap. But "Program Headers" alone do not have enough information.

[nctnico]

But there must be a way to specify gaps otherwise if you generated some text for bottom 100k and then top 40k, you would end up with a 1MB binary block, mostly empty. I don't think this is happening.

Exactly this is happening. My understanding is that objcopy copies all loadable sections in the .elf file to the .bin file. Gaps betwen sections are filled. Non-loadable sections (like debug, etc.) are not copied.

AFAIK you need to tell objcopy to fill the gaps. There is an option for that as well which also specifies the byte value to use.

[ataradov]

That's why the more expensive debuggers claim to run faster.

They claim it to justify the price. Past certain point (at least using USB HS for transfers) there is no real difference. Most of the time is spent waiting for the flash to program anyway.

The debugger is probably programming every byte as a separate transaction.

No, because there are limitations on how you can program the flash. A debugger does the same exact things as you would from the code. So you can read the device datasheet and know exactly what debugger is doing. And it is not really a debugger doing those things, but the IDE. A debugger is just a pipe.

Some debuggers have more complicated logic they can run on the device itself. This is helpful for devices with poorly designed flash controllers (like ST) where fast programming is a time-sensitive operation and if the data is not delivered in time, the whole operation fails.

Is there any way to incorporate objcopy -O binary.dat into the Cube script, so that along the .elf (which is used by the debugger) it pops out the binary?

No idea. Usually it is done through post-build events.

[ajb]

Is there any way to incorporate objcopy -O binary.dat into the Cube script, so that along the .elf (which is used by the debugger) it pops out the binary?

Cube IDE is based on eclipse, so unless they've really munged it up there should be a 'post build' section in the build configuration where you can tell it to invoke whatever external process you want.  Off the top of my head I don't remember where exactly. You can throw the objcopy command in there, and even invoke your packager from there was well. Post build steps are per build configuration, so you can have this all set up on a 'release' build config and leave your debug config as is since you probably don't need it there.

I'm not a huge fan of python in general but it's well suited to these sorts of things, being cross platform and having a ton of libraries. You could probably do a basic packager in <20 lines including hashing.

[gf]

But there must be a way to specify gaps otherwise if you generated some text for bottom 100k and then top 40k, you would end up with a 1MB binary block, mostly empty. I don't think this is happening.

Exactly this is happening. My understanding is that objcopy copies all loadable sections in the .elf file to the .bin file. Gaps betwen sections are filled. Non-loadable sections (like debug, etc.) are not copied.

AFAIK you need to tell objcopy to fill the gaps. There is an option for that as well which also specifies the byte value to use.

$ objcopy -O binary a.out a.bin

$ ls -l a.bin
-rwxr-xr-x 1 fuer users 8288 Jul 31 09:35 a.bin

$ objdump -h a.out

a.out:     Dateiformat elf64-x86-64

Sektionen:
Idx Name          Size      VMA               LMA               File off  Algn
  0 text1         00000007  0000000000003000  0000000000003000  00003000  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 text2         00000007  0000000000005000  0000000000005000  00005000  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  2 .eh_frame     00000058  0000000000005008  0000000000005008  00005008  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .comment      00000029  0000000000000000  0000000000000000  00005060  2**0
                  CONTENTS, READONLY

The size of a.bin is 8288 = LMA(.eh_frame) + SIZEOF(.eh_frame) - LMA(text1).
Obviously it does keep and fill the gap between text1 and text2, and between text2 and .eh_frame, preserving the offsets beween the copied sctions' load addresses also in the .bin file.

[peter-h]

Thank you everyone. This is great. Yes the post build steps can be found under Project / Properties



I am going to put a job up on freelancer.com to get a win32 command line utility done which appends four crc32 bytes, little-endian, to a file supplied on a command line I see some other IDE (IAR, I think) offers this as a built-in function. Or maybe it's been done before... well it must have been.

[brucehoult]

I am going to put a job up on freelancer.com to get a win32 command line utility done which appends four crc32 bytes, little-endian, to a file supplied on a command line I see some other IDE (IAR, I think) offers this as a built-in function. Or maybe it's been done before... well it must have been.

On Mac or Linux the following ...

Code: [Select]

perl -e 'print pack("V",int($ARGV[0])),"\n"' $(cksum foo) >>foo

... will literally append crc32 to foo, little endian. As in modify the given file. If you want big endian change "V" (Vax) to "N" (Network).

On Windows you can (and should) install Linux in WSL.

[abyrvalg]

A Python tool appending CRC32 matching STM32 hw CRC:

Code: [Select]

#!/usr/bin/python3
'''CRC32 matching STM32 hardware algo'''

_table = {}
_poly = 0x04C11DB7
for i in range(256):
    c = i << 24

    for j in range(8):
        c = (c << 1) ^ _poly if (c & 0x80000000) else c << 1

    _table[i] = c & 0xffffffff


def crc32stm(buf, prev=0xFFFFFFFF):
    crc = prev
    for i in range(0, len(buf), 4):
        crc = ((crc << 8) & 0xffffffff) ^ _table[(crc >> 24) ^ buf[i+3]]
        crc = ((crc << 8) & 0xffffffff) ^ _table[(crc >> 24) ^ buf[i+2]]
        crc = ((crc << 8) & 0xffffffff) ^ _table[(crc >> 24) ^ buf[i+1]]
        crc = ((crc << 8) & 0xffffffff) ^ _table[(crc >> 24) ^ buf[i]]
    return crc


from sys import argv, exit
from struct import pack


if len(argv)!=2:
    exit("Usage: "+argv[0]+" <file>")

with open(argv[1], 'rb+') as fp:
    crc = crc32stm(fp.read())
    fp.write(pack('<I', crc))

To use in Cube:
- install Python 3 (enable "install launcher" and "add to PATH")
- add post-build command: python crc32stm.py yourbinary.bin   

[brucehoult]

Oh crud .. cksum for some reason appends the length of the file to the data before calculating the CRC. That sucks.