Rigol DG1000Z RAF file format

[mike47203]

I have a need to generate some tremendously large arbitrary waveforms for my Rigol DG1062Z (with 16M option) and the Rigol Ultra Sigma software just won't get the job done for me. My plan is to convert my waveforms directly into RAF files. I tried using CSV and they do work, but take an incredibly long time to load from a USB drive.

While Rigol provides the file format information there are two CRC's that are not standard calculations and they don't provide any details. I managed to reverse engineer them using Greg Cook's CRC RevEng tool. I just want to document them here in case this information can be of use to anyone else.

http://reveng.sourceforge.net/

You can use a standard CRC-16/CCITT-FALSE algorithm mentioned in the following link and XOR the output with 0x0069 or you can just use 0x1021 for the polynomial and 0xebcc for the intial value instead of the default values.

http://reveng.sourceforge.net/crc-catalogue/16.htm#crc.cat.crc-16-ccitt-false

The CRC values are byte swapped. As such they will not work as is with CRC RevEng unless they are swapped. At least I can't figure out how.

The waveform CRC input data starts at byte 57 and runs to the end of the file
The header waveform is bytes 1 to 50. You have to calculate the waveform CRC first and place it in the header before you calculate the header CRC.

Interesting thing about the CRC is that the function generator doesn't tell you there is a bad CRC, but I get some strange results if it is. Maybe the sample rate is off or there can be a bad sample point. Seems to be fine with a good CRC. There could be something else going on, I haven't experimented with this too much. Maybe a firmware issue.

The Ultra Sigma software will give you an error and won't load the RAF file if the CRC is bad.

I hope to write some code (maybe in Python) to generate this file. When I do I will post it too.

A big thanks to this HACKADY article for pointing me to Greg's tool.
https://hackaday.com/2019/06/27/reverse-engineering-cyclic-redundancy-codes/

[tv84]

Is the CRC16 different from these cases?

https://www.eevblog.com/forum/testgear/rigol-dsxxxx-gel-firmware-file-format/msg1415612/#msg1415612


Edit:

Sorry!

I never added this info to that post but some months ago i also discovered this CRC16, as it's used in many Rigol FWs:

                // check = 0x6809 residue = 0x0000 name = "CRC-16/RIGOL 2"
                0x1021, 0xEBCC, 0, false, false

[mike47203]

Yes. That's it. I never ran across it since I was only searching for information on the RAF files instead of the firmware.

No problem. At least we know it's the same. 

[tv84]

After discovering it the hard way, as you, I saw the algo in decompilation as then it's extremely easy to find the init value in the code.

BTW, did you read?

https://www.eevblog.com/forum/testgear/rigol-dg4000-series-raf-file-format/msg559476/#msg559476

[mike47203]

Take a look at CRC RevEng when you get a chance. It's pretty nice. Was easy to figure out once I realized the bytes were swapped.

I plan to make use of it in the future.

[tv84]

I know it. 

But I've implemented my own search function based on theirs. Gives me flexibility to use in my programs.

BTW, they are not "swapped", they're in little-endian format as most of Rigol FW.

[mike47203]

Sorry, I had to "swap" them when I ran them through the reversing program to make it work. I fed the data in as files and I couldn't find any options that allowed them to stay in little-endian with the rest of the header.

I could be missing something simple in the settings.

[Ben321]

I have a need to generate some tremendously large arbitrary waveforms for my Rigol DG1062Z (with 16M option) and the Rigol Ultra Sigma software just won't get the job done for me. My plan is to convert my waveforms directly into RAF files. I tried using CSV and they do work, but take an incredibly long time to load from a USB drive.

While Rigol provides the file format information there are two CRC's that are not standard calculations and they don't provide any details. I managed to reverse engineer them using Greg Cook's CRC RevEng tool. I just want to document them here in case this information can be of use to anyone else.

http://reveng.sourceforge.net/

You can use a standard CRC-16/CCITT-FALSE algorithm mentioned in the following link and XOR the output with 0x0069 or you can just use 0x1021 for the polynomial and 0xebcc for the intial value instead of the default values.

http://reveng.sourceforge.net/crc-catalogue/16.htm#crc.cat.crc-16-ccitt-false

The CRC values are byte swapped. As such they will not work as is with CRC RevEng unless they are swapped. At least I can't figure out how.

The waveform CRC input data starts at byte 57 and runs to the end of the file
The header waveform is bytes 1 to 50. You have to calculate the waveform CRC first and place it in the header before you calculate the header CRC.

Interesting thing about the CRC is that the function generator doesn't tell you there is a bad CRC, but I get some strange results if it is. Maybe the sample rate is off or there can be a bad sample point. Seems to be fine with a good CRC. There could be something else going on, I haven't experimented with this too much. Maybe a firmware issue.

The Ultra Sigma software will give you an error and won't load the RAF file if the CRC is bad.

I hope to write some code (maybe in Python) to generate this file. When I do I will post it too.

A big thanks to this HACKADY article for pointing me to Greg's tool.
https://hackaday.com/2019/06/27/reverse-engineering-cyclic-redundancy-codes/

Aren't the Rigol waveform files just simple text files? In my previous research, I never before found anything about CRCs at all. No mention that a Rigol device might use a CRC, or anything about it being a binary file. I assumed that it was just a plain text ascii file.

As for your statement "While Rigol provides the file format information", can you point me to where this file format information is stored on the Rigol website? I couldn't find ANYTHING on the format at all. Nothing. So if you know where I can find some official documentation on the file format, PLEASE post the link to it here.

[Ben321]

Hey @mike47203 I REALLY need your help on something. PLEASE help. I posted a reply with quote already, and you still haven't responded with the answer to my question. I have my own Rigol DG1022Z and I NEED YOUR HELP with the thing I asked above.

[nfb]

Here you are.

https://rigol.force.com/support/s/article/what-is-the-file-format-used-with-dg-arbitrary-waveforms2

[Ben321]

Here you are.

https://rigol.force.com/support/s/article/what-is-the-file-format-used-with-dg-arbitrary-waveforms2

Thanks.

[mike47203]

Sorry Ben321. I never saw the notification about your post or I would have helped. Hoped you found what you needed.

[Ben321]

The waveform CRC input data starts at byte 57 and runs to the end of the file
The header waveform is bytes 1 to 50. You have to calculate the waveform CRC first and place it in the header before you calculate the header CRC.

Bytes 1 through 50? Don't you actually mean bytes 0 through 49? In computer stuff like what the byte offset in a file is, those numbers usually start at 0, not at 1. Unless of course you literally do mean byte 1 (the second byte of the file) is where it starts. Also, in the previous sentence you mention byte 57 with no reference to what you consider the first byte, so can I assume that in that sentence you also assume that you consider 1 to be the first byte? Or in that sentence do you consider 0 to be the first byte of the file?

In general, can you please be more clear on where you are starting your numbering of the bytes? I always start my numbering of bytes at 0 whenever I'm describing the structure of a file. I need to know exactly what terminology you are using in your description, if I am to actually write software that implements what you are describing.

Interesting thing about the CRC is that the function generator doesn't tell you there is a bad CRC, but I get some strange results if it is. Maybe the sample rate is off or there can be a bad sample point. Seems to be fine with a good CRC. There could be something else going on, I haven't experimented with this too much. Maybe a firmware issue.

What it's doing is that if the CRC is bad, it assumes that it's a headerless waveform data file. All the remaining bytes of the header it assumes are actually part of the waveform. Yes, my DG1022Z does in fact correctly load headerless waveform files, as samples for it to output, but what it doesn't do is set the configuration for the scope (such as voltage amplitude and frequency settings) and thus it needs to be manually configured in such a case, or it needs to load a separate settings file (which can be made initially simply by manually configuring it correctly and then saving the configuration to a file).

[Ben321]

You can ignore the first part of my above post, I re-found this document https://beyondmeasure.rigoltech.com/acton/attachment/1579/f-0595/1/-/-/-/-/DG%20file%20format%20and%20arbitrary%20waveform%20examples.pdf that explains the header. And by the way, I have discovered that all the multi-byte values used in the header (as well as each 2-byte sample) are little endian. This post https://www.eevblog.com/forum/testgear/rigol-dsxxxx-gel-firmware-file-format/msg1415612/#msg1415612 shows the different CRC specs for the different Rigol devices, and I discovered experimentally that the one I needed for my Rigol DG1022Z (which was not in the list of CRC specs in that post) was the same as the one that list refers to as "CRC-16/RIGOL 2". And the opening post in this thread showed where to calculate the CRCs. The waveform CRC is calculated first, and placed after the LowLevel field in the header. Then another CRC is calculated over the header from its first byte up through the last byte of the waveform CRC just calculated. This is the header CRC, and it's appended to the header right after the waveform CRC. After that, there's 4 bytes all with a value of zero. Then all the remaining bytes are the waveform.

I discovered that if at least one of the CRCs doesn't match, then the entire file is treated as a headerless waveform file. This means there's no settings stored in the RAF file for the funcgen to use, so things like sample-rate or frequency, and the voltage levels, will need to be manually set with the controls, or loaded from a previously saved settings file. This is fine if you wrote software to generate a waveform without a header, but it also means that if a header is present but just has a bad CRC, then the funcgen will mistakenly think that there is no header (it will assume it's a headerless waveform file), so it will treat every byte-pair in the header as a waveform sample. Suffice to say, its actual output analog signal will be incorrect in such a case.

[Ben321]

I also discovered the format for the other signal specification values in the header. They aren't meant to be read and used straight away. They are actually scaled by various constants. When the output mode is PERIOD, the 64bit period/samplerate value represents the period * 10billion. When reading from the file, you need to take the number and divide it by 10billion to get the actual period as measured in seconds. When the output mode is SAMPLE RATE, the 64bit period/samplerate value represents the samplerate * 1million. To get the actual sample rate, as measured in smp/sec you need to take this value and divide it by 1million. As for the 32bit voltage level values, those represent the actual voltage * 10million.  To get the actual voltage levels, you need to take these values and divide by 10million.

I can't find ANY of that documented anywhere. I discovered it experimentally by using Rigol Ultra Station to generate several test files, and then writing my own loader software in Visual Basic, and trying different scale constants for the various values, until I found the ones that produced the correct results. Keep in mind, that these constants are correct when working with RAF files for the DG1022Z (the funcgen I have), but there's no guaranty that they will also be correct when working with RAF files for other Rigol funcgens. Also, the headers for these files don't contain any fields for storing the model number of the funcgen they are for, so for the software to interpret the values in the header correctly when loading an RAF, the software actually needs external input to tell it what device the RAF file had been created for. Even Ultra Station gives you a dropdown menu when you load an RAF file, asking which model number that the file is for, so it can know how to load it properly. That's not very good software engineering practice in my opinion, making a file format that requires user input to even tell the software what kind of file it's supposed to be, but that seems to be the way it is with the Rigol RAF files.

[Ben321]

I finished writing that program now. It loads an RAF file, and assumes it's an RAF file for a Rigol DG1022Z funcgen. It processes the header, does CRC comparisons, and displays the header info (if a header exists, which it determines based on if the HeaderCRC is correct), as well as displaying the waveform. The box on the left contains the header info (if the header exists), or otherwise states that the HeaderCRC check failed and it's assuming no header. The right box contains the waveform (scaled down to fit, if it's wider than the box), and it gets the waveform from after the header if the HeaderCRC check passes, while treating the entire file as waveform data with no header, if the HeaderCRC check fails. I've attached a zipped copy of the EXE file to this post. To use, just extract it from the zip and run it. Note that running it doesn't involve doubleclicking the exe file in this case, because it needs the path to the file to be loaded, in the command line. Either run it from a command prompt with a valid filename as the first commandline argument, or drag-drop an RAF file onto the exe file.

[Njk]

Guys, you're the best. Discovered this thread by chance. BTW the new crop (DG800, DG900, DG2000) supports only the "sample rate" output mode. So that byte is re-purposed for the filter setting specification. In all other respects, the format looks similar.

RAF file format for DG800, DG900, DG2000
______________________________________________
Offset,  Length,   Description.
bytes.   bytes.
______________________________________________
0          4      points number, N (u32, LE)
4          1      1
5          1      0
6          1      low-pass filter setting:  2 - "interpolation", 3 - "step", 4 - "smooth"
7          25      file name, a null-terminated ASCII char string
32        8      sample rate, *10^6 Sa/s (s64, LE)
40        4      high level voltage, *10^7 V (s32, LE)
44        4      low level voltage, *10^7 V (s32, LE)
48        2      points data CRC (u16, LE)
50        2      header CRC (u16, LE)
52        4      0
56       2N      16-bit points data ([s16], LE)

[sken]

Dear Njk, and all,
fist of all, thank you for all this work done!

I have some questions for Njk and anyone who has experience with the DG800: is the RAF file for the DG800 model CRC-encoded as well?
And, what about the CSV header for the DG800: anyone can kindly provide me with an example header? Despite both type of file are mentioned in the manual, there are no specs at all.

Thank you,
Luca

[Njk]

is the RAF file for the DG800 model CRC-encoded as well?

The Rigol Ultra app can produce the files for DG800/900/2000. And the viewer mentioned above does recognize the header as valid. Based on that, I think we can safely assume the CRSs are calculated in exactly similar way.

And, what about the CSV header for the DG800: anyone can kindly provide me with an example header? Despite both type of file are mentioned in the manual, there are no specs at all.

Check this manual. There is.
https://www.eevblog.com/forum/testgear/new-rigol-16-bit-function-generators-dg800900-series/msg5004445/#msg5004445

[tv84]

Any CRC that Rigol might be using is inside this list:

https://www.eevblog.com/forum/testgear/rigol-dsxxxx-gel-firmware-file-format/msg1415612/#msg1415612