MSO5000 waveform data format problem

[quakeman]

Hi,

i wanted to get the signal from my new MS05074 to the PC in a format which i can use for other programs. Therefor i studied a bit the possible formats which can be saved on the scope or transferred via Ethernet.

I liked the WFM format from my old DS1052E because it stores all settings within the header. But at the moment I don’t know the specific locations for the needed informations within the header. And the format seems to have changed compared to the DS1054Z or DS1052E. So I tried the BIN format and after I compared these two I found out that only the header differs.

For the BIN format I only know that the header is 168 bytes long before the data for the samples start and I know the following offsets.

Offset	Content
000-004	Rigol Identifier (RG01H)
044-04D	Record date (2019-03-29)
054-05B	Record time (19:08:46)
064-06A	Modelname (MSO5074)
06C-078	Serialnumber of the model (MS5A2108XXXXX)
0A8-end	1 Byte for each sample

All other Bytes within the header doesn’t change with different settings. So it seems that there is no information about the timebase or voltage scaling.

For the WFM format I only know the offset at which the data for the samples start which is 0x197D. The data format is identical to the BIN format. If somebody has more information on the WFM format for the MSO5000 series I would be happy to know it.

So at the moment I have the problem that with both formats I can’t decode the data into representable data on the PC. The CSV format is not practical because it gets too large and is too slow to store for a larger amount of samples.

Then I tried to get the data from the scope via Ethernet and the LXI interface which started better compared to the approach with the files. I wrote a little python program which grabs the samples within the internal memory.

There I could select an ASCII format, which shows the finished Voltage level for each sample but needed 14 bytes for each sample. So this is only practical for small amount of data but directly usable without doing any math. As an alternative I can select a binary format as Bytes or Words. Because the scope only have an 8Bit ADC i used Bytes to lower the amount of transferred data.

At first I visualized the raw bytes without any scaling or offset to see the waveform and was wondering why it looked a bit different from the screenshot. After I overlayed the transferred signal on the screenshot I saw the difference.



The thicker blue line is copied out of LibreOffice Calc which I used to get the waveform from the raw Bytes. The input signal had a larger y range than the screen would display. Interestingly within the transferred raw Bytes I get a wider voltage range compared to the values seen on the screen.

The problem I had was how to calculate the measured voltage from the sampled Byte. To get a starting point I transferred the same waveform in ASCII and Byte format while the scope was in stop mode. So for each byte I had the corresponding calculated floating point value.

Here is an example for the 500mV/DIV range with an 10x probe.

Byte	Float
000	-2,246298E+00
001	-2,228748E+00
…
127	-1,754920E-02
128	+0,000000E+00
129	+1,754920E-02
…
254	+2,211199E+00
255	+2,228748E+00

It could be seen that the whole range of 255 possible values are used. The value for one Bit is 17,5494mV and the zero line is at Byte 128.

After I did this analysis for a few Voltage ranges I saw that they are not linear scaled to each other. So for the 5V/DIV Range the value for one Bit is not 10 times the value for 500mV/DIV. So I get the values for all possible ranges from a little python script which automatically sets the scope to all possible ranges, gets the waveform and calculates the corresponding value for one Bit and possible max and min values. I attached the result as text file.

So I thought I have what I was looking for and could transfer the desired data directly over Ethernet to my PC and calculate the final waveform. But after a few more tests I found a big problem with my approach. When I try to get the data in ASCII format the scope starts to behave strange at around 110000 samples to be transferred. It sometimes doesn’t start at all and sometimes it works. When I go beyond 110000 it doesn’t transmit them at all. And I see the same behavior when selecting Byte as format. There I can get up to 13800000 Bytes until I see this problem. So it is way more but the scope also doesn’t send the data anymore when I try to get a greater amount of samples.
I don’t know what the problem could be because in the programming spec for the MSO5000 series it is written that I could get as much samples as the current setting is within the scope.

Here are a few console commands to test this behavior with Linux. My scope has the IP address 192.168.2.150 and the scope memory depth is set to 20M.

Stop updating the screen

Code: [Select]

echo ":STOP" | nc -w1 192.168.2.150 5555
Select the raw mode to get all samples and not only the ones shown on the screen

Code: [Select]

echo ":WAV:MODE RAW" | nc -w1 192.168.2.150 5555
Select channel 1

Code: [Select]

echo ":WAV:SOUR CHAN1" | nc -w1 192.168.2.150 5555
Select the wanted format as ASCII

Code: [Select]

echo ":WAV:FORM ASCII" | nc -w1 192.168.2.150 5555or Byte

Code: [Select]

echo ":WAV:FORM BYTE" | nc -w1 192.168.2.150 5555
Select the wanted sample range from 1 to 10000

Code: [Select]

echo ":WAV:STAR 1" | nc -w1 192.168.2.150 5555

Code: [Select]

echo ":WAV:STOP 10000" | nc -w1 192.168.2.150 5555
Get the samples and save them to a file

Code: [Select]

echo ":WAV:DATA?" | nc -w1 192.168.2.150 5555 > /tmp/test.data
As soon as I change "WAV:STOP" to >=110000 for ASCII or >=13800000 for Byte the data is no more transmitted from the scope. I used Wireshark to verify if the data really doesn't get transmitted or maybe I have a problem on the receiving side of my PC but the scope only acknowledges (TCP ACK) the command and than it does nothing.

It would be nice if somebody can try this and tell me if he can reproduce this behavior or not. I don’t think that such a bug exists in the firmware but at the moment I don’t know how to transfer the whole waveform from the scope memory to the PC over Ethernet.

Ciao,
       Rainer

PS:
If somebody is interested he can download my self written Python program from my private GIT repository. The comments are in german but I think it is quite clear what it does.
https://quakeman.mooo.com/gitweb/?p=code/python.git;a=tree;f=Verschiedenes;hb=HEAD

The needed files are MSO5000_Connection.py for the basic connection to the scope and MSO5000_Tools.py for executing different tasks with the scope. At the moment MSO5000_Tools needs the following parameters:

Required

• --path=/temp/test (Path to where the resulting files will be created)
• --ip=192.168.1.2 (IP address of the scope)
• --port=5555 (TCP Port of the scope, normally 5555)

Optional

• --savedata=VCD/CSV (Stores the waveform from all active channels as VCD or CSV as file)
• --testscope (Get test files for “Raw” and “Normal” data as ASCII and BIN files)
• --getscaling (Measure all voltage ranges and calculate Voltage/Bit scalings and max/min values for each range)

[quakeman]

I use the SCPI command ":WAVeform:PREamble?" for the MSO2000A. The command is also available for the MSO5000.

I have no problem when using the SCPI commands. There i could get all information i need but have the arbitrary limit of bytes i can transfer at maximum. I meant the BIN format file which could be saved directly on the scope on an USB stick. There it seems that it has no valuable information in the header compared to the WFM format.

With the MSO2000A, there are limitations when importing the data. A maximum of 250,000 bytes or 15,625 ASCII characters are allowed in one piece. In the manual of the MSO5000 I couldn't find anything.

You have a very good point. I can't find any limit in the manual but maybe i can transfer all the data when i split it up into smaller pieces. With help of the "WAV:START" and "WAV:STOP" i can select the range i want and transfer it into more smaller parts and concatenate it in my program to one big waveform. This was a very good hint and i will try this as soon as i have a little spare time to test it.

Ciao,
      Rainer

[quakeman]

That hasn't left me alone now.

So i updated my python script to get the data in chunks to 10000000 Points for BIN and 100000 for ASCII and now i can transfer the all memory points stored in the scope.
For a larger amount of data like 20M Points the performance to calculate the voltage from the Byte value is not very good so i have to look at this. But it works and now i can store the waveform as CSV or VCD to view it on the PC with GTKwave for example.

I also added the option to store a screenshot to my program with "--screenshot" and fixed a few bugs.

Ciao,
      Rainer

[TopLoser]

https://www.eevblog.com/forum/testgear/upload-a-full-mso7000-la-buffer/

Similar discussion here

[quakeman]

Here is just a short addition to what i have done since my last post.

The python program is working fine now and i added multithreading to do the math for the VCD output format. I have an AMD Threadripper with 16 (32) cores and it does a real good job. With only 1 thread a 200M points dataset from 1 channel needed around ~30 minutes and ~33GB Ram to calculate the final VCD file. With 32 threads i still need that much RAM but only ~55 seconds to calculate the file. But when i do a bit of math i can guess that i need ~132GB Ram for all 4 channels to calculate the corresponding VCD file which is a bit too much. It seems that python is not very conservative while processing a very large amount of data. 

The amount of needed Ram was a show stopper for me so i reimplemented the VCD calculation in pure C and tried again. Now i only need ~5GB of Ram for a 200M points file and 1 channel. I could also calculate for all 4 channels and still "only" need 20GB of Ram. And the math is way faster compared to python. Even with all 4 channels i only need ~5s to calculate the result. Now the time for writing the resulting ~4,5GB VCD file takes longer than the whole calculation.

My python program still is the "master" program which does the communication to the scope and only for the math it calls the C program to do the heavy work.

The updated Python code is still on my private GIT repository i meantioned above. The newly created C program is available at
https://quakeman.mooo.com/gitweb/?p=code/c-x86.git;a=tree;f=src;hb=HEAD

It is still a work in progress but at the moment i have a quite usable code base. Maybe it helps somebody who wants to get the data from the scope.

Ciao,
      Rainer

PS: I'm still in search for the specifications of the WFM data format for the MSO5000 to implement it into my converter.