Author Topic: Download speed from Rigol DS1054Z or similar oscilloscope to a PC  (Read 6146 times)

0 Members and 2 Guests are viewing this topic.

Offline alm

  • Super Contributor
  • ***
  • Posts: 2903
  • Country: 00
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #25 on: November 17, 2022, 11:37:30 pm »
I don't know how to apply read/write_termination before opening the instrument.

If I open the instrument first, by default the driver sends '*CLS' without the 0x0a terminator, which hangs the oscilloscope communication, and can recover only by a power cycle.
Maybe calling dso._write('\n') right after instantiating the DS1104Z object would remove the hang? This would send the termination character the scope is waiting for.

However, the read/write_termination settings seems to apply only for ivi functions.  For example, if I use osc._ask_raw(':WAV:DATA?\n'), then I must add the 0x0a manually, as '\n'.
Looking at the PyVISA source, write adds a termination character, and write_raw not. So it's not a matter of ivi vs non-ivi functions, but of write_raw() not adding termination, because you're saying you want to write verbatim what you send. Does it work if you use dso._write(':WAV:DATA?') and then output = dso._read_raw()? ask is nothing more than write + read.

I had to look more into the ivi sources.  Maybe there is a way to set the automatic addition of the 0x0a terminator in pyvisa-py, so I won't end with a mixture of functions where some requires 0x0a while others don't.
Setting dso._interface.instrument.write_termination already sets the terminator in PyVISA, not ivi. dso._interface.instrument is the PyVISA resource.

The transfer speed is 65 seconds to fetch 24MSa.

Not bad, but still 3 times slower than the 22 seconds I get when downloading with https://github.com/morgan-at-keysight/socketscpi
I wonder if it would be possible to use 'socketscpi' instead of 'pyvisa', or maibe to configure 'pyvisa' to use 'socketscpi' instead of 'pyvisa-py'.
Changing Python-ivi to use socketscpi would probably be a big change. What you could consider is just like python-ivi currently supports using python-vxi11 directly without going through PyVISA, you could also add support for calling the python socket library directly for ::SOCKET resources without going through PyVISA. This would involve writing an instrument class around the socket library like the PySerial interface, and some minor changes to ivi.py around hereto add a case where it instantiates the newly created PySocket instrument instead of the vxi11 of PyVISA instruments.
 
The following users thanked this post: RoGeorge

Online RoGeorgeTopic starter

  • Super Contributor
  • ***
  • Posts: 6628
  • Country: ro
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #26 on: November 18, 2022, 09:08:24 am »
Talking about configuring PyVISA to use 'socketscpi' as its LAN back-end because of this recent commit into socketscpi, commit that changed the old names of the socketscpi functions in order to match the pyvisa syntax, https://github.com/morgan-at-keysight/socketscpi/commit/3d8bea9b012c03a29d6676b60a21ac7e5a655ae4.

By the commit description, 'socketscpi' might be already usable as a backend for PyVISA, just like now it is possible to force PyVISA to use 'pyvisa-py' instead of proprietary ni-visa shared libs (when both ni-visa and pyvisa-py are installed) like in this example from https://pyvisa.readthedocs.io/projects/pyvisa-py/en/latest/index.html
Quote
You can select the PyVISA-py backend using @py when instantiating the visa Resource Manager:
Code: [Select]
# pyvisa usage example
import pyvisa as visa

# rm = visa.ResourceManager()       # will look for proprietary ni-visa or alike first, if installed
rm = visa.ResourceManager('@py')    # will ignore any installed VISA libs, and use pyvisa-py instead

inst = rm.open_resource('<VISA_format_resource_of_an_instrument_to_open_here>')

print(inst.query("*IDN?"))

I don't know how to tell PyVISA to use socketscpi instead of pyvisa-py, or if socketscpi is ready to use as a pyvisa backend or not.  Will try to learn that by digging through sources.  Does this look feasible as a fix (python-ivi with prefer pyvisa  -->  pyvisa with forced backend socketscpi  -->  socketscpi using python's SOCKET)?
« Last Edit: November 18, 2022, 09:10:31 am by RoGeorge »
 

Online RoGeorgeTopic starter

  • Super Contributor
  • ***
  • Posts: 6628
  • Country: ro
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #27 on: November 18, 2022, 01:34:32 pm »
Got 24MSamples in 22 seconds, with python-ivi, pyvisa, pyvisa-py and SOCKET connection!!!   :scared:
:-DMM

Found what was making 'pyvisa-py' 3 times slower than 'socketscpi' by looking side by side at the opening socket parameters in 'pyvisa-py' vs 'socketscpi'.  It was the default value of the TCP_NODELAY.  The download becomes fast if I add this extra line
Code: [Select]
            self._set_tcpip_nodelay(constants.VI_ATTR_TCPIP_NODELAY, True)
at the end of the '_connect()' definition, inside the file lib/python3.10/site-packages/pyvisa_py/tcpip.py.  It's the line number five hundred and something in the modules installed with pip, probably would have to be inserted after line 840 if the same change were to be added in the github source file:
https://github.com/pyvisa/pyvisa-py/blob/main/pyvisa_py/tcpip.py#L840


However, I suspect there might be a way to set the nodelay parameter as True, but I don't know enough Python and/or OOP to figure myself the syntax for setting TCPIP NODELAY from the end user code, without changing the sources inside pyvisa-py files.

Any idea how to set the nodelay parameter using python-ivi?

Offline alm

  • Super Contributor
  • ***
  • Posts: 2903
  • Country: 00
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #28 on: November 18, 2022, 09:23:53 pm »
Talking about configuring PyVISA to use 'socketscpi' as its LAN back-end because of this recent commit into socketscpi, commit that changed the old names of the socketscpi functions in order to match the pyvisa syntax, https://github.com/morgan-at-keysight/socketscpi/commit/3d8bea9b012c03a29d6676b60a21ac7e5a655ae4.

I don't know how to tell PyVISA to use socketscpi instead of pyvisa-py, or if socketscpi is ready to use as a pyvisa backend or not.  Will try to learn that by digging through sources.  Does this look feasible as a fix (python-ivi with prefer pyvisa  -->  pyvisa with forced backend socketscpi  -->  socketscpi using python's SOCKET)?
I know that this might be a red herring at this point, but when I said earlier that I thought integrating socketscpi with PyVISA, I was confusing socketscpi with another library, so my statement that it would be hard was wrong. It does not look super hard, but would require looking at how PyVISA-py registers itself with PyVISA, and doing the same with a class that wraps socketscpi.

Found what was making 'pyvisa-py' 3 times slower than 'socketscpi' by looking side by side at the opening socket parameters in 'pyvisa-py' vs 'socketscpi'.  It was the default value of the TCP_NODELAY.  The download becomes fast if I add this extra line
Code: [Select]
            self._set_tcpip_nodelay(constants.VI_ATTR_TCPIP_NODELAY, True)
at the end of the '_connect()' definition, inside the file lib/python3.10/site-packages/pyvisa_py/tcpip.py.  It's the line number five hundred and something in the modules installed with pip, probably would have to be inserted after line 840 if the same change were to be added in the github source file:
https://github.com/pyvisa/pyvisa-py/blob/main/pyvisa_py/tcpip.py#L840
Congratulations on finding this! It makes sense that it's a socket option that has such a performance impact. It suggests me that there's something sub-optimal in the Rigol scope about assembling the data into TCP packets, but this is a good workaround.

Any idea how to set the nodelay parameter using python-ivi?
It took a bit of digging, but I think this should work:
dso._interface.instrument.visalib.sessions[dso._interface.instrument.session]._set_tcpip_nodelay(...)
 
The following users thanked this post: RoGeorge

Online RoGeorgeTopic starter

  • Super Contributor
  • ***
  • Posts: 6628
  • Country: ro
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #29 on: November 19, 2022, 12:04:57 am »
Any idea how to set the nodelay parameter using python-ivi?
It took a bit of digging, but I think this should work:
dso._interface.instrument.visalib.sessions[dso._interface.instrument.session]._set_tcpip_nodelay(...)

Wow, thanks, I've tried today a thousand combinations and couldn't find any that works.  Now it gets all the 24 million samples in 22 seconds, connected like this:
Code: [Select]
import ivi

# 24Msa in 22 seconds
ivi.set_prefer_pyvisa()

dso = ivi.rigol.rigolDS1104Z(resource='TCPIP0::192.168.1.3::5555::SOCKET', id_query=False, reset=False)

from pyvisa import constants
dso._interface.instrument.visalib.sessions[dso._interface.instrument.session]._set_tcpip_nodelay(constants.VI_ATTR_TCPIP_NODELAY, True)

# 24Msa in 16 minutes
# dso = ivi.rigol.rigolDS1104Z(resource='TCPIP0::192.168.1.3::INSTR', id_query=False, reset=False)




Don't celebrate yet, (re)found one more issue!   ;D

- A (telnet) data transfer will drop if inside any data packets it receives a 0x00.  This is some RFC spec (for text Telnet IIRC).
- The ADC samples are coming as bytes, which means the incoming data packets will be truncated if one of the bytes is zero, because in the Telnet RFC 0x00 is considered a terminator, and will drop any bytes coming after a 0x00.
- This issue is only observed when the input signal in the ADC is less than 5 divisions on the screen (ADC outputs 0x7f for zero volts, and 0x00 for any input voltage that is -5*volts/div or under)
- Both the pyvisa->pyvisa-py->socket and the socketscpi are affected

Don't know how to overcome this, or if it is possible at all to open some other kind of socket connection, one that doesn't consider 0x00 as a terminator.  :-//




From when I've written my own SCPI scripts, I had to modify the Python's telnet module such that it won't end a data transfer at the first 0x00 in a data stream.  Since the 0x00 as a terminator is part of the RFC854 spec, this modification can not be merged upstream, so I've just made a local copy of the telnet.py (module) file, and modified the local copy to disregard 0x00 as terminator (commented out those two "if c==" lines:  https://github.com/RoGeorge/DS1054Z_screen_capture/blob/master/telnetlib_receive_all.py
Code: [Select]
        try:
            while self.rawq:
                c = self.rawq_getchar()
                if not self.iacseq:
                    #if c == theNULL:
                    #    continue
                    #if c == "\021":
                    #    continue
                    if c != IAC:
                        buf[self.sb] = buf[self.sb] + c
                        continue
                    else:
                        self.iacseq += c
                elif len(self.iacseq) == 1:

Not elegant, but it worked.  The plan was to remove the telnetlib.py module later, and use a SOCKET connection instead of a Telnet one, though never implemented that/




TL;DR
So far PythonIVI using SOCKET and NO_DELAY is the fastest one, but it only works if the oscilloscope trace is nicely contained inside the screen.

Do you happen to know any SOCKET parameter, so to transfer a specified number of bytes until all bytes were transferred (or timeout has occurred), but for binary, such that 0x00 is not considered a data terminator?
« Last Edit: November 19, 2022, 07:30:15 am by RoGeorge »
 

Offline lundmar

  • Frequent Contributor
  • **
  • Posts: 441
  • Country: dk
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #30 on: November 19, 2022, 02:41:58 am »
RoGeorge,

FYI - liblxi also supports connecting to instruments using raw sockets. Simply use 'RAW' instead of 'VXI11' in the lxi_connect() call.

RAW is faster because it does not suffer the overhead of the quite heavy VXI11 (SUN OneRPC) protocol.

Soon liblxi will also support HiSLIP which basically brings the same speed as RAW but with the benefit of some instrument relevant protocol features.

I plan to release the HiSLIP feature start next year.
https://lxi-tools.github.io - Open source LXI tools
https://tio.github.io - A simple serial device I/O tool
 
The following users thanked this post: RoGeorge

Online RoGeorgeTopic starter

  • Super Contributor
  • ***
  • Posts: 6628
  • Country: ro
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #31 on: November 19, 2022, 09:34:37 am »
Tried liblxi with TCP RAW, and I don't know how to handle the transfers.

- it puts consecutive lxi_send() strings in the same data packet, which the oscilloscope won't understand.  I don't know how to flush the transmit buffer after each lxi_send(), so as a workaround I'm adding an *OPC? at the end of each command.

- even so, when the oscilloscope sends the first 250 000 data bytes, the lxi_receive() returns only the payload of the first TCP packet, which is 1460 bytes, while the oscilloscope continues to send more packets.  I was expecting for the lxi_receive() to concatenate the payload of all the incoming TCP packets, and to stop only when the whole 250000 requested bytes are received (or if a timeout occurs), but lxi_receive() returns after the first TCP packet.  Not sure if this is a bug or it's the NO_BLOCKING setting of a TCP RAW connection.

I don't know how to send TCP_BLOCKING (or alike) to the socket, or if such a feature exists in a liblxi TCP RAW connection.

Code: [Select]
// liblxi git at https://github.com/lxi-tools/liblxi
// to install it from the ubuntu repository (lxilib-dev is in most of the Linux distros repos and in FreeBSD):
//      sudo apt install liblxi-dev

// headers in   /usr/include/lxi.h
// .so lib in   /usr/lib/x86_64-linux-gnu/liblxi.so
// display all installed files with 'dpkg -L liblxi-dev' or
//      sudo apt install apt-file
//      sudo apt-file update
//      (sudo???) apt-file list liblxi-dev
// no need to append LIBRRY_PATH or LD_LIBRARY_PATH

// man lxi_<TAB><TAB>
//      lxi_connect  lxi_discover  lxi_init  lxi_send  lxi_disconnect  lxi_discover_if  lxi_receive

// compile and run this 'demo.c' from a Linux terminal (!!! -l options _must_ be positioned after the .c sources, not before)
//      cd /home/muuu/wd8TB/2019/Z/_hobby/__Pnnn/221103___VNA_DG4102_DS1054Z_python/sw/C
//      gcc -Werror -Wfatal-errors ./dl_24M_ADC_samples_ds1054z.c -llxi -ltirpc && time ./a.out

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <lxi.h>

int main()
{
    printf("liblxi w DS1054Z using TCP mode RAW");
    // 12 bytes preamble + max of 24 mil bytes of ADC samples + 1 byte for the end of string terminator
    #define CHARS_BUFFER_MAX    24000000L // in specs max Rx in a single chunk is 250k chars for Rigol DS1054Z

    char *rx24MB = NULL;                                    // declare a pointer to char, and initialize it with NULL
    rx24MB = malloc(CHARS_BUFFER_MAX * sizeof *rx24MB);     // allocated memory on the heap for rx24MB[CHARS_BUFFER_MAX]
    if (!rx24MB) {                                          // !!! always check the memory was allocated successfully !!!
        fputs ("ERROR: allocation memory failed for the 'rx24MB' buffer, exiting.", stderr);
        exit (EXIT_FAILURE);
    }

    int device, length, timeout = 2000;     // timeout is in ms
    char response[1000];    // allocated on the stack, segmentation fault if too large, Linux max stack is 8MiB

    //char *command = "*IDN?";   // with or without \n, the osc responded in a VXI11 connection
    char *command = "*IDN?\n";
   
    lxi_init(); // Initialize LXI library
    //device = lxi_connect("192.168.1.3", 0, "inst0", timeout, VXI11);        // instument responded to either *IDN? or *IDN?\n
    // device = lxi_connect("192.168.1.3", 0, NULL, timeout, VXI11);           // Connect to LXI device
    device = lxi_connect("192.168.1.3", 5555, "inst0", timeout, RAW);       // must end in \n, or else *IDN? timeout as incomplete
    // device = lxi_connect("192.168.1.3", 5555, "TCP", timeout, RAW);       // *IDN? Timeout err for timeout 2000
    // device = lxi_connect("192.168.1.3", 0, "inst0", timeout, RAW);       // NOPE
    // device = lxi_connect("192.168.1.3", 5555, "inst0", timeout, HISLIP);       // *IDN? garbled chars for timeout 2000
   
    lxi_send(device, command, strlen(command), timeout);                // Send SCPI command ("*IDN?")
    lxi_receive(device, response, sizeof(response), timeout);           // Rx until "\n", or sizeoff(response) chars, or timeout

    printf("%s\n", response);
    // exit if no DS1054Z
// exit(0);


    // enable only ch1
    command = ":CHAN1:DISP 1;*OPC?\n";
    lxi_send(device, command, strlen(command), timeout);
    lxi_receive(device, response, sizeof(response), timeout);           // Rx until "\n"?, or sizeoff(response) chars, or timeout

    command = ":CHAN2:DISP 0;*OPC?\n";
    lxi_send(device, command, strlen(command), timeout);
    lxi_receive(device, response, sizeof(response), timeout);

    command = ":CHAN3:DISP 0;*OPC?\n";
    lxi_send(device, command, strlen(command), timeout);
    lxi_receive(device, response, sizeof(response), timeout);

    command = ":CHAN4:DISP 0;*OPC?\n";
    lxi_send(device, command, strlen(command), timeout);
    lxi_receive(device, response, sizeof(response), timeout);

// ??? here to insert ":RUN""
    // set acquisition mode to max 24_000_000 points memory depth
    // "auto", 12K, 120K, 1.2M, 12M, 24M"
    long mdep = 24000000L;

    command = ":RUN;*OPC?\n";
    lxi_send(device, command, strlen(command), timeout);
    lxi_receive(device, response, sizeof(response), timeout);

    // sprintf(command, ":ACQ:MDEP %li", mdep);
    // sprintf(command, ":ACQ:MDEP %li%s", mdep, "\0");
    command = ":ACQ:MDEP 24000000;*OPC?\n";
    lxi_send(device, command, strlen(command), timeout);
    lxi_receive(device, response, sizeof(response), timeout);

    // wait for triggered
    do {
        command = ":TRIGger:STATus?\n";
        lxi_send(device, command, strlen(command), timeout);
        lxi_receive(device, response, sizeof(response), timeout);
    } while (response[0] != 'T');

    // then wait for ADC to aquire enough data
    clock_t stop_time;
    printf("wait %ld\n", clock());
    stop_time = clock() + 30000;             //30 seconds
    printf("beep %ld\n", stop_time);
    while (clock() < stop_time) ;
    printf("go %ld\n", clock());

    // go to stop mode and prepare to dl 24MSa
    command = ":STOP;*OPC?\n";
    lxi_send(device, command, strlen(command), timeout);
    lxi_receive(device, response, sizeof(response), timeout);

    command = ":WAV:SOUR CHAN1;*OPC?\n";
    lxi_send(device, command, strlen(command), timeout);
    lxi_receive(device, response, sizeof(response), timeout);

    command = ":WAV:MODE RAW;*OPC?\n";
    lxi_send(device, command, strlen(command), timeout);
    lxi_receive(device, response, sizeof(response), timeout);

    command = ":WAV:FORM BYTE;*OPC?\n";
    lxi_send(device, command, strlen(command), timeout);
    lxi_receive(device, response, sizeof(response), timeout);

// char c2[80] = {0};
char c2[80];
// loop to dl all samples in chunks of (max) 250_000 samples
    // set start-stop indexes for next chunk
    long rx_len, chunk_len;
    // long chunk_start, chunk_stop, chunk_size =  125000L;    //dl 24Msa/134s
     long chunk_start, chunk_stop, chunk_size =  250000L;    //dl 24Msa/86s  max from specs
    // long chunk_start, chunk_stop, chunk_size =  750000L;    //dl 24Msa/65s
    // long chunk_start, chunk_stop, chunk_size = 1000000L;    //dl 24Msa/56s  preferred ???
    // long chunk_start, chunk_stop, chunk_size = 1175000L;    //dl 24Msa/54s
    long samples_to_dl = 24000000L;

    if (samples_to_dl > mdep) {
        samples_to_dl = mdep;
    }
    for(chunk_start = 1; chunk_start < samples_to_dl; chunk_start += chunk_size) {
        sprintf(c2, ":WAV:STAR %li;*OPC?\n", chunk_start);
        puts(c2);
        rx_len = lxi_send(device, c2, strlen(c2), timeout);
        printf("ack_len=%ld\n", rx_len);
        lxi_receive(device, response, sizeof(response), timeout);

        //at the last chunk, chunk_len might be shorter than chunk_size
        if (chunk_start + chunk_size > samples_to_dl) {
            chunk_len = samples_to_dl - chunk_start + 1;
            chunk_stop = samples_to_dl;
        } else {
            chunk_len = chunk_size;
            chunk_stop = chunk_start + chunk_size - 1;
        }

        sprintf(c2, ":WAV:STOP %li;*OPC?\n", chunk_stop);
        puts(c2);
        rx_len = lxi_send(device, c2, strlen(c2), timeout);
        printf("ack_len=%ld\n", rx_len);
        lxi_receive(device, response, sizeof(response), timeout);

        command = ":WAV:DATA?\n";
        puts(command);
        rx_len = lxi_send(device, command, strlen(command), timeout);
        printf("ack_len=%ld\n", rx_len);

        rx_len = lxi_receive(device, (char *)(&rx24MB[chunk_start-1]), chunk_len, timeout);
        printf("rx_data_bytes=%ld\n", rx_len);

        int err = 0;
        if(rx_len != chunk_len) {
            printf("ERROR - received too few bytes: ");
            err = 1;
        }
        printf("chunk_len=%li rx_len=%li\n", chunk_len, rx_len);
        printf("\n");
        if(err) exit(err);
    }

// Disconnect
lxi_disconnect(device);
}



will output is this:
Code: [Select]
liblxi w DS1054Z using TCP mode RAWRIGOL TECHNOLOGIES,DS1104Z,DS1ZA164658712,00.04.05.SP2

wait 15947
beep 45970
go 45970
:WAV:STAR 1;*OPC?

ack_len=18
:WAV:STOP 250000;*OPC?

ack_len=23
:WAV:DATA?

ack_len=11
rx_data_bytes=1460
ERROR - received too few bytes: chunk_len=250000 rx_len=1460
« Last Edit: November 19, 2022, 09:37:13 am by RoGeorge »
 

Online RoGeorgeTopic starter

  • Super Contributor
  • ***
  • Posts: 6628
  • Country: ro
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #32 on: November 20, 2022, 12:17:00 pm »
To learn what SOCKET parameters are optimal for a connection with the instrument, I had to read about TCP sockets, and realized that for LXI instruments it is trivial to send SCPI commands (and read the response) directly from Python, using its included 'socket' module.  ;D

Code: [Select]
import socket
socket.setdefaulttimeout(5000)

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.connect(('192.168.1.3', 5555))
    s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

    s.sendall(b'*IDN?\n')
    ans = s.recv(1024)

print(ans)
Will print
Quote
b'RIGOL TECHNOLOGIES,DS1104Z,DS1ZA123456789,00.04.05.SP2\n'

Where:
    #192.168.1.3    # is the fixed IP of the instrument (a Rigol DS1054Z oscilloscope)
    # 5555              # PORT number (fixed by manufacturer) 5555 for Rigol, 5025 for Agilent, etc.
    # 5000              # the timeout, in ms
    # 1024              # the size of the receive buffer

I only need to connect by LAN, using fixed IP, don't need any VXI11 or PyVISA or Telnet or any other dependency.  :-//
« Last Edit: November 20, 2022, 12:31:02 pm by RoGeorge »
 

Offline switchabl

  • Frequent Contributor
  • **
  • Posts: 445
  • Country: de
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #33 on: November 20, 2022, 01:37:12 pm »
One thing to be aware of when using plain TCP sockets is that TCP is a stream-oriented protocol. Oh, of course it operates on packets at the implementation level. But it tries to hide all of that away and provide you with a pipe where you can put in bytes at one end (send) and take some bytes out again at the other end (recv). There is no concept of a "message" and one send on the scope might not correspond to a single recv on your side.

In other words, your call to recv is allowed to return
Code: [Select]
b'RIGOL TECHNOLOGIES,DS1'or even just
Code: [Select]
b'R'The result will depend on the buffering strategies (and socket options) on either end as well as things like the underlying network packet size. So your example might not work on a different PC and it might not even work the next time you try it. You really need to keep calling recv until you have the expected number of bytes (or the termination character or hit a timeout).

In order to minimize latency, the other socket option you might want to try is TCP_QUICKACK (in addition to or instead of TCP_NODELAY).
 
The following users thanked this post: RoGeorge

Offline lundmar

  • Frequent Contributor
  • **
  • Posts: 441
  • Country: dk
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #34 on: November 20, 2022, 06:24:51 pm »
As it has been hinted, TCP is a stream oriented protocol so talking packets etc. does not make much sense.

When you use RAW mode with liblxi the interface behaves a bit more low level and you basically have to do two things when sending commands and potentially receiving large amounts of response data.

1. Send command: Append newline ('\n') to command string so that receiving instrument knows a command has been received and proceeds to process it
2. Receive response: Keep calling lxi_receive() until it returns error (timeout) which will be the only indication that the instrument has no more response data to send.

That is the downside of using TCP/RAW - one must rely on and wait for timeout because there is no protocol telling the client how much data to expect from the server.

This is exactly what protocols like VXI11 and HiSLIP takes care of. VXI11 does it in a bad way because of its heavy protocol overhead but HiSLIP solves that.

Now, I could move some of that complexity from the example to the liblxi implementation but I decided against it to give more control to the user so they can decide on buffering strategy etc.

I've added some example code here which demonstrates how to send a SCPI command to request screenshot image data from an instrument and receive all the image data:

https://github.com/lxi-tools/liblxi/blob/master/test/receive-image-data.c
https://lxi-tools.github.io - Open source LXI tools
https://tio.github.io - A simple serial device I/O tool
 
The following users thanked this post: RoGeorge

Online RoGeorgeTopic starter

  • Super Contributor
  • ***
  • Posts: 6628
  • Country: ro
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #35 on: November 21, 2022, 10:14:54 am »
In order to minimize latency, the other socket option you might want to try is TCP_QUICKACK (in addition to or instead of TCP_NODELAY).

Thanks for the hint.  Searched today about TCP_QUICKACK, and it seems to be Linux specific.  Other TCP stacks don't have the TCP_QUICKACK setting (e.g. the BSD one in FreeBSD or MacOS).
Quote
       TCP_QUICKACK (since Linux 2.4.4)
              Enable quickack mode if set or disable quickack mode if
              cleared.  In quickack mode, acks are sent immediately,
              rather than delayed if needed in accordance to normal TCP
              operation.  This flag is not permanent, it only enables a
              switch to or from quickack mode.  Subsequent operation of
              the TCP protocol will once again enter/leave quickack mode
              depending on internal protocol processing and factors such
              as delayed ack timeouts occurring and data transfer.  This
              option should not be used in code intended to be portable.
Source:  https://man7.org/linux/man-pages/man7/tcp.7.html



I've added some example code here which demonstrates how to send a SCPI command to request screenshot image data from an instrument and receive all the image data:

https://github.com/lxi-tools/liblxi/blob/master/test/receive-image-data.c

You are too kind, thank you.  I understand now the need for '\n' with lxilib in RAW mode, makes sense.  Also it clarifies my question about consolidating consecutive SCPI commands before sending them:
Tried liblxi with TCP RAW, and I don't know how to handle the transfers.

- it puts consecutive lxi_send() strings in the same data packet, which the oscilloscope won't understand.  I don't know how to flush the transmit buffer after each lxi_send(), so as a workaround I'm adding an *OPC? at the end of each command.
'liblxi' works just fine, no need to flush.  The instrument will understand multiple commands from a single TCP packet, just that I was sending a wrong SCPI command, a bug in my code, sorry.
« Last Edit: November 21, 2022, 10:20:26 am by RoGeorge »
 
The following users thanked this post: lundmar

Offline alm

  • Super Contributor
  • ***
  • Posts: 2903
  • Country: 00
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #36 on: November 21, 2022, 05:57:25 pm »
- A (telnet) data transfer will drop if inside any data packets it receives a 0x00.  This is some RFC spec (for text Telnet IIRC).
- The ADC samples are coming as bytes, which means the incoming data packets will be truncated if one of the bytes is zero, because in the Telnet RFC 0x00 is considered a terminator, and will drop any bytes coming after a 0x00.
- This issue is only observed when the input signal in the ADC is less than 5 divisions on the screen (ADC outputs 0x7f for zero volts, and 0x00 for any input voltage that is -5*volts/div or under)
- Both the pyvisa->pyvisa-py->socket and the socketscpi are affected
I'm not sure why this is happening. As far as I know these libraries should use plain TCP sockets, and not implement the Telnet protocol. I would be careful with the word "raw TCP sockets", because raw sockets is generally understood to mean something else in network programming.

I just tested it with PyVISA, and it works as expected for me:
I used this as a mock server returning data containing null characters: echo '\0ff\0' | nc -vl -p 1052

And then succesfully retrieved this data:
Code: [Select]
>>> import pyvisa
>>> rm = pyvisa.ResourceManager()
>>> res = rm.open_resource('TCPIP0::127.0.0.1::1052::SOCKET')
>>> res.write_termination = '\n'
>>> res.read_termination = '\n'
>>> data = res.read()
>>> data
'\x00ff\x00'

You don't happen to have the read termination set to \00, have you? Because then I could understand this happening.
 
The following users thanked this post: RoGeorge

Online RoGeorgeTopic starter

  • Super Contributor
  • ***
  • Posts: 6628
  • Country: ro
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #37 on: November 22, 2022, 06:00:29 pm »
Might have been something I was doing wrong.  Can not reproduce that error any longer.  :-//

Offline MiDi

  • Frequent Contributor
  • **
  • Posts: 611
  • Country: ua
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #38 on: March 11, 2023, 04:29:14 pm »
Had similar problems:
On one PC the readout of 24MSamples took more than 40s and on another ~22s (Both Win10 latest updates, no TCP mods, but different python & module versions).
Turning Nagle off in Win did not help - as was expected as PyVisa turns it off by default for the connection itself.
Turning delayed ACK off in Win did improve the readout times a bit (~36s), but still slow.

As it turned out changing the code did the trick to get ~22s on both PCs:

instead single commands:
Code: [Select]
instrument.write(f":WAV:STAR {interval[0]}") # start index of memory
instrument.write(f":WAV:STOP {interval[1]}") # stop index of memory
data += (instrument.query_binary_values(f":WAV:DATA?", datatype='B')) # get the data, B = unsigned char


all commands at once did it:
Code: [Select]
data += (instrument.query_binary_values(f":WAV:STAR {interval[0]}\n:WAV:STOP {interval[1]}\n:WAV:DATA?", datatype='B')) # get the data, B = unsigned char


For reference the full source code used:
Code: [Select]
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# developed & tested with Python 3.6, pyvisa 1.11.3

# Rigol DS1000Z data acquisition

#from ctypes.wintypes import FLOAT
#from unicodedata import decimal
import pyvisa as visa # needs a backend e.g. NI-VISA or PyVISA-py
import sys
from time import time, sleep
from matplotlib.ticker import EngFormatter
from decimal import *

def ef(value, places=None):
    return EngFormatter(sep="", places=places).format_eng(float(value)) #round(value, 4) - need to round in exp form

def crange(start,end,step):
    i = start
    while i < end-step+1:
        yield i, i+step-1
        i += step
    yield i, end

def wait_ready(instrument):
#instrument.write("*OPC")
instrument.write("*WAI")
ready = instrument.query("*OPC?").strip()
#print(ready)
while ready != "1": # never occured, needed?
ready = instrument.query("*OPC?").strip()
print(f"\n-------------------not ready: {ready}-----------------------")
#pass

def is_error(instrument):
wait_ready(instrument)
status = instrument.query("*ESR?").strip()
if status not in ('0', '1'):
wait_ready(instrument)
return instrument.query(":SYST:ERR?").strip() #:SYSTem:ERRor[:NEXT]?
else:
return False

# connect to scope
# initialize scope
##*RST                     p. 96 - Restore the instrument to the default state
##*WAI                     p. 97 - Wait for the operation to finish (maybe not helpful as it is only for parallel execution of commands)
##:ACQ:MDEP?               p. 21 - Set or query the memory depth of the oscilloscope (namely the number of waveform points that can be stored in a single trigger sample)
##:ACQ:TYPE?               p. 22 - Set or query the acquisition mode of the oscilloscope
##:ACQ:SRAT?               p. 23 - Query the current sample rate. The default unit is Sa/s.

#def connect(timeout = 10000, chunk_size = 20*1024):
def connect():
try:
# Get the USB device, e.g. 'USB0::0x1AB1::0x0588::DS1ED141904883'
rm = visa.ResourceManager() # "@py" for PyVISA-py
instruments = rm.list_resources() # rm.list_resources('USB?*')
print(instruments)
usb = list(filter(lambda x: 'USB' in x, instruments)) # TODO filter for ::DS1Z
print(usb)
#if len(usb) != 1:
# print(f"No or multiple instruments found: {instruments}")
# sys.exit(-1)
#from pyvisa.highlevel import ascii, single, double
#instrument = rm.open_resource(usb[0])

# instrument = rm.open_resource(f"TCPIP0::192.168.1.26::5555::SOCKET", write_termination = '\n', read_termination = '\n', timeout = timeout, chunk_size = chunk_size)
instrument = rm.open_resource(f"TCPIP0::192.168.1.26::5555::SOCKET", write_termination = '\n', read_termination = '\n')

#instrument = rm.open_resource("TCPIP::192.168.1.26::INSTR", write_termination = '\n', read_termination = '\n')
#instrument.write_termination = '\n'
#instrument.read_termination = '\n'
#instrument = rm.open_resource("USB0::0x1AB1::0x04CE::DS1ZA172316530::INSTR")

#instrument.encoding='latin1' # hack - better do ask raw
#rm.timeout = timeout
#rm.chunk_size = chunk_size
return instrument

except Exception as e:
print(f"Cannot open instrument: {str(e)}")
sys.exit(-1)

def setup(instrument, timebase_scale, ch1_scale, mem_depth):
#timebase_scale = Decimal("50") # timebase scale in s
#ch1_scale = Decimal("1e-3") # set channel scale in units (V, A, W)
#mem_depth = "24000000" # set memory depth in points: 12000, 120000, 1200000, 12000000, 24000000
timebase_scale = Decimal(timebase_scale) # timebase scale in s
ch1_scale = Decimal(ch1_scale) # set channel scale in units (V, A, W)
mem_depth = mem_depth # set memory depth in points: 12000, 120000, 1200000, 12000000, 24000000

instrument.write(f"*CLS") # Clear event registers and error queue
wait_ready(instrument)

status = instrument.query(":TRIG:STAT?").strip() # Stop instrument: defined state
while status != "STOP":
instrument.write(":STOP")
wait_ready(instrument)
status = instrument.query(":TRIG:STAT?").strip()

set_timebase_scale = Decimal(instrument.query(":TIM:SCAL?").strip()) # set timebase scale
while set_timebase_scale != timebase_scale:
print(f"set_timebase_scale: {set_timebase_scale}, timebase_scale: {timebase_scale}")
instrument.write(f":TIM:SCAL {timebase_scale}")
wait_ready(instrument)
set_timebase_scale = Decimal(instrument.query(":TIM:SCAL?").strip())
print(f"Timebase scale: {ef(float(set_timebase_scale))}s/div")

set_ch1_scale = Decimal(instrument.query(":CHAN1:SCAL?").strip()) # query channel scale
while set_ch1_scale != ch1_scale:
print(f"set_ch1_scale: {set_ch1_scale}, ch1_scale: {ch1_scale}")
instrument.write(f":CHAN1:SCAL {ch1_scale}")
wait_ready(instrument)
set_ch1_scale = Decimal(instrument.query(":CHAN1:SCAL?").strip())
print(f"Ch1 scale: {ef(float(set_ch1_scale))}V/div")

set_mem_depth = instrument.query(":ACQ:MDEP?") # memory depth = bytes to read
if set_mem_depth != mem_depth:
status = instrument.query(":TRIG:STAT?").strip() # Start instrument: needed for setting MDEP
while status == ("STOP"):
print(f"status: {status}")
instrument.write(":RUN")
wait_ready(instrument)
status = instrument.query(":TRIG:STAT?").strip()

while set_mem_depth != mem_depth:
print(f"set_mem_depth: {set_mem_depth}, mem_depth: {mem_depth}")
instrument.write(f":ACQ:MDEP {mem_depth}")
wait_ready(instrument)
set_mem_depth = instrument.query(":ACQ:MDEP?")
print(f"Memory depth: {ef(float(set_mem_depth))}pts")

trig_pos = set_timebase_scale * 6 # horizontal position of trigger on left edge (no pre-trigger)
set_trig_pos = Decimal(instrument.query(":TIM:OFFS?").strip())
while set_trig_pos != trig_pos:
print(f"set_trig_pos: {set_trig_pos}, trig_pos: {trig_pos}")
instrument.write(f":TIM:OFFS {trig_pos}")
wait_ready(instrument)
set_trig_pos = Decimal(instrument.query(":TIM:OFFS?").strip())
print(f"Trigger position: {ef(float(set_trig_pos))}s")

# read data (p. 241)
#   :STOP - better use single and wait until finished (if that is possible)
#   :WAV:FORM WORD - are there values beyond 255? Else BYTE would be sufficient
##S1.  :STOP               Set the instrument to STOP state (you can only read the waveform data in the internal memory when the oscilloscope is in STOP state)
##S2.  :WAV:SOUR CHAN1     Set the channel source to CH1
##S3.  :WAV:MODE RAW       Set the waveform reading mode to RAW
##S4.  :WAV:FORM WORD      Set the return format of the waveform data to WORD
##Perform the first reading operation
##S5.  :WAV:STAR 1         Set the start point of the first reading operation to the first waveform point
##S6.  :WAV:STOP 125000    Set the stop point of the first reading operation to the 125000th waveform point
##S7.  :WAV:DATA?          Read the data from the first waveform point to the 125000th waveform point
##Perform the second reading operation
##S8.  :WAV:STAR 125001    Set the start point of the second reading operation to the 125001th waveform point
##S9.  :WAV:STOP 250000    Set the stop point of the second reading operation to the 250000th waveform point
##S10. :WAV:DATA?          Read the data from the 125001th waveform point to the 250000th waveform point
##Perform the third reading operation
##S11. :WAV:STAR 250001    Set the start point of the third reading operation to the 250001th waveform point
##S12. :WAV:STOP 300000    Set the stop point of the third reading operation to the 300000th waveform point (the last point)
##S13. :WAV:DATA?          Read the data from the 250001th waveform point to the 300000th waveform point (the last point)

#   improved read data
##S1.  :SING               p. 19 - Set the oscilloscope to the single trigger mode
##S2.  :TFOR               p. 19 - Generate a trigger signal forcefully. This command is only applicable to the normal and single trigger modes (see the :TRIGger:SWEep command) and is equivalent to pressing the FORCE key in the trigger control area on the front panel.
##S3.  wait for scope to finish
##S4.  :WAV:SOUR CHAN1     Set the channel source to CH1
##S5.  :WAV:MODE RAW       Set the waveform reading mode to RAW
##S6.  :WAV:FORM BYTE      p. 239 - Set the return format of the waveform data to WORD|BYTE|ASCii (WORD: higher byte is always 0) - default: BYTE

def capture_data(instrument):
force_trigger = True

set_trig_swe = instrument.query(":TRIG:SWE?").strip() # query trigger sweep
print(f"set_trig_swe: {set_trig_swe}")
while set_trig_swe == "SING": # reset single trigger (defined state)
instrument.write(":TRIG:SWE AUTO")
wait_ready(instrument)
set_trig_swe = instrument.query(":TRIG:SWE?").strip()

while set_trig_swe != "SING": # single trigger
instrument.write(":SING")
wait_ready(instrument)
set_trig_swe = instrument.query(":TRIG:SWE?").strip()
print("measurement started")

# TODO refactor
# :TRIG:STAT? returns TD, WAIT, RUN, AUTO, or STOP
# RUN (Pre-Trigger measurement) - WAIT (for trigger, may not be read out if triggered by signal) - TD (Post-Trigger) - STOP (measurement finished)

#wait until measurement finished
start_time = time()
run_time = start_time
current_time = start_time
wait_ready(instrument) # even with wait_ready sometimes
status = instrument.query(":TRIG:STAT?").strip() # get trigger status: TD, WAIT, RUN, AUTO or STOP
prev_status = status
wait_count = 0
finished = False
while not finished:
if status != prev_status:
print()
run_time = time()
current_time = time()
if status in ("RUN", "TD"): # RUN/TD - measurement is runnning
print(f"{status} {current_time - run_time:7.3f}s          ", end="\r")
elif status == "WAIT": # WAIT - waiting for trigger
print(f"{status} {current_time - run_time:.3f}s")
if force_trigger:
wait_ready(instrument)
instrument.write(":TFOR") # force trigger (may need multiple tries to start)
wait_count += 1
print(f'trigger forced {wait_count}x')
elif status == "STOP": # end on STOP (measurement finished)
finished = True
print(f"{status} {current_time - run_time:.3f}s")
else: # AUTO - should not happen
print(f"{status} {current_time - run_time:7.3f}s          ", end="\r")
prev_status = status
status = instrument.query(":TRIG:STAT?").strip() # get trigger status: TD, WAIT, RUN, AUTO or STOP
#sleep(0.01)
current_time = time()
print(f"measurement finished in {current_time - start_time:.3f}s")

def read_data(instrument, ch: int = 1):
mem_depth = int(instrument.query(":ACQ:MDEP?")) # memory depth

# :WAV needs to be en bloc, at least between STAR & STOP
instrument.write(f":WAV:SOUR CHAN{int(ch)}") # data from channel 1-4
wait_ready(instrument)
instrument.write(":WAV:MODE RAW") # data from internal memory
wait_ready(instrument)
instrument.write(":WAV:FORM BYTE") # data as byte
data = []
start_time = time()
for interval in crange(1, mem_depth, 250_000): # 250_000 is max chunk size for bytes
#instrument.write(f":WAV:STAR {interval[0]}") # start index of memory
#instrument.write(f":WAV:STOP {interval[1]}") # stop index of memory
#data += (instrument.query_binary_values(f":WAV:DATA?", datatype='B')) # get the data, B = unsigned char
data += (instrument.query_binary_values(f":WAV:STAR {interval[0]}\n:WAV:STOP {interval[1]}\n:WAV:DATA?", datatype='B')) # get the data, B = unsigned char
print(f"{len(data)/mem_depth:3.0%} memory read in {time() - start_time:.3f}s ({len(data)}/{mem_depth}) reading: {interval[0]}:{interval[1]}", end="\r")
print()
# debug
print(mem_depth)
sps = float(instrument.query(":ACQ:SRAT?").strip()) # query the current sample rate in Sa/s
print(f"SPS: {ef(sps)}")
timebase_scale = float(instrument.query(":TIM:SCAL?").strip()) # query timebase scale
print(f"Timebase scale: {ef(timebase_scale)}s/div")
# debug end
# debug
#instrument.write(":WAV:FORM ASCii") # data as ASCII
#ascii_data = instrument.query(":WAV:DATA?") # get the data
#print("ASCII:")
#print(ascii_data[11:1000])
#print("ASCII END")
# debug end

return data

# scale data
# formula for converting in V (p. 242): scaled value = (value - YORigin - YREFerence) x YINCrement
##:WAV:YINC?  # p. 244, return in scientific notation, see details
##:WAV:YOR?   # p. 244, return as integer, see details
##:WAV:YREF?  # p. 245, returns 127 always?, see details
##:WAV:PRE?   # p. 246, returns 10 waveform parameters separated by ",", see details

def scale_data(instrument, data):
wait_ready(instrument)
mem_depth = Decimal(instrument.query(":ACQ:MDEP?").strip()).normalize() # memory depth
wait_ready(instrument)
wf_parameters = instrument.query(":WAV:PRE?").strip().split(",") # get waveform parameters: <format>,<type>,<points>,<count>,<xincrement>,<xorigin>,<xreference>,<yincrement>,<yorigin>,<yreference>
print(f"Waveform parameters: {wf_parameters}")
while Decimal(wf_parameters[2]).normalize() not in (mem_depth, mem_depth/2, mem_depth/3, mem_depth/4):
wait_ready(instrument)
wf_parameters = instrument.query(":WAV:PRE?").strip().split(",")
print(f"Waveform parameters: {wf_parameters}")
yincrement = float(wf_parameters[7])
yorigin = float(wf_parameters[8])
yreference = float(wf_parameters[9])
scaled_data = []
for byte in data:
scaled_data.append((float(byte) - yorigin - yreference) * yincrement)
return scaled_data

def save_data():
pass

def screenshot(instrument):
sleep(2) # wait until all messages are gone (e.g. can operate now)
start_time = time()
screenshot = instrument.query_binary_values(":DISP:DATA? ON,OFF,PNG", datatype='B') #  ON,OFF,PNG - BMP is a bit faster
stop_time = time()
print(f"Screenshot took {stop_time - start_time:.3f}s")
return screenshot


def view_image(image):
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
img = mpimg.imread(image)
imgplot = plt.imshow(img)
plt.show()

def main():
#global measurement_filename
timebase_scale = "1" # timebase scale in s
ch1_scale = "1e-3" # set channel scale in units (V, A, W)
mem_depth= "24000000" # set memory depth in points: 12000, 120000, 1200000, 12000000, 24000000

data_path = r"D:\Dokumente\Software-Projekte\Rigol DS1000Z data acquisition"
data_dir = ""
data_ext = "csv"
data_file = "test"

scope = connect()
setup(scope, timebase_scale, ch1_scale, mem_depth)

# TODO get from capture_data and read_data as return
probe_ratio = Decimal(scope.query(":CHAN1:PROB?").strip()).normalize()
ch1_scale_ef = ef(Decimal(ch1_scale), places=0)
sps_ef = ef(Decimal(scope.query(":ACQ:SRAT?").strip()).normalize(), places=0)
acq_mode = scope.query(":ACQ:TYPE?").strip()
mem_depth_ef = ef(scope.query(":ACQ:MDEP?").strip(), places=0)
acq_time_ef = ef(Decimal(timebase_scale)*12, places=0)

for n in range(1, 2):
capture_data(scope)
data = read_data(scope)

print()
print(f"data length: {len(data)}")
#print(data[:100])

scaled_data = scale_data(scope, data)
print(f"scaled data length: {len(scaled_data)}")
#print(scaled_data[:100])

data_file_tail = f"x{probe_ratio} {ch1_scale_ef}V {sps_ef}SPS {acq_mode} {mem_depth_ef}pts {acq_time_ef}s"
data_file_full = f"{data_path}\{data_dir}\{data_file} {n:02} - {data_file_tail}"
import csv
with open(f"{data_file_full}.{data_ext}", "w", newline="") as f:
writer = csv.writer(f)
for item in scaled_data:
writer.writerow([format(item, ".4g")]) #to reduce the file size further, limit number to significant decimal digits / resolution (8bit/10div) depending on channel range

#import numpy as np
##np_sd = np.asarray(scaled_data)
#np.savetxt("D:\Dokumente\Software-Projekte\Rigol DS1000Z data acquisition\DS1000Z_DAQ.csv", scaled_data, fmt = "%.4g") #to reduce the file size further, limit number to significant decimal digits / resolution (8bit/10div) depending on channel range

print (f"{data_file_full}.{data_ext} written")

image = screenshot(scope)
#view_image(image)

with open(f"{data_file_full}.png", "wb") as f: # write screenshot to file
f.write(bytearray(image))
print (f"{data_file_full}.png written")

if __name__ == "__main__":
main()


For the curious, the relevant parts of the wireshark logs are attached.
« Last Edit: March 11, 2023, 04:32:15 pm by MiDi »
 
The following users thanked this post: alm


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf