Download speed from Rigol DS1054Z or similar oscilloscope to a PC

[RoGeorge]

Asking because I've just tried, and it took more than 21 minutes to download all the 24 million data points. 

The PC can easily download from Internet at 700-800 Mbps, so the bottleneck is not the computer's hardware.
    - Software:  Kubuntu -> VScode (Codium) -> Jupyter Notebooks -> my Python scripts -> python-ivi -> PyVISA -> PyVISA-py -> python-vxi11
    - Hardware:  i7 Desktop -> LAN cable -> 100Mbps router (switch) -> LAN cable -> Rigol DS1104Z oscilloscope

Anybody else tried to download all the data sampled by a Rigol oscilloscope?
How fast does that worked for you, and with what software/hardware, please?

[Fungus]

What format did you download it in?  CSV?

Bottom line: The rigol is an oscilloscope, not a high speed internet device.

[RoGeorge]

With the DS1000Z series it takes 55 seconds to download 24Mpts via usbtmc. Via ethernet it is more or less the same.
The bottleneck is the scope, not the connection.
Data needs to be downloaded in chunks of max 250,000 bytes.
Between those chunks there's a long delay in the response of the scope's firmware.
Why? Only Rigol knows...

Edit: I did a new test with the latest firmware (the one Rigol removed) and now the download time is 33 seconds for 24Mpts.

That was 8+ years ago.  Back then I was experiencing very slow downloads, but only when using Windows.  From Linux it was OK.  I thought that issue was sorted out.  Now, many firmware updates and many OS versions later, I'm seeing about the same problem, but with Linux. 

Anybody else tried downloading the data samples from a Rigol recently, please?

[Fungus]

Anybody else tried downloading the data samples from a Rigol recently, please?

yes 30-60 seconds is the right figure to download raw 24Mpts data using visa driver from VisaDSO in Windows here.. if it takes 21 minutes for you, something is wrong.

We know nothing at all about how OP is "downloading"the data...

[Karel]

Using DSRemote it takes exactly 30 seconds (per channel) to download the whole 24Mpts.

[RoGeorge]

Thanks for mentioning DSremote, forgot about that one.  Installed DSremote in Kubuntu 22.04 LTS, and it takes only 30 seconds to download data in its Wave Inspector. 

At first I was trying from Python-IVI, which is a layer on top on PyVISA, as described in the OP.  It was way too slow, so this weekend tried sending SCPI commands from C, by calling into the 'liblxi.so' library (liblxi is available in most of the official repos, "sudo apt install liblxi-dev" or from sources at https://github.com/lxi-tools/liblxi ).  Then tried with DSremote, and DSremote is still 3 times faster than anything else I've tried.

Always used RAW mode and BYTES, so 24MB of data to be downloaded for all the 24MSa of ADC data.  No matter the size of each consecutive download batches, the duration benchmark is for a complete 24Msa download.

• 23 minutes - Python scripts, IVI calls -> python-ivi -> PyVISA -> PyVISA-py -> python-vxi11 -> LAN -> DS1104Z
• 134 seconds - C program, SCPI requesting batches of 125'000 bytes at once -> lib lxi -> lib tirpc  -> LAN -> DS1104Z
•   86 seconds - C program, SCPI requesting batches of 250'000 bytes at once (maximum size according to the DS1000Z programming guide) -> lib lxi -> lib tirpc  -> LAN -> DS1104Z
•   65 seconds - C program, SCPI requesting batches of 750'000 bytes at once -> lib lxi -> lib tirpc  -> LAN -> DS1104Z
•   55 seconds - C program, SCPI requesting batches of 1'000'000 bytes at once -> lib lxi -> lib tirpc  -> LAN -> DS1104Z
•   30 seconds  :o - DSremote downloading all the samples to be saved or viewed in its waveform inspector

It's interesting how DSremote is at least twice than any other setup.  I've looked at the LAN packets with Wireshark, and it seems DSremote is not using VXI11 protocol for download, it is using plain TCP on the port 5555.  The SCPI commands from DSremote are as expected:

Code: [Select]

    STOP:
    *OPC?
    WAV:SOUR CHAN1
    *OPC?
    WAV:FORM BYTE
    *OPC?
    WAV:MODE:RAW
    *OPC?
    WAV:YINC?
    WAV:YREF?
    WAV>YOR?

    WAV:STAR 1
    WAV:OPC?
    WAV:STOP 250000
    WAV:OPC?
    WAV:DATA?
        // incoming first 250'000 ADC samples

    WAV:STAR 250001\0x0a
    *OPC?\0x0a
        // (expected response from *OPC is 1\0x0a)
    WAV:STOP 500000\0x0a
    *OPC?\0x0a
    WAV:DATA?\0x0a
        // (data response is #9000250000<250kB_of_data_over_many_TCP_packets_of_mostly_65535_TCP_bytes_each>\0x0a

    // and so on until all the wanted 24 million samples are fetched
    // ...

 - SCPI terminator is always the 0x0a char <LF> (beware printf("\n") in Windows is two cars, <CR><LF>, printf("%c", 10) might be safer)
 - OPC? (operation completed) is probed by DSremote after each command (which I didn't probe, maybe I should)
 - timeout in the DSremote settings was left as found, 50ms, but if I put the same timeout in my C code, the value is too small and leads to timeout errors sooner or later.  I had to use 2000ms instead of 50ms. 


This is my fastest so far, with C and liblxi, which is still about 3 times slower than DSremote:

Code: [Select]

// liblxi git at https://github.com/lxi-tools/liblxi
// to install it from the ubuntu repository (lxilib-dev is in most of the Linux distros repos and in FreeBSD):
//      sudo apt install liblxi-dev

// headers in   /usr/include/lxi.h
// .so lib in   /usr/lib/x86_64-linux-gnu/liblxi.so
// display all installed files with 'dpkg -L liblxi-dev' or
//      sudo apt install apt-file
//      sudo apt-file update
//      (sudo???) apt-file list liblxi-dev
// no need to append LIBRRY_PATH or LD_LIBRARY_PATH

// man lxi_<TAB><TAB>
//      lxi_connect  lxi_discover  lxi_init  lxi_send  lxi_disconnect  lxi_discover_if  lxi_receive

// compile and run this 'demo.c' from a Linux terminal (!!! -l options _must_ be positioned after the .c sources, not before)
//      gcc -Werror -Wfatal-errors ./demo.c -llxi -ltirpc && time ./a.out

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <lxi.h>

int main()
{
    // 12 bytes preamble + max of 24 mil bytes of ADC samples + 1 byte for the end of string terminator
    #define CHARS_BUFFER_MAX    24000000L // in specs max Rx in a single chunk is 250k chars for Rigol DS1054Z

    char *rx24MB = NULL;                                    // declare a pointer to char, and initialize it with NULL
    rx24MB = malloc(CHARS_BUFFER_MAX * sizeof *rx24MB);     // allocated memory on the heap for rx24MB[CHARS_BUFFER_MAX]
    if (!rx24MB) {                                          // !!! always check the memory was allocated successfully !!!
        fputs ("ERROR: allocation memory failed for the 'rx24MB' buffer, exiting.", stderr);
        exit (EXIT_FAILURE);
    }

    int device, length, timeout = 2000;     // timeout is in ms
    char response[1000];    // allocated on the stack, segmentation fault if too large, Linux max stack is 8MiB

    char *command = "*IDN?";
   
    lxi_init(); // Initialize LXI library
    device = lxi_connect("192.168.1.3", 0, "inst0", timeout, VXI11);    // Connect to LXI device
   
    lxi_send(device, command, strlen(command), timeout);                // Send SCPI command ("*IDN?")
    lxi_receive(device, response, sizeof(response), timeout);           // Rx until "\n", or sizeoff(response) chars, or timeout

    printf("%s\n", response);
    // exit if no DS1054Z
// exit(0);


    // enable only ch1
    command = ":CHAN1:DISP 1";
    lxi_send(device, command, strlen(command), timeout);
    command = ":CHAN2:DISP 0";
    lxi_send(device, command, strlen(command), timeout);
    command = ":CHAN3:DISP 0";
    lxi_send(device, command, strlen(command), timeout);
    command = ":CHAN4:DISP 0";
    lxi_send(device, command, strlen(command), timeout);

// ??? here to insert ":RUN""
    // set acquisition mode to max 24_000_000 points memory depth
    // "auto", 12K, 120K, 1.2M, 12M, 24M"
    long mdep = 24000000L;

    command = ":RUN";
    lxi_send(device, command, strlen(command), timeout);
   
    // sprintf(command, ":ACQ:MDEP %li", mdep);
    // sprintf(command, ":ACQ:MDEP %li%s", mdep, "\0");
    command = ":ACQ:MDEP 24000000";
    lxi_send(device, command, strlen(command), timeout);

    // wait for triggered
    do {
        command = ":TRIGger:STATus?";
        lxi_send(device, command, strlen(command), timeout);
        lxi_receive(device, response, sizeof(response), timeout);
    } while (response[0] != 'T');

    // then wait for ADC to aquire enough data
    clock_t stop_time;
    printf("wait %ld\n", clock());
    stop_time = clock() + 30000;             //30 seconds
    printf("beep %ld\n", stop_time);
    while (clock() < stop_time) ;
    printf("go %ld\n", clock());

    // go to stop mode and prepare to dl 24MSa
    command = ":STOP";
    lxi_send(device, command, strlen(command), timeout);
    command = ":WAV:SOUR CHAN1";
    lxi_send(device, command, strlen(command), timeout);
    command = ":WAV:MODE RAW";
    lxi_send(device, command, strlen(command), timeout);
    command = ":WAV:FORM BYTE";
    lxi_send(device, command, strlen(command), timeout);

// char c2[80] = {0};
char c2[80];
// loop to dl all samples in chunks of (max) 250_000 samples
    // set start-stop indexes for next chunk
    long rx_len, chunk_len;
    // long chunk_start, chunk_stop, chunk_size =  125000L;    //dl 24Msa/134s
     long chunk_start, chunk_stop, chunk_size =  250000L;    //dl 24Msa/86s  max from specs
    // long chunk_start, chunk_stop, chunk_size =  750000L;    //dl 24Msa/65s
    // long chunk_start, chunk_stop, chunk_size = 1000000L;    //dl 24Msa/56s  preferred ???
    // long chunk_start, chunk_stop, chunk_size = 1175000L;    //dl 24Msa/54s
    long samples_to_dl = 24000000L;

    if (samples_to_dl > mdep) {
        samples_to_dl = mdep;
    }
    for(chunk_start = 1; chunk_start < samples_to_dl; chunk_start += chunk_size) {
        sprintf(c2, ":WAV:STAR %li", chunk_start);
        puts(c2);
        rx_len = lxi_send(device, c2, strlen(c2), timeout);
        printf("ack_len=%ld\n", rx_len);

        //at the last chunk, chunk_len might be shorter than chunk_size
        if (chunk_start + chunk_size > samples_to_dl) {
            chunk_len = samples_to_dl - chunk_start + 1;
            chunk_stop = samples_to_dl;
        } else {
            chunk_len = chunk_size;
            chunk_stop = chunk_start + chunk_size - 1;
        }

        sprintf(c2, ":WAV:STOP %li", chunk_stop);
        puts(c2);
        rx_len = lxi_send(device, c2, strlen(c2), timeout);
        printf("ack_len=%ld\n", rx_len);

        command = ":WAV:DATA?";
        puts(command);
        rx_len = lxi_send(device, command, strlen(command), timeout);
        printf("ack_len=%ld\n", rx_len);

        rx_len = lxi_receive(device, (char *)(&rx24MB[chunk_start-1]), chunk_len, timeout);
        printf("rx_data_bytes=%ld\n", rx_len);

        int err = 0;
        if(rx_len != chunk_len) {
            printf("ERROR - received too few bytes: ");
            err = 1;
        }
        printf("chunk_len=%li rx_len=%li\n", chunk_len, rx_len);
        printf("\n");
        if(err) exit(err);
    }

// Disconnect
lxi_disconnect(device);
}


Not sure if I am doing something wrong, or if I should just use TCP instead of VXI11? 

[Fungus]

if I should just use TCP instead of VXI11? 

Yes.

[alm]

At first I was trying from Python-IVI, which is a layer on top on PyVISA, as described in the OP.  It was way too slow

Python-IVI is transport-agnostic, so I'd think that you can tell the IVI driver to connect using TCP instead of VXI-11 using a different VISA resource name. I don't see why you'd need to use C for a throughput of maybe 2 MB/s.

[RoGeorge]

I used C because the SW stack from python was heaving too many layers and I didn't know which one was slowing down the transfers.  The simplest way I was aware of (other than making direct socket calls) was to use liblxi from C.

I did met some similar issues (10-100 slower transfers than it should be) about 7-8 years ago, when I was using SCPI sent to a TCP socket, directly from Python with no drivers, or IVI/VISA/VXI or alike.  Back then it was happening in Windows only, because of some specific algorithm Windows was using when it was increasing the TCP packets size.  That was broken by design for Windows, but it was working fine from Linux.

Now I am seeing those 20+ minutes long transfers in Linux.  Don't know yet if it's caused by the same issue from 8 years ago, or if it's something different.  Maybe the wrong by designed RWIN (TCP Receive WINdow) algorithm used by Windows back then was meanwhile added as a new Linux "feature", IDK. 
https://hackaday.io/project/5807-driverless-rigol-ds1054z-screen-capture-over-lan/log/18807-solved-a-painfully-slow-lan-on-windows-and-a-wasted-weekend

[alm]

Now I am seeing those 20+ minutes long transfers in Linux.

Is that over VXI-11 or over TCP sockets (TCPIP0::host address::port::SOCKET)? VXI-11 is probably not optimized for high throughput, but plain TCP sockets should be pretty straight-forward.

[DiTBho]

too confusing 

[DiTBho]

I don't have a Rigol DS1054Z, but sigrok seems to have support, see here

[RoGeorge]

Now I am seeing those 20+ minutes long transfers in Linux.

Is that over VXI-11 or over TCP sockets (TCPIP0::host address::port::SOCKET)? VXI-11 is probably not optimized for high throughput, but plain TCP sockets should be pretty straight-forward.

Wireshark was telling the LAN packets were of type VXI-11 protocol, most probably the slow transfer was for VXI.  In fact, it might be the only one supported, I don't know how to configure IVI to use SOCKET instead of VXI-11.

I've just tried "TCPIP0::host address::port::SOCKET" and it breaks with the err message "unrecognized resource string format".  By looking inside the code, the regular expressions used to validate the resource string doesn't know about the word "SOCKET", can only end in "INSTR".  Same in the man page, all examples are of type "TCPIP................INSTR"
https://www.systutorials.com/docs/linux/man/1-pythonivi


It is still possible to use python-ivi for all the other commands, and write a C function to fetch the ADC data points only using SOCKET and forcing the TCP packets size, the oscilloscope allows multiple connections.  Or I can dig in my own python2 repos from 8 years ago and check if the WAVE:DATA? scripts from back then are still fast (in Linux).

IVI is very appealing, but for some reason it didn't get enough traction, and now  it feels like abandonware, the last accepted pull in the main project was in 2016 or so.  For example, two years ago I had to merge from many forks and found no python-ivi driver for my signal generator.  Now, I had to search again through 108 forks and merge the most active (active in the sense that they appear to have extra python-ivi drivers), thought there is still no driver for my generator, but there is a generator driver from another instrument, and it happens that the main controls are the same, so at least I can change the frequency from python-ivi and do the rest from buttons.

I really like python-ivi, but it sunk me way too much time.  After all, I don't think any body else will reuse my automated test software, not even myself, so I can just use SCPI.  It's been 10 days since I've started to fool around with python-ivi and still tinkering around it instead of working at the data processing, which was the actual goal of the project.

[alm]

I've just tried "TCPIP0::host address::port::SOCKET" and it breaks with the err message "unrecognized resource string format".  By looking inside the code, the regular expressions used to validate the resource string doesn't know about the word "SOCKET", can only end in "INSTR".  Same in the man page, all examples are of type "TCPIP................INSTR"
https://www.systutorials.com/docs/linux/man/1-pythonivi

Reading the source code, this regular expression is for the resources python-ivi can handle directly without PyVisa, like VXI-11. Anything else should be passed to PyVisa, unless PyVisa is not imported, which sounds like is the case for you (the exception on line 1837).

IVI is very appealing, but for some reason it didn't get enough traction, and now  it feels like abandonware, the last accepted pull in the main project was in 2016 or so.  For example, two years ago I had to merge from many forks and found no python-ivi driver for my signal generator.  Now, I had to search again through 108 forks and merge the most active (active in the sense that they appear to have extra python-ivi drivers), thought there is still no driver for my generator, but there is a generator driver from another instrument, and it happens that the main controls are the same, so at least I can change the frequency from python-ivi and do the rest from buttons.

I really like python-ivi, but it sunk me way too much time.  After all, I don't think any body else will reuse my automated test software, not even myself, so I can just use SCPI.  It's been 10 days since I've started to fool around with python-ivi and still tinkering around it instead of working at the data processing, which was the actual goal of the project.

For me python-ivi is still worth it because it provides a fairly sensible way to abstract instruments, so I can use the same code for a Tektronix scope and a Lecroy scope, for example, which as a hobbyist with a heterogeneous lab with instruments from different manufacturers is very helpful. Even though I've had to develop most of the drivers that I use myself, this was a one-time effort. I don't just have three identical instruments to throw at the same problem, so if I need multiple scopes, power supplies or multimeters at the same time, they'll probably of different brands and vintages. I have not found an abstraction I liked better than python-ivi yet, and I certainly don't feel like inventing my own.

[RoGeorge]

It looks like either python-ivi or pyvisa will use the same python-vxi11 module (import vxi11) to connect to an instrument over LAN.  I've tested and they are both very slow, it takes 23 minutes to fetch 24MSa.

Code: [Select]

# pip install python-vxi11
# Successfully installed python-vxi11-0.9
import vxi11

dso = vxi11.Instrument('192.168.1.3')
# alternatives (transfer speed is the same):
# dso = vxi11.Instrument("TCPIP::192.168.1.3::INSTR")

print(dso.ask('*IDN?'))



dso.write(':RUN')
while dso.ask('*OPC?') != '1': print('Waiting for command to finish')
dso.write(':ACQ:MDEP 24000000')     # !!! can only be set when in :RUN mode
while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

dso.write(':WAV:MODE MAX')          # !!! 1200 points in :RUN mode, all MDEP points in STOP mode
while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

dso.write(':WAV:FORM BYTE')
while dso.ask('*OPC?') != '1': print('Waiting for command to finish')



dso.write(':WAV:SOUR CHAN1')
while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

dso.write(':WAV:STAR 1')
while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

dso.write(':STOP')
while dso.ask('*OPC?') != '1': print('Waiting for command to finish')
dso.write(':WAV:STOP 250000')       # !!! will be ignored and always 1200 in :RUN mode
while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

wvfm_pre = dso.ask('WAV:PRE?')
while dso.ask('*OPC?') != '1': print('Waiting for command to finish')
print(wvfm_pre)

print(dso.ask('WAV:MODE?'), dso.ask('WAV:STAR?'), dso.ask('WAV:STOP?'))



dso.write(':STOP')
while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

import time
t0 = time.time()
wvfm = dso.ask_raw(b':WAV:DATA?')      # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP mode
t1 = time.time() - t0
print('Time to fetch the first waveform chunk: ' , t1)
print(type(wvfm), len(wvfm))
while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

to_fetch = 1_000_000    # fetch only 1Msa of ADC samples
MAX_CHUNK = 250_000
M_DEPTH = int(dso.ask(':ACQ:MDEP?'))

all_wvfm = bytes()
t0 = time.time()
for i in range(0, min(to_fetch, M_DEPTH), MAX_CHUNK):

    dso.write(':WAV:STAR ' + str(i+1))
    while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

    dso.write(':WAV:STOP ' + str(i + min(MAX_CHUNK, M_DEPTH, to_fetch-i)))
    while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

    wvfm = dso.ask_raw(b':WAV:DATA?')      # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP mode
    all_wvfm += wvfm[12:]

    print(len(wvfm))

t1 = time.time() - t0
print('Time to fetch ' + str(to_fetch) + ' ADC samples using "python-vxi11": ' + str(t1))



dso.close()



If I open a connection with SOCKET, or if I use something like the 'socketscpi' module, it fetches 24MSa in 23 seconds. 

Code: [Select]

# socketscpi is using direct TCP socket instead of VXI-11
# pip install socketscpi
# Successfully installed socketscpi-2022.8.0
#   might work as a backend for PyVISA ????? for LXI instruments
#   https://github.com/morgan-at-keysight/socketscpi/blob/master/examples.py

import socketscpi

dso = socketscpi.SocketInstrument(ipAddress='192.168.1.3', port=5555, timeout=1)
# dso = socketscpi.SocketInstrument(ipAddress='192.168.1.3', port=5555, globalErrCheck=True)

print(dso.instId)



dso.write(':RUN')
while dso.query('*OPC?') != '1': print('Waiting for command to finish')
dso.write(':ACQ:MDEP 24000000')     # !!! can only be set when in :RUN mode
while dso.query('*OPC?') != '1': print('Waiting for command to finish')

dso.write(':WAV:MODE MAX')          # !!! 1200 points in :RUN mode, all MDEP points in STOP mode
while dso.query('*OPC?') != '1': print('Waiting for command to finish')

dso.write(':WAV:FORM BYTE')
while dso.query('*OPC?') != '1': print('Waiting for command to finish')



dso.write(':WAV:SOUR CHAN1')
while dso.query('*OPC?') != '1': print('Waiting for command to finish')

dso.write(':WAV:STAR 1')
while dso.query('*OPC?') != '1': print('Waiting for command to finish')

dso.write(':STOP')
while dso.query('*OPC?') != '1': print('Waiting for command to finish')
dso.write(':WAV:STOP 250000')       # !!! will be ignored and always 1200 in :RUN mode
while dso.query('*OPC?') != '1': print('Waiting for command to finish')

wvfm_pre = dso.query('WAV:PRE?')
while dso.query('*OPC?') != '1': print('Waiting for command to finish')
print(wvfm_pre)

print(dso.query('WAV:MODE?'), dso.query('WAV:STAR?'), dso.query('WAV:STOP?'))



dso.write(':STOP')
while dso.query('*OPC?') != '1': print('Waiting for command to finish')

import time
t0 = time.time()
wvfm = dso.query(':WAV:DATA?')      # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP mode
t1 = time.time() - t0
print('Time to fetch the first waveform chunk: ' , t1)
print(type(wvfm), len(wvfm))
while dso.query('*OPC?') != '1': print('Waiting for command to finish')

to_fetch = 1_000_000    # fetch only 1Msa of ADC samples
MAX_CHUNK = 250_000
M_DEPTH = int(dso.query(':ACQ:MDEP?'))

all_wvfm = b''
t0 = time.time()
for i in range(0, min(to_fetch, M_DEPTH), MAX_CHUNK):

    dso.write(':WAV:STAR ' + str(i+1))
    while dso.query('*OPC?') != '1': print('Waiting for command to finish')

    dso.write(':WAV:STOP ' + str(i + min(MAX_CHUNK, M_DEPTH, to_fetch-i)))
    while dso.query('*OPC?') != '1': print('Waiting for command to finish')

    wvfm = dso.query(':WAV:DATA?')      # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP mode

    b = bytes(wvfm[12:], 'latin_1')
    all_wvfm += b

    print(len(wvfm))

t1 = time.time() - t0
print('Time to fetch ' + str(to_fetch) + ' ADC samples using "socketscpi": ' + str(t1))



dso.close()
The downside is if I send a wrong command I have to power cycle the oscilloscope.



My oscilloscope is Rigol DS1104Z (in python-ivi its driver module is 'rigol.rigolDS1054'), it has the fixed IP 192.168.1.3, and the port for SOCKET access is 5555.

Do you have a working code example, please, for connecting to the oscilloscope from python-ivi, or even from PyVISA, but using SOCKET instead of VXI11, and then print the result of the '*IDN?'

[alm]

I don't have any Rigol scope to test on, so I used netcat to create a listening TCP socket: nc -l -p 1052. Then I installed python-ivi from your Github, removed all the Python-2 style print statements that would give errors in Python 3 (after from __future__ import print_function, print(...) with parentheses should work in any Python version released after 2008), installed pyvisa and pyvisa-py, and made a small change to ivi/interfaces/pyvisa.py lines 32-39 and replaced the word visa by pyvisa. Apparently the module changed names in the past six years. And then it seems to work fine for me:

Code: [Select]

% python
>>> import ivi
>>> rig = ivi.rigol.rigolDS1054Z('TCPIP0::localhost::1052::SOCKET')
# This was necessary because I'm using a Linux terminal and not a real DS1054 scope
>>> rig._interface.instrument.read_termination = '\n'
>>> rig._interface.instrument.write_termination = '\n'
>>> rig.identity.instrument_model
'DS1102E'

In netcat I'd see:

% nc -l -p 1052
*CLSRIGOL TECHNOLOGIES,DS1102E,DS1EB104702974,00.02.01.01.00
*IDN?
Red is what the driver sent, and green is what I typed to simulate the response of a scope.

*CLS is what the DS1054Z driver sends when you instantiate it, and RIGOL TECHNOLOGIES,... is what I typed as a pre-emptive response to the *IDN? query since I can't type fast enough to stay within the timeout. I'm pretty sure that the same Python code, probably without changing the termination, would work with a real scope.

[RoGeorge]

Sorry for the invalid Python2 prints, not mine, I've removed them only locally, now removed them from my git fork, too.
The problem is that I get invalid resource string for something like:
>>> rig = ivi.rigol.rigolDS1054Z('TCPIP0::localhost::1052::SOCKET')

Code: [Select]

...
python-ivi_RoGeorge_fork/python-ivi/ivi/ivi.py", line 1817, in _initialize
    raise IOException('Invalid resource string')
ivi.ivi.IOException: Invalid resource string
>>>
If that resource string worked for you, then you must be having one of the proprietary VISA libraries installed, something like NI-VISA, Keysight VISA, R&S VISA, Tektronix VISA, and python-IVI or maybe pyvisa detects the closed source library and sends the resource string to it.

I don't have any of the proprietary VISA libraries, only the FOSS python modules:  python-ivi, python-vxi11, pyvisa and pyvisa-py.  IIRC pyvisa also has some options to configure its backend.

Also I've noticed your prompt is % (% prompt is usually for csh).  What OS did you used to test the line "ivi.rigol.rigolDS1054Z('TCPIP0::localhost::1052::SOCKET')"?

TL DR;
'TCPIP0::localhost::1052::SOCKET' doesn't work on my Kubuntu 22.04 LTS, I have no proprietary NI-VISA or alike installed.  Any hints what am I missing, or mis-configured, or what should I install to make my python-IVI work with SOKET connection just like yours, please?

[alm]

This was on Ubuntu 22.04.1 LTS without any proprietary VISA installed, just PyVISA and PyVISA-py. These are the only modules installed inside the virtual environment where I ran these commands:

Code: [Select]

python = "^3.10" (resolved to Python 3.10.6)
PyVISA = "^1.12.0" (resolved to 1.12.0)
PyVISA-py = "^0.5.3" (resolved to 0.5.3)
numpy = "^1.23.4" (resolved to 1.23.4)
python-ivi = {git = "https://github.com/RoGeorge/python-ivi.git"}
and as indirect dependency of PyVISA typing-extensions 4.4.0

PyVISA used the PyVISA-py backend without any configuration on my side. This is with using PyVISA directly:

Code: [Select]

>>> import pyvisa
>>> rm = pyvisa.ResourceManager()
>>> res = rm.open_resource('TCPIP0::localhost::1052::SOCKET')
>>> res.read_termination = '\n'
>>> res.write_termination = '\n'
>>> res.write('*IDN?')
7
>>> res.read()
'ME.' (this is just what I typed in netcat)

If this doesn't work for you, then maybe try explicitly setting the PyVISA-py backend as described here. I did have to modify ivi/interface/pyvisa.py as I described above before python-ivi would detect pyvisa. Without that modification I also got the Invalid resource string error.

[RoGeorge]

Thanks for pointing to the visa -> pyvisa name changes, wouldn't have noticed that too soon, if ever!
Now it can open a SOCKET connection from python-ivi. 

[alm]

Yeah, I guess the module was called "visa" back in 2015 when that code was written. I've never run into it myself because all devices I've controlled so far have a GPIB interface that I access via VXI-11 using a LAN to GPIB bridge (HP/Agilent, Tektronix, etc). So I have no use for anything but VXI-11 for now.

Nice, does it work with the DS1054Z now?

[RoGeorge]

The connection, yes it works, the PythonIVI driver for DS1054Z, not really.  For example I was expecting to see the <LF> (0x0a) at the end of the SCPI commands, but the PC doesn't send 0x0a in the LAN packets.  The oscilloscope replies with the expected 0x0a terminator.  Will look closer these days.

[alm]

Did you try playing with read and write termination attributes that I set in my samples? I'd expect setting write termination to '\n' to fix the problem of missing line feeds when sending to the scope.

[RoGeorge]

Only took a brief look, it was very late yesterday.  The read or write_termination are from pyvisa, ivi doesn't seem to have them.  I guess I should configure the pyvisa backend before making a connection from ivi, not sure how to do it.

There is not much documentation, and I'm not a programmer.  Digging through code is much more difficult then reading a manual.  Overriding or extending classes bamboozles me a lot, since I don't have yet a clean map of the objects used in PythonIVI. 

[alm]

See my earlier posts on how to set termination through python-ivi.

[RoGeorge]

I don't know how to apply read/write_termination before opening the instrument.

If I open the instrument first, by default the driver sends '*CLS' without the 0x0a terminator, which hangs the oscilloscope communication, and can recover only by a power cycle.  By looking through sources, found out that there are optional parameters possible to send when opening a new connection, one of the parameters being to not emit a "*CLS" command by default. 

Code: [Select]

import ivi
ivi.set_prefer_pyvisa()

dso = ivi.rigol.rigolDS1104Z(resource='TCPIP0::192.168.1.3::5555::SOCKET', id_query=False, reset=False)
dso._interface.instrument.read_termination = '\n'
dso._interface.instrument.write_termination = '\n'

dso.identity.instrument_model
dso._ask('*IDN?\n')



while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')
dso._write(':ACQ:MDEP 24000000\n')     # !!! can only be set when in :RUN mode
while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

dso._write(':WAV:MODE MAX\n')          # !!! 1200 points in :RUN mode, all MDEP points in STOP mode
while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

dso._write(':WAV:FORM BYTE\n')
while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

dso._write(':RUN\n')



dso._write(':WAV:SOUR CHAN1\n')
while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

dso._write(':WAV:STAR 1\n')
while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

dso._write(':STOP\n')
while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')
dso._write(':WAV:STOP 250000\n')       # !!! will be ignored and always 1200 in :RUN mode
while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

wvfm_pre = dso._ask('WAV:PRE?\n')
while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')
print(wvfm_pre)

print(dso._ask('WAV:MODE?\n'), dso._ask('WAV:STAR?\n'), dso._ask('WAV:STOP?\n'))



dso._write(':STOP\n')
while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

import time
t0 = time.time()
wvfm = dso._ask_raw(b':WAV:DATA?\n')      # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP mode
t1 = time.time() - t0
print('Time to fetch the first waveform chunk: ' , t1)
print(type(wvfm), len(wvfm))
while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

to_fetch = 24_000_000    # fetch only 1Msa of ADC samples
MAX_CHUNK = 250_000
M_DEPTH = int(dso._ask(':ACQ:MDEP?\n'))

all_wvfm = b''
t0 = time.time()
for i in range(0, min(to_fetch, M_DEPTH), MAX_CHUNK):

    dso._write(':WAV:STAR ' + str(i+1) + '\n')
    while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

    dso._write(':WAV:STOP ' + str(i + min(MAX_CHUNK, M_DEPTH, to_fetch-i)) + '\n')
    while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

    wvfm = dso._ask_raw(b':WAV:DATA?\n')      # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP mode

    # b = bytes(wvfm[12:], 'latin_1')
    # all_wvfm += b
    all_wvfm += wvfm[12:]

    print(len(wvfm))

t1 = time.time() - t0
print('Time to fetch ' + str(to_fetch) + ' ADC samples using "socketscpi": ' + str(t1))

dso.close()

This makes possible to connect to an instrument without sending any (wrong terminated) commands, then configure read/write_termination, which worked.  However, the read/write_termination settings seems to apply only for ivi functions.  For example, if I use osc._ask_raw(':WAV:DATA?\n'), then I must add the 0x0a manually, as '\n'.

I had to look more into the ivi sources.  Maybe there is a way to set the automatic addition of the 0x0a terminator in pyvisa-py, so I won't end with a mixture of functions where some requires 0x0a while others don't.

The transfer speed is 65 seconds to fetch 24MSa.

Not bad, but still 3 times slower than the 22 seconds I get when downloading with https://github.com/morgan-at-keysight/socketscpi
I wonder if it would be possible to use 'socketscpi' instead of 'pyvisa', or maibe to configure 'pyvisa' to use 'socketscpi' instead of 'pyvisa-py'.

I've uploaded my current PythonIVI changes on gitlab, https://gitlab.com/RoGeorge/python-ivi, because github got crazy and doesn't let me any more to remote update my own repositories, unless I got a security token from them, which I don't want.  Adios GitHub, hello GitLab.