SDS1204X-E scope freezing when collecting waveform

[AnarchistWolf]

Hello,

Currently I am trying to get my SDS1204X-E scope to collect the waveform through TCP using Julia as the programming language (Though i have tested with Python). When i collect the waveform and process the data about 10-20 times the scope suddenly freezes. I need to acquire data like every second or so for hours on end.

Ive already ruled out possible packet problems using wireshark. ive played with the delays/sleeps in my code to make sure it has enough time to process the data but doesnt help no matter the interval.

What is funny to me is that we have a Rigol DS1102Z-E scope thats like a fraction of the cost and i can actually get working with data acquisition every 1 second for more that 12 hours. (To me this is an indicator that my code isnt necessarily the problem.)

Ive contacted Siglent and they have never seen/heard of this problem so im seeing if maybe someone else has some suggestions or comments on the matter. (Id like ya know the more expensive 4 channel oscilloscope to work compared to the Rigol, which while it works i think it is causing some slight problems in reading the waveform.)

Thanks,
W

[tautech]

Welcome to the forum.

Sorry I don't have any answers, only questions.
Please state the firmware version in use, the latest is 37R9.
https://int.siglent.com/download/firmwares/?ProId=12

What format is the data you're grabbing ?
Could you show us a screenshot from the USB port ?
Have you worked through the pages of operating tips and App notes ?
https://siglentna.com/operating-tips/digital-oscilloscopes/sds1000x-e-series/
https://siglentna.com/application-notes/sds1000x-e-series/

Maybe this could work better for you:
https://siglentna.com/application-note/lxi-tools/

[AnarchistWolf]

Hello,
I am running the latest Firmware (The one you have listed).

The format is in hexadecimal, 14k points (Lowest mem depth). I wouldnt be able to show you a screenshot because the scope freezes, not allowing anything. I could send a picture but it would just be of a static scope. I have gone through the tips and tried looking through many forums (and also contacting Siglent themselves multiple times) but havent found much.

The reason i am using a TCP connection is so i dont have to install libraries, i dont have to be near the instrument, and so that my code can read and process the data in realtime without having to download and then read the data. 

I hope those answered some of your questions,
Best,
W

[tautech]

The format is in hexadecimal

Sorry, I should've asked for the filetype.

As you can see from the screenshot only certain types are available.

[AnarchistWolf]

When sending the WF? (Waveform acquire) SCPI commands i do not get a file back, i get a header and an array of hexadecimal units.

[tautech]

When sending the WF? (Waveform acquire) SCPI commands i do not get a file back, i get a header and an array of hexadecimal units.

Does your WF? command syntax comply with the conditions listed on P262 and on from the programming manual ?
https://siglentna.com//wp-content/uploads/dlm_uploads/2021/01/SDS1000-SeriesSDS2000XSDS2000X-E_ProgrammingGuide_PG01-E02D.pdf

Have you considered using the new Logging feature to collect your data ?
https://siglentna.com//wp-content/uploads/dlm_uploads/2021/08/Data-Logger-of-SDS-Application-Notes_EN01A.pdf
However it seems the logging can only be saved locally after which you'd need to physically transfer it to a PC.

[bd139]

To note, if it works once it should work N times where N is > 20. Smells like a bug. But at which end?

To isolate this I would list carefully:

1. How the scope is connected to the computer.
2. The type of host computer and operating system.
3. A minimum subset of the code that reproduces the issue.

I have a 1202X-E here I can probably set up quickly to try and reproduce it.

This could be anything from an reentrancy issue on the scope side to a threading bug on the client side.

[AnarchistWolf]

Yes I have followed the programming guide to a T. I've gone through their settup and code guide too on how it should function.

They might have missed something in their manual (as I've seen not all commands are documented readily) so I could just be missing part of the puzzle because I am using it for a niche use case I think. (Could be wrong.)

[AnarchistWolf]

Hello,
The scope is connected by ethernet in a local managed network but i have tried a direct connection using a crossover ethernet cable and it still froze. (I havnt tried serial because the whole point is to not use libraries.)

I mainly code and run the script on a windows computer but we do use Ubuntu to run the code overnight and it still hangs on those computers. (Havnt tried Mac but would expect the same results.)

Im interested too if its my code or something wrong with the scope (At this point i think both, but leaning more towards it being a problem with the scope because im missing something in my code they dont have listed.)

I have used their basic python code as a test and just replaced it with the waveform command instead of *IDN? and it still hangs. (https://siglentna.com/application-note/open-socket-lan-python/). This code collects the waveform of the first channel 100 times.

Code: [Select]

#!/usr/bin/env python
#-*- coding:utf-8 –*-
#-----------------------------------------------------------------------------
# The short script is a example that open a socket, sends a query,
# print the return message and closes the socket.
#
#No warranties expressed or implied
#
#SIGLENT/JAC 05.2018
#
#-----------------------------------------------------------------------------
import socket # for sockets
import sys # for exit
import time # for sleep
#-----------------------------------------------------------------------------

remote_ip = "192.168.55.121" # should match the instrument’s IP address
port = 5025 # the port number of the instrument service

#Port 5024 is valid for the following:
#SIGLENT SDS1202X-E, SDG2X Series, SDG6X Series
#SDM3055, SDM3045X, and SDM3065X
#
#Port 5025 is valid for the following:
#SIGLENT SVA1000X series, SSA3000X Series, and SPD3303X/XE

count = 0

def SocketConnect():
    try:
        #create an AF_INET, STREAM socket (TCP)
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    except socket.error:
        print ('Failed to create socket.')
        sys.exit();
    try:
        #Connect to remote server
        s.connect((remote_ip , port))
    except socket.error:
        print ('failed to connect to ip ' + remote_ip)
    return s

def SocketQuery(Sock, cmd):
    try :
        #Send cmd string
        Sock.sendall(cmd)
        Sock.sendall(b'\n')
        time.sleep(1)
    except socket.error:
        #Send failed
        print ('Send failed')
        sys.exit()
    reply = Sock.recv(4096)
    return reply

def SocketClose(Sock):
    #close the socket
    Sock.close()
    time.sleep(1)

def main():
    global remote_ip
    global port
    global count

    # Body: send the SCPI commands C1:WF? DAT? 100 times and print the return message
    s = SocketConnect()
    for i in range(100):
        qStr = SocketQuery(s, b'C1:WF? DAT2')
        time.sleep(0.5)
        print (str(count) + ":: " + str(qStr))
        count = count + 1
    SocketClose(s)
    print('Query complete. Exiting program')
    sys.exit

if __name__ == '__main__':
    proc = main()

If you know Julia i can also send a Pluto notebook but i think python is much more universal.

[bd139]

Ok will get back to you later once I’ve had a chance to look at it 

[bd139]

Ok had a few mins while on a pointless zoom call.

I'm running macOS 12.4 python 3.8.9 --> SDS1202X-E running 1.3.26 over wifi then ethernet.

Mine successfully runs everything up to "Query complete. Exiting program". At that point none of the physical controls are working. The network interface stops responding then as well which requires that I power cycle it by physically pulling the plug on it. I suspected that this would be related to LOCK being initiated when switching into remote mode which is a possibility but the whole network side pukes so I can't even validate that.

This is definitely a scope bug that is reproducible.

Edit: the firmware I have appears to be the latest published.

Edit 2: if you ctrl+C the python process then the whole scope hangs solid as well.

That is quite frankly shit!

[2N3055]

So code opens the socket, asks for data 100 times and then closes the socket.
Could somebody try to open and close socket before and after each data transfer..
Maybe some buffers are not alloc/dealloc correctly, and they might get released with the socket destroy..
I'm not saying it's a solution but might point to exactly where the problem is...

[bd139]

Edit: just added that test case.

More info.

1. If you open the socket, run 100 commands through, then close it ->  CRASH.
1. If you open the socket and close it immediately without executing any commands -> NO CRASH.
2. If you open the socket, run one command and close it -> NO CRASH.
3. If you open the socket 100 times and run one command then close it, it takes bloody forever -> NO CRASH

So workaround is:

Code: [Select]

    for i in range(100):
        s = SocketConnect()
        qStr = SocketQuery(s, b'C1:WF? DAT2')
        time.sleep(0.5)
        print (str(count) + ":: " + str(qStr))
        count = count + 1
        SocketClose(s)

So it can't be trusted to execute more than one command per connection reliably. Smells like some stupid C somewhere.

Someone at Siglent should get a JTAG debugger on it and gdb, crash it, get a bt and fix this ASAP. Big  .

[tv84]

Edit: just added that test case.

More info.

1. If you open the socket, run 100 commands through, then close it ->  CRASH.
1. If you open the socket and close it immediately without executing any commands -> NO CRASH.
2. If you open the socket, run one command and close it -> NO CRASH.
3. If you open the socket 100 times and run one command then close it, it takes bloody forever -> NO CRASH

So workaround is:

Code: [Select]

    for i in range(100):
        s = SocketConnect()
        qStr = SocketQuery(s, b'C1:WF? DAT2')
        time.sleep(0.5)
        print (str(count) + ":: " + str(qStr))
        count = count + 1
        SocketClose(s)

So it can't be trusted to execute more than one command per connection reliably. Smells like some stupid C somewhere.

Someone at Siglent should get a JTAG debugger on it and gdb, crash it, get a bt and fix this ASAP. Big  .

0.5 is the smallest delay you can get reliably?

I think you will always have a limit in that number of consecutive operations. It's better for the user to control it in a specific situation than for Siglent to put a hardcoded limit that will affect everybody in all situations.

[bd139]

There's no concurrency here. Operations always fully return before the next one starts.

Either way it should be reentrant and shouldn't crash the remote end. If it's not that's a bug.

[tv84]

There's no concurrency here.

Concurrency is what a scope never lacks...

What happens when the SCPI command inside the 100x loop is only "*IDN?"?

(It could be a bug, a feature and/or a buggy feature...) 

[bd139]

Ok just ran that and it doesn't crash if it's just '*IDN?' and a read back. So it's the WF? query doing it.

[AnarchistWolf]

Hello,

I appreciate all the feedback on this! Im glad i am not crazy, but also pissed its a problem.

I am not at my scope at the moment but my theory is since Siglent has such a weird history thing going on its actually storing the waveforms to internal memory, then its hanging because it is filling its memory and cant read or accept anymore commands.

I could be wrong on that but thats the only thing i can think of. All other commands like you just did work fine and can be ran hundreds of times. To get the actual voltage correct for the waveform you need to run many more commands than just WF?, but those run fine and i have never had a problem.

Thanks,
W

[bd139]

Well it's doing everything mostly correct.

I got it to entirely hang once with the display frozen but sometimes it would leave the display and acquisition running as well with just they keyboard dead. Definitely a sign of something probabilistic i.e. memory related. I'd say it was most likely a simple memory management or pointer handling bug somewhere.

I'm actually quite disappointed with this as well.

Pokes tautech... 

[bdunham7]

I am not at my scope at the moment but my theory is since Siglent has such a weird history thing going on its actually storing the waveforms to internal memory, then its hanging because it is filling its memory and cant read or accept anymore commands.

Since you only want data once per second, you could try setting the holdoff to something like 0.5s to reduce the scope's internal workload.  What timebase, etc are you doing your capture at?

[AnarchistWolf]

Id like to acquire the waveform every second or so (If your interested what im doing is Cavity Ring Down). for hours at a time, days if possible.

Im actually quite new to lingo in scopes so i havnt heard of holdoff. As for timebase it changes depending on how many waveforms i want to capture but that last i had set it to was 200ms/division.

Edit: More info, Freq is at 7540Hz, and i can get about 16 waveforms on the screen with that.

[bdunham7]

Id like to acquire the waveform every second or so (If your interested what im doing is Cavity Ring Down). for hours at a time, days if possible.

Im actually quite new to lingo in scopes so i havnt heard of holdoff. As for timebase it changes depending on how many waveforms i want to capture but that last i had set it to was 200ms/division.

Edit: More info, Freq is at 7540Hz, and i can get about 16 waveforms on the screen with that.

I'll assume from your other info that you meant 200µS (microseconds) not milliseconds.  OK, so holdoff is a delay before the scope retriggers.  If you set it to 0.5s, it will not capture more than 2 screens per second.  You can find it in the trigger menu.

[AnarchistWolf]

Okay, so i have my trigger set a freq and im reading the waveform produced by a laser pulsing and some decay time.
Again maybe i dont understand this well enough but the scope is constantly being triggered on that freq. I am not stopping it and retriggering it every waveform capture.

(Unless its just doing that automatically and again im to new to know about it.)

Thanks,
W

[2N3055]

There's no concurrency here.

Concurrency is what a scope never lacks...

What happens when the SCPI command inside the 100x loop is only "*IDN?"?

(It could be a bug, a feature and/or a buggy feature...)

When you execute *IDN? only, it doesn't have to allocate multi megabyte application buffer and adjacent stuff to TCP that to the other side...
One more thing to try is to try different sample memory size to see if it crashes faster or slower.

It smells like running out of memory to the point system is starved for resources and crash. Probably not deallocating all memory, but not happening when closing sockets might point to problem being with TCP transfer buffers..

Also a single capture-transfer cycle could be tried... But those are workarounds..

For OP use "open socket-transfer-close socket" once a second does the job though...

I would prefer to gather quality info so Siglent can easily reproduce and fix the deficiency if there is one...

[tv84]

It smells like running out of memory to the point system is starved for resources and crash. Probably not deallocating all memory, but not happening when closing sockets might point to problem being with TCP transfer buffers..

Too much vague to smell something specific. Although my nose smells concurrency. 

If a loop with 1 works and a loop with 100 doesn't work, how about a loop with 50, or 25, etc?

Once we get to a stable number, let's see how many times that stable number is stable. Always, failing occasionally, etc, etc?

This theme needs more info.