Pocket-Sized 6 GHz 1 TS/s ET Scope

[joeqsmith]

Power cycle did shake it loose.  If an automotive companies firmware required you to pull the battery terminal every now and then, would you find that acceptable?  What about an aircraft?     

I started to write some simple test scripts to stress test your product and it seems easy enough to reproduce the lockups.  I could provide you with a sequence of commands that will cause it to hang.  I suspect you could just go through your firmware and clean that up.    I've also now seen where the scope will spam the data bus, sending  "FAIL 752440 0"  non stop for example.   The manual makes no mention of it.   Maybe it means something to you. 

Not having your firmware source I can't really guide you.  I guess we can rug sweep it like the double cal and hope no one runs into it.     

[SJL-Instruments]

I started to write some simple test scripts to stress test your product and it seems easy enough to reproduce the lockups.  I could provide you with a sequence of commands that will cause it to hang.  I suspect you could just go through your firmware and clean that up.    I've also now seen where the scope will spam the data bus, sending  "FAIL 752440 0"  non stop for example.   The manual makes no mention of it.   Maybe it means something to you. 

Could you provide the command sequence? We will try to reproduce the hang.

Are you able to reliably reproduce the "fail" message? What was happening just before it occurred?

[joeqsmith]

I started to write some simple test scripts to stress test your product and it seems easy enough to reproduce the lockups.  I could provide you with a sequence of commands that will cause it to hang.  I suspect you could just go through your firmware and clean that up.    I've also now seen where the scope will spam the data bus, sending  "FAIL 752440 0"  non stop for example.   The manual makes no mention of it.   Maybe it means something to you. 

Could you provide the command sequence? We will try to reproduce the hang.

Are you able to reliably reproduce the "fail" message? What was happening just before it occurred?

The firmware will need improvements before I could hunt down your problem with the spam messages.    Send the following two commands.  Scope will respond with "CH1 BAD: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0".   Close the test software and try to connect with your software.

5332 DA62 2D43 A11C 1456 0D
74F5 71ED 8586 AEF6 3B79 0D

[SJL-Instruments]

The firmware will need improvements before I could hunt down your problem with the spam messages.    Send the following two commands.  Scope will respond with "CH1 BAD: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0".   Close the test software and try to connect with your software.

5332 DA62 2D43 A11C 1456 0D
74F5 71ED 8586 AEF6 3B79 0D

The first command is interpreted as "S2", since the third byte (0xDA) is non-printable and the rest of the command is ignored.
This will cause subsequent commands to fail, as a minimum of 3 triggers per sample (S3) is required. The default value is 4096.
We will document this in the next manual revision.

[joeqsmith]

Try this script, should get the same response.  Scope should not reconnect after sending them.   Both scripts fail 100% of the time. 
6A22 912D 41A5 DCC3 8EA6 0D
7832 9A43 72E4 011D 2F2A 0D
2491 424C 10D3 D3E3 A99C 0D
7467 A795 2B74 ADB8 15F1 0D

IMO, the firmware should NEVER require a hard reset like this.   Normally, I would subject it to random packets, varying lengths, different terminators, patterns, speeds, attempt overflows....  Just your basic 101 stuff.  That's all am doing now.   That's how I made it spam the bus.  The CH1 BAD:  fault is the most prevalent.  I suspect the firmware is just not well written to make sure it handles packet faults.  You may be better off just going over it than trying to hunt down why a a specific set of commands takes it out. 

[joeqsmith]

The first command is interpreted as "S2", since the third byte (0xDA) is non-printable and the rest of the command is ignored.
This will cause subsequent commands to fail, as a minimum of 3 triggers per sample (S3) is required. The default value is 4096.
We will document this in the next manual revision.

I think you are missing the point.  I don't think pointing out that your firmware is not robust and to treat it like a princess or it will require a power cycle is time well spent.  Far better to harden the firmware so it handles all unforeseen conditions gracefully.  Again just MO. 

[joeqsmith]

Of course, if you had a simple checksum and other checks builtin to the protocol like most,  I would still write scripts to create valid checksums with an invalid payload.   I like things that are solid.   If it were a $50 nanovna, I would say who cares.

[SJL-Instruments]

The first command is interpreted as "S2", since the third byte (0xDA) is non-printable and the rest of the command is ignored.
This will cause subsequent commands to fail, as a minimum of 3 triggers per sample (S3) is required. The default value is 4096.
We will document this in the next manual revision.

I think you are missing the point.  I don't think pointing out that your firmware is not robust and to treat it like a princess or it will require a power cycle is time well spent.  Far better to harden the firmware so it handles all unforeseen conditions gracefully.  Again just MO. 

We understand your point, and intend to harden the firmware, likely in the same release where we add SCPI support.
Until then, we want to document where all the edge cases occur. Our comment was to point out that your fuzzing uncovered one of the bounds we forgot to document - we wanted to be public about this change to the documentation.

[joeqsmith]

I'll try to work around the firmware and see if I can replicate the spamming issue. 

With that first script, sending only the first command should not hang the scope.  It should still respond to other commands.  It seems to need both to cause the scope to only send the CH1 BAD response. 

Attached 1st column is sent command in ASCII, 2nd is command in HEX, 3rd is the response.   T is not a documented command in the manual.   A lot of these where it responds, it has an invalid payload that does not match with the manual.  The firmware decides to use it anyway.      My guess is some random miss without any error checking/recovery and then interpreting bad payloads as somehow good it the reason I have seen the scope hang when using it normally.   Personally, adding SCPI doesn't really bring much.  Hardening what you have would be far more beneficial.   

[SJL-Instruments]

I'll try to work around the firmware and see if I can replicate the spamming issue. 

With that first script, sending only the first command should not hang the scope.  It should still respond to other commands.  It seems to need both to cause the scope to only send the CH1 BAD response. 

Attached 1st column is sent command in ASCII, 2nd is command in HEX, 3rd is the response.   T is not a documented command in the manual.   A lot of these where it responds, it has an invalid payload that does not match with the manual.  The firmware decides to use it anyway.      My guess is some random miss without any error checking/recovery and then interpreting bad payloads as somehow good it the reason I have seen the scope hang when using it normally.   Personally, adding SCPI doesn't really bring much.  Hardening what you have would be far more beneficial.   

We're working on hardening the firmware. The command parser itself is solid. As a first pass, we just need to add range checks for the parameters in each command.
The T command is aliased to the *TST? command. We will document this in the next manual revision.
For clarity, the sequence S2, then T will reproduce your result.

[joeqsmith]

What about a these?  None are valid commands at the time it crashes.   

3&4, I changed my test to run up till five failures so you can see how the scope only returns CH1 BAD responses. 

I suspect you are going to spend a lot of time documenting  outlier cases.  IMO, it would be better to address the root problem.  Once the communications are robust, you can start to build off that solid foundation.

[SJL-Instruments]

What about a these?  None are valid commands at the time it crashes.   

3&4, I changed my test to run up till five failures so you can see how the scope only returns CH1 BAD responses. 

I suspect you are going to spend a lot of time documenting  outlier cases.  IMO, it would be better to address the root problem.  Once the communications are robust, you can start to build off that solid foundation.

The current manual (Section 4.5.1) states that issuing an H command with parameter below 40 will cause undefined behavior.
This will be caught and rejected when we add the bounds checking.

***

We are also restructuring the serial parser to be inherently more robust. We will perform our own fuzzing tests after the rewrite. When we are happy with the results of these tests, we can send you a prerelease firmware update if you'd like.

[joeqsmith]

As I add more and more case statements to avoid having your scope lock up, it is running longer but I have yet to reproduce the spam problem. 

At the time it did this, I was sending a 10-Byte payload + CR.  I have change my test to now send variable payloads.   Hopefully I can get it to repeat and trap it. 

[joeqsmith]

The first command is interpreted as "S2", since the third byte (0xDA) is non-printable and the rest of the command is ignored.

The current manual (Section 4.5.1) states that issuing an H command with parameter below 40 will cause undefined behavior.
...
When we are happy with the results of these tests, we can send you a prerelease firmware update if you'd like.

s,S,h,H are not being sent, along with seven other codes that have cause the scope to hang.   As you can see, it still hangs.  Because your firmware seems to in some cases ignore the first byte of the command (7F,54 for example will interpret as a T), I now also prevent all of these cases on the second byte.  Basically I will continue to increase the test cases until you provide improved firmware, or I find a combination that allows the scope to run long enough that the spam error repeats.     

I have no problem running any prerelease firmware,  now that I am setup to reprogram the device.  FP stands for Field Programmable.  As the product evolves, you may want to consider how to update the FPGA as well as the microcontroller across the USB interface.   Guessing it would require a hardware change.   

[joeqsmith]

We used the 1708 serial interface for automotive.  Something like a RS485/422.  One of my motorcycles uses something similar.   We were using J1587  protocol.  We adopted a similar protocol for other industrial projects I have been involved with.   The following link provides a simple overview of it:

https://www.kvaser.com/about-can/can-standards/j1587/

Note this is a very old standard but may still be in use today.  It has certainly stood the test of time.  Maybe have a quick read and see if there is something you can leverage from an existing standard like this.   


 

[joeqsmith]

Assuming that "Unknown command!"s are ignored  and not causing the scope to hang,  I have added a filter for them.  Even with my 12 test cases in the two first fields, it will still hang..    Changing to the first three fields. 

[joeqsmith]

Adding a check for the first three fields wasn't enough so it now checks every field.   Even with all of this, the scope will lock up. 

About 4 hours into this hunt and about the only thing I have to show for it is worn out USB connectors.   

[joeqsmith]

Maybe you can tell the cause of the FAIL 752440 0 from the attached dump.   When I exited the script and attempted to run your software, it threw the attached exception. 

[joeqsmith]

So, another new one.  The scope stops responding all together.   Exiting the test script and attempting to run your software, it does not find the scope. 

Lots of problems.  I will hold off doing anything more until you have time to go over your firmware.   Hopefully I have provided you with enough clues. 

[SJL-Instruments]

So, another new one.  The scope stops responding all together.   Exiting the test script and attempting to run your software, it does not find the scope. 

Lots of problems.  I will hold off doing anything more until you have time to go over your firmware.   Hopefully I have provided you with enough clues. 

We have just sent you a revised firmware version with a hardened parser and strict bounds checking. All the test cases you have shown no longer cause an issue. Our fuzzing tests are ongoing and have not revealed any issues so far.

Because your firmware seems to in some cases ignore the first byte of the command (7F,54 for example will interpret as a T), I now also prevent all of these cases on the second byte.

The byte 0x7F is a backspace character. This behavior is intentional, and simplifies use of the serial interface from an interactive terminal.

[joeqsmith]

First, thank you for taking the time to dig into the root cause.   I know it wasn't your intention to go after it so early but I am glad you did.   

It's installed and I am currently testing with all of my case statements removed.   My plan is to let it run overnight but it is over 150,000 cycles now.  Consider I was lucky if it made it past 10,000 with the old firmware.   So what ever changes you have made greatly improved it's robustness.   

***
Job well done! It ran all night without any further problems.  I also ran some other basic checks using your software as well as mine and it all seems to work. 

With the software and firmware shaping up, the only thing missing now is those 5X probes.        We can always revisit that.   Looks like no one had anything specific they wanted to see during the review.  My plan is to finish up this weekend and let you get back to your normal routine.      Again, good job.  You have made a lot of progress over this last month.   

[SJL-Instruments]

Job well done! It ran all night without any further problems.  I also ran some other basic checks using your software as well as mine and it all seems to work. 

With the software and firmware shaping up, the only thing missing now is those 5X probes.        We can always revisit that.   Looks like no one had anything specific they wanted to see during the review.  My plan is to finish up this weekend and let you get back to your normal routine.      Again, good job.  You have made a lot of progress over this last month.   

Thanks for the update, and glad the patch has resolved the problems.  We are continuing to test the firmware on several units. If all goes well, we will release it sometime next week.

[joeqsmith]

As promised,  I provided a preview of my review to SJL,  but we ran into small setback when looking at eye diagrams.  I had previously posted some data and it was suggested I increase the intensity to get a better view of what the eye looks like:

https://www.eevblog.com/forum/testgear/pocket-sized-6-ghz-1-tss-et-scope/msg5301991/#msg5301991

Even back then, then you can see that the eye diagram isn't all that easy to see what is happening.    The attached image is showing  a similar setup using the latest software.  With the advanced user settings I can attempt to increase the number if data points to add more detail to the eye, but the software errors out.  It seems that it limits the maximum data points to 3072.   It's a PC with basically unlimited resources.   Why is there such a limit?

I can certainly change that part of the review but the software really needs some improvements to better show the eye, before going ahead with it.  Once we have this sorted out, we can go ahead and release the review.   

[nctnico]

Color grading would be much better. The worse thing about intensity grading is that you are typically interested in the parts of the signal which occur the least. At low (or near zero) intensity, that information is not visible. Some oscilloscopes (like the ones from R&S) even have reverse intensity grading which highlights the rarely occuring waveforms.

[joeqsmith]

Color grading would be nice.  Of course, a lot of features, like changing the horizontal axis from time to distance would also be nice to have.  The attached screen shot was taken from the following PicoTech 9400 sampling scope training class:

https://eltesta.com/wp-content/uploads/2010/05/2010.07.01-Sampling-Oscilloscope-Training.-Munchen-Olympic-Tower.pdf

Note the several display styles they offer, Dots, Vectors, Variable Persistence.....   I don't think anyone would suggest the software is polished.  However,  I thought based on the following that they felt it was good enough and to drive the stake into the ground and move ahead.  Which is what I did.   

You should review the product as it exists now.

For now, I will hold off with the review until they decide what they want to do.

***
Using the power user mode to zoom in as much as possible and using the Light Mode and disabling vector trace mode for all channels.  This is about the best contrast I can get.