Pocket-Sized 6 GHz 1 TS/s ET Scope

[joeqsmith]

The state machine looks for the unique response for each command.  It can't get out of sync, unless the scope starts sending the wrong acknowledges.   The software isn't yet self aware and can't modify its commands.  That's what I like about computers, they do exactly what you program them to, day in and out...

I do like protocols where every frame is processed the same.  Not where you have different end of frame markers and double CRLF in some cases.   Simple to me is making things consistent (and robust).   A goal of talking to the scope using a dumb terminal program seems of little value,  especially if it is making the system less robust, IMO. 

I saved the most recent responses and dumped them when this error happens.  Note the back to back calibrations prior to the D110000 fault.   It appears every time the scope receives two CAL commands, even though the state machine is waiting for them to send the OK CAL,  it seems to cause the fault.  Note that the first time I mention it, you can see the two CAL commands.   

I forced it to a single CAL command with no delays and it has not yet faulted out.   Are you only sending one CAL command or multiple?

***
It's been running for about 40 minutes now using a single CAL command and still no faults.   It is certainly a big clue into what is going on. 

[SJL-Instruments]

I do like protocols where every frame is processed the same.  Not where you have different end of frame markers and double CRLF in some cases.   Simple to me is making things consistent (and robust).   A goal of talking to the scope using a dumb terminal program seems of little value,  especially if it is making the system less robust, IMO. 

This does make sense. The intention was to have a simple single-CRLF response whenever possible. This doesn't work for binary data, and so is the one exception. The double CRLF was unintentional, since the delay underflow warning is intended to never happen in normal operation.

I saved the most recent responses and dumped them when this error happens.  Note the back to back calibrations prior to the D110000 fault.   It appears every time the scope receives two CAL commands, even though the state machine is waiting for them to send the OK CAL,  it seems to cause the fault.  Note that the first time I mention it, you can see the two CAL commands.   

I forced it to a single CAL command with no delays and it has not yet faulted out.   Are you only sending one CAL command or multiple?

***
It's been running for about 40 minutes now using a single CAL command and still no faults.   It is certainly a big clue into what is going on. 

This is very interesting, and puzzling. We still can't reproduce this behavior - issuing an arbitrary number of CAL commands, then a D command, works every time on our test units. It also doesn't explain why adding a 30 ms delay seems to avoid the problem.
That said, the way the official software is set up, it happens to issue at least one other command in between every two CAL commands.

***

We're running stress-tests where we send a random sequence of CAL and D commands (with random parameters) with no extra delays. So far, everything is working as intended..

***

After carefully checking the relevant code path in the firmware, the only way a "Delay underflow -inf" message can occur is if the parameter passed to the D command is below 20000 (2 ns). This is regardless of what the CAL command does, even if the calibration result is completely wrong.
Of course, we're sure this isn't what you intended to do - the problem could lie on either the scope side or the LabView side, but it's definitely a serial issue.

If you'd like, we can send you a firmware patch which returns more information when this warning is issued. It could help clarify what is happening.

[joeqsmith]

Serial communications have been in use long before I started my career.  I wouldn't have thought there was much of a point in reinventing the wheel.  I don't see using your company name as an end of frame marker as being professional.   Again, just MO.  I don't like having to deal with these special fringe cases that fall outside of what is considered normal.   Its not a productive conversation to be having as it sounds like there is no chance in changing it.  I will drop it and work around what you have. 

My setup has been running for about two hours without a warning after removing back-to-back calibrations.   It appears to solve what ever the problem is.  So does adding a small delay after the second CAL.   It's odd you are unable to replicate it.   My code performs error checking and re-transmission already and I will also change it to make sure your scope never sees back to back calibration commands.  Basically my plan forward is to bury the problem as I am not sure what else I can offer you to help in this matter.   Just be aware something odd is happening with calibrations and setting the delay, and not understanding it may come back to haunt you in the future.    

This scope has been in pretty much non-stop use since I had reworked the board and while there could still be something abnormal with this unit, I have not seen any other strange behaviors outside of what I have mentioned here.  It now appears to be very solid.   But maybe there is something else going on with part variances (FPGA) that effects the design.   Time will tell as you build more units. 

[SJL-Instruments]

Serial communications have been in use long before I started my career.  I wouldn't have thought there was much of a point in reinventing the wheel.  I don't see using your company name as an end of frame marker as being professional.   Again, just MO.  I don't like having to deal with these special fringe cases that fall outside of what is considered normal.   Its not a productive conversation to be having as it sounds like there is no chance in changing it.  I will drop it and work around what you have. 

Given the interface we have already defined, there is still enough leeway to implement an SCPI interface without breaking backwards compatibility. This will not come out before your review is published. You should review the product as it exists now. But this is just to say that we understand your point, and want to do better.
We will later release an SCPI-compatible firmware update, and any future product we launch will have an SCPI interface by default.

This scope has been in pretty much non-stop use since I had reworked the board and while there could still be something abnormal with this unit, I have not seen any other strange behaviors outside of what I have mentioned here.  It now appears to be very solid.   But maybe there is something else going on with part variances (FPGA) that effects the design.   Time will tell as you build more units. 

Thanks again for the careful testing you've done. We haven't received any other reports of hardware issues outside of the the ones you've mentioned. For now, we are confident that the "growing pains" have been resolved, and the hardware is stable.

My setup has been running for about two hours without a warning after removing back-to-back calibrations.   It appears to solve what ever the problem is.  So does adding a small delay after the second CAL.   It's odd you are unable to replicate it.   My code performs error checking and re-transmission already and I will also change it to make sure your scope never sees back to back calibration commands.  Basically my plan forward is to bury the problem as I am not sure what else I can offer you to help in this matter.   Just be aware something odd is happening with calibrations and setting the delay, and not understanding it may come back to haunt you in the future.    

As a precaution, we have added a note about repeated CAL commands in the next manual revision. The official software happens to be set up in a way that never sends back-to-back calibrations, and so avoids the issue. Thanks for bringing it to our attention.

[joeqsmith]

I let it continue to run without the back-to-back CAL and saw no problems.   I have gone ahead and made these changes to my software.   To be clear,  I have no plans to write an interface for your scope, but at least now I have what I consider a good platform to build on if I need it.   
 

This will not come out before your review is published. You should review the product as it exists now. But this is just to say that we understand your point, and want to do better.

For the review, I had planned to only show your software.  So these details about your choice for the serial interface are not relative.  I have thought about doing a segment on my LabView source code.  With it being pretty much a dead end tool now with their extortion tactics and subscription model  there may not be any value in including it.  Someone wrote me the other day who had attempting to reinstall their older version of LabView and claimed they could not get their license to authenticate.  That's twice now I have heard that.  One was from a university.   

Some parts of my review that I had already recorded will need to be updated.   For example I had done a tear down at the time I reprogrammed the unit.  Because I had not removed the PCB from the case, you only got to see the top side.   When I reworked the board, I did not record anything.  The focus wasn't about making a video but rather attempting to solve the random problems I was seeing.  Working with these tiny parts gets more difficult as I age and the risk of damaging the unit because of not being focused on the task at hand was too great.   Now, with the new case coming, I will redo this section of the review.  Pretty sure I can swap out the case on video and not damage the scope.     Other parts of the video where I was limited by your software may also change.   

Honestly, had I just reviewed the scope the way it stood when it arrived, it wouldn't have presented itself well.  Over the last month, things have really started to shape up.  I would like that power user mode but can certainly proceed without it.   

Making reviews isn't really something I am good at to start with.  Compound that with the moving target you have provided, it adds an extra layer of difficulty.  If it were a simple unbox, look at the parts supplied, five stars... Maybe include some music and talk about what an expert I am, then sure, I can have it done today.   I suggest you be patient.     

Thanks again for the careful testing you've done. We haven't received any other reports of hardware issues outside of the the ones you've mentioned. For now, we are confident that the "growing pains" have been resolved, and the hardware is stable.

I too am confident in the the hardware design.  Your build quality is still a question as I have not seen what your production process looks like.  The firmware/software seems a bit lacking but as you have said, you plan to continue to develop it.  Potential customers just need to be aware that any limitations I may show during the review could possibly be addressed in the the future.  Of course, no guarantees. 
 

As a precaution, we have added a note about repeated CAL commands in the next manual revision. The official software happens to be set up in a way that never sends back-to-back calibrations, and so avoids the issue. Thanks for bringing it to our attention.

If you can't replicate it, and nothing in your design appears to match up with the symptoms I have described, I really don't know what else you can do at this point. 

***
grammar 

[joeqsmith]

The main menus size should be saved with the default settings.  It is nice that it saves the power-user mode.  Problem is the default screen is too small and some of your selectors overlap. 

The min/max and current all being one color make it difficult to determine what going on.   If for example, you have a digital signal.   It is high, then it is low.  Now I have a line at the high state and one at the low.  Where is that signal....  Better to allow multiple colors.     I do like how the min/max works now compared with the previous version.   I don't like that when I move the graph horizontally, the min/max data still doesn't clear properly and requires a manual reset. 

When FFT is selected, the main graph flickers badly.   Not a functional problem but just doesn't look professional.

When selecting finer horizontal resolutions, the main graph flickers badly (even without running an FFT).     

I like the new scale settings.   Both horizontal and vertical.   So much faster now setting it.   

Big thanks for including all the features we talked about for the power-user.   

Overall, this version is a big improvement IMO.  So much easier to drive.   

[SJL-Instruments]

The main menus size should be saved with the default settings.  It is nice that it saves the power-user mode.  Problem is the default screen is too small and some of your selectors overlap. 

The min/max and current all being one color make it difficult to determine what going on.   If for example, you have a digital signal.   It is high, then it is low.  Now I have a line at the high state and one at the low.  Where is that signal....  Better to allow multiple colors.     I do like how the min/max works now compared with the previous version.   I don't like that when I move the graph horizontally, the min/max data still doesn't clear properly and requires a manual reset. 

When FFT is selected, the main graph flickers badly.   Not a functional problem but just doesn't look professional.

When selecting finer horizontal resolutions, the main graph flickers badly (even without running an FFT).     

Thanks for the feedback. We have just released v2.5.8 on our website, including fixes for all of these issues.

The flickering issue is hardware-dependent, and we were only able to reproduce it on one machine. There is a chance that our patch may not fix it on your machine. Let us know if the problem is resolved for you.

[joeqsmith]

Default settings and the overlap of menu items appear fixed. I like the new colors for the min/max.  Min/Max now seems to reset correctly. 

Flicker is still very poor.   Even when using the internal trigger with no signals applied, it flickers.  Let me put it this way, it isn't something I would have ever released.  It's that bad.  It reminds me of that original NanoVNA software they had when I first looked at it.   

NVIDIA Quadro M4000,  Xeon E5-1650 v4,  32G RAM, Win 10 Pro 1903. 

[SJL-Instruments]

Default settings and the overlap of menu items appear fixed. I like the new colors for the min/max.  Min/Max now seems to reset correctly. 

Flicker is still very poor.   Even when using the internal trigger with no signals applied, it flickers.  Let me put it this way, it isn't something I would have ever released.  It's that bad.  It reminds me of that original NanoVNA software they had when I first looked at it.   

NVIDIA Quadro M4000,  Xeon E5-1650 v4,  32G RAM, Win 10 Pro 1903.

Which portion of the screen flickers? The entire graph, or only the traces? Do the axis labels flicker as well? What about the intensity grading in CDF mode?
We can't currently reproduce this on our machines, so this will help us to narrow down the problem.
***
We can cause the whole graph to flicker if we modify the timings of some graphics calls. There is a narrow window which can cause a race condition. We'll send you a v2.5.9 prerelease version tomorrow morning which will hopefully fix the issue.

[joeqsmith]

I would call it the entire graph area, including the axis labels, cursors and traces.  Everything in the darker colored area flickers.   Without the FFT running, if I hold the mouse button over the white bar at the top, it will increase the flicker frequency.

You should review the product as it exists now.

Careful what you ask for.  Imagine making a review and this is what potential customer see:

[SJL-Instruments]

Thanks for the additional information and screen capture. We've just sent you a prerelease of v2.5.9, which we think fixes the root cause of the flickering.

[joeqsmith]

The flickering issue is hardware-dependent, and we were only able to reproduce it on one machine.

I also ran it on a newer PC with the following:

NVIDIA RTX A4500, CUDA cores 7168, Compute Compatibility 8.6
Intel(R) Core(TM) i9-10980XE CPU, 3.00GHz   
64.0 GB
Windows 10 Pro, 22H2, 19045.3930

Same flicker problem.  I obviously can't explain why you are not able to replicate it or why you felt the need to release it when it was clear there was a major problem with the prerelease.   IMO, your goal should be to release high quality software.  Seeing a problem like this flicker in the wild just isn't something that should ever happen.   

Running the prerelease 2.5.9 on the above hardware does not appear to exhibit the flicker.   I will test it tonight on the old PC and get back with you. 

[joeqsmith]

I have tried the 2.5.9 prerelease on my old PC and it no longer exhibits the flicker.   

The updated case arrived today as well. 

[joeqsmith]

2.5.9 prerelease, error when saving settings.   Not repeatable. 

***
Strike that.  It repeats.   

Also,  I suspect there is something still different between your software and mine when it comes to the power user mode.   I can set your software to what I believe are the same settings as my software and will get different results.   Does your software for example no allow trigger values in the mV range?  Maybe the boundary settings are limited?   For what ever reason, with my software I am able to work with much smaller signals than with yours.  Something in your software appears to be limiting me.   Really strange.

[SJL-Instruments]

2.5.9 prerelease, error when saving settings.   Not repeatable. 
***
Strike that.  It repeats.   

We're looking into this, and will send you an updated prerelease version when we find the root cause (hopefully today).
We can reproduce the error if the scope is disconnected prior to saving settings. Is this what happened in your case? (To be clear, we consider this a bug, and will fix it.)

Also,  I suspect there is something still different between your software and mine when it comes to the power user mode.   I can set your software to what I believe are the same settings as my software and will get different results.   Does your software for example no allow trigger values in the mV range?  Maybe the boundary settings are limited?   For what ever reason, with my software I am able to work with much smaller signals than with yours.  Something in your software appears to be limiting me.   Really strange.

There are two possible reasons:
1. On startup, the software performs the DC offset trim automatically (Section 3.7.2) if nothing is connected. This will shift all voltages on the ~mV scale. This isn't relevant if signals are connected on startup.
2. For backwards compatibility reasons, the trigger level command sent to the scope is 22 DAC units (~1 mV) higher than entered in the software.
Otherwise, there are no limits set in power-user mode. Any boundary limits are explicitly shown (the displayed value will be clamped).

Same flicker problem.  I obviously can't explain why you are not able to replicate it or why you felt the need to release it when it was clear there was a major problem with the prerelease.   IMO, your goal should be to release high quality software.  Seeing a problem like this flicker in the wild just isn't something that should ever happen.

Going forward, we will cut back our release schedule to once per two weeks. This will give us more time for QA, and still gives a reasonable turnaround time for user requests. In hindsight, weekly releases were on the edge of what we could handle.

[joeqsmith]

2.5.9 prerelease, error when saving settings.   Not repeatable. 
***
Strike that.  It repeats.   

We're looking into this, and will send you an updated prerelease version when we find the root cause (hopefully today).
We can reproduce the error if the scope is disconnected prior to saving settings. Is this what happened in your case? (To be clear, we consider this a bug, and will fix it.)

It's possible but I am not certain.  I really like having this feature and appreciate you adding it.   

Also,  I suspect there is something still different between your software and mine when it comes to the power user mode.   I can set your software to what I believe are the same settings as my software and will get different results.   Does your software for example no allow trigger values in the mV range?  Maybe the boundary settings are limited?   For what ever reason, with my software I am able to work with much smaller signals than with yours.  Something in your software appears to be limiting me.   Really strange.

There are two possible reasons:
1. On startup, the software performs the DC offset trim automatically (Section 3.7.2) if nothing is connected. This will shift all voltages on the ~mV scale. This isn't relevant if signals are connected on startup.
2. For backwards compatibility reasons, the trigger level command sent to the scope is 22 DAC units (~1 mV) higher than entered in the software.
Otherwise, there are no limits set in power-user mode. Any boundary limits are explicitly shown (the displayed value will be clamped).

This is where I would like to see an auto setup.  I am working with sub 10mV signals and can believe it is a DC offset problem and the trigger will need to change.  I adjusted it a fair amount trying to locate a level but may not have gone far enough.  I'll try again later.   Think about it, one button, auto set.  Just getting the vertical and trigger set would be very helpful compared with my trial an error technique.  The computer would be much faster. 

Same flicker problem.  I obviously can't explain why you are not able to replicate it or why you felt the need to release it when it was clear there was a major problem with the prerelease.   IMO, your goal should be to release high quality software.  Seeing a problem like this flicker in the wild just isn't something that should ever happen.

Going forward, we will cut back our release schedule to once per two weeks. This will give us more time for QA, and still gives a reasonable turnaround time for user requests. In hindsight, weekly releases were on the edge of what we could handle.

I want to be clear about this.  Feel free to send me prereleases while I continue to evaluate the product.  I don't mind testing them along with running my other tests.  I think giving yourself more time to test your software is a big step in the right direction.   

[joeqsmith]

If I do disconnect the scope from the PC, how do I get your software to reconnect to it without having to exit and restart? 

[SJL-Instruments]

If I do disconnect the scope from the PC, how do I get your software to reconnect to it without having to exit and restart? 

Currently the software does not support this. We will add this to the next version.

This is where I would like to see an auto setup.  I am working with sub 10mV signals and can believe it is a DC offset problem and the trigger will need to change.  I adjusted it a fair amount trying to locate a level but may not have gone far enough.  I'll try again later.   Think about it, one button, auto set.  Just getting the vertical and trigger set would be very helpful compared with my trial an error technique.  The computer would be much faster. 

This is being worked on - we aren't happy enough with the reliability yet, but with any luck it should be ready for next release.

We're looking into this, and will send you an updated prerelease version when we find the root cause (hopefully today).
We can reproduce the error if the scope is disconnected prior to saving settings. Is this what happened in your case? (To be clear, we consider this a bug, and will fix it.)

It's possible but I am not certain.  I really like having this feature and appreciate you adding it.   

We've found the root cause of this issue, and will send you a patched version tomorrow to test.

[joeqsmith]

For the power-user, consider making the End the total collection time rather than the end time.  If I want 1.6ns of data, I have add it to the Start in my head.  Then if I change the start, I have to manually calculate and enter a new end.   Maybe others would not run it that way but that would be my preference.  If you allow both a start/stop times and start/duration, and cover both would be great.   

DC offset was the problem.  Adjusting the trigger to a different level corrected it. 

Not finding any other major problems with the software.

***
When you start the program, it starts out with the menu that allows selection of the simulation.  You have to select the scope, then select OK.  I have been trained so well by Microsoft,  could you allow the double click selection so I don't have to move the mouse a second time and select another button?   

When you start the program, it flashes a few boxes on the screen before it loads.  This is something new.  Could you clean this up?

While I really like the new vertical scale with the drop down + custom, after selecting custom the only way to get back to drop down mode is to right click, select Select All,  right mouse click, select Delete, click outside the box or select the spin icon.   That's a lot going on for what should be a one click operation. 

FFT could use some cursor measurements.  Peak readouts... 

I was watching some of Picotech's reviews of their 9400.  I can't say I am a fan of their UI but they do have some good ideas that make it easier to drive their scopes. 

***
FFT settings are not included in the Save Settings. 

***
Closing the FFT window and reopening it will cause it to reset to some defaults rather than using the last settings causing me to have to reset it up every time.   

[SJL-Instruments]

For the power-user, consider making the End the total collection time rather than the end time.  If I want 1.6ns of data, I have add it to the Start in my head.  Then if I change the start, I have to manually calculate and enter a new end.   Maybe others would not run it that way but that would be my preference.  If you allow both a start/stop times and start/duration, and cover both would be great.

Got it - we will make this change in the next version.

Not finding any other major problems with the software.

We have sent you an updated prerelease of v2.5.9. If the last issue is resolved, we will release this publicly as a bugfix update.

[joeqsmith]

It corrects the save settings problem I saw. 

When you create multiple prereleases, you may want to consider having a unique version number in the header.   As I share screen shots, I don't want you asking me what version of software I was running.  It is up to you to track your code.  Internally, I don't care, but anything that leaves your office should have a unique identifier, IMO.

About the lockup.  I was running some longer sweeps and I think what may be happening is I do something that tells the software to do something besides sweeping.  The scope continues to dump data and the software can't abort it.  This is a total guess on my part.  It had looked like it hung as I have seen.  Nothing displayed in the graph as before.  This time I just let it sit for a long while and sure enough, it recovered.   It is very possible I was running high resolution wide time ranges when I have seen it hang before. 

[SJL-Instruments]

It corrects the save settings problem I saw.

Thanks for confirming. We've released this version as v2.5.9 (primarily to resolve the flicker problem).

When you create multiple prereleases, you may want to consider having a unique version number in the header.   As I share screen shots, I don't want you asking me what version of software I was running.  It is up to you to track your code.  Internally, I don't care, but anything that leaves your office should have a unique identifier, IMO.

We will do this going forward - thanks.

About the lockup.  I was running some longer sweeps and I think what may be happening is I do something that tells the software to do something besides sweeping.

To clarify, this means the hangup happened as you were changing some settings? Do you remember which ones?
This narrows it down quite a bit - likely a inter-process communication issue. Hopefully we can resolve this by next release.

When you start the program, it starts out with the menu that allows selection of the simulation.  You have to select the scope, then select OK.  I have been trained so well by Microsoft,  could you allow the double click selection so I don't have to move the mouse a second time and select another button?   

When you start the program, it flashes a few boxes on the screen before it loads.  This is something new.  Could you clean this up?

While I really like the new vertical scale with the drop down + custom, after selecting custom the only way to get back to drop down mode is to right click, select Select All,  right mouse click, select Delete, click outside the box or select the spin icon.   That's a lot going on for what should be a one click operation. 

FFT could use some cursor measurements.  Peak readouts... 

I was watching some of Picotech's reviews of their 9400.  I can't say I am a fan of their UI but they do have some good ideas that make it easier to drive their scopes. 

***
FFT settings are not included in the Save Settings. 

***
Closing the FFT window and reopening it will cause it to reset to some defaults rather than using the last settings causing me to have to reset it up every time.   

We will work on these bugfixes/features for the next release. As we get through this list, we'll send you occasional prereleases for feedback.

[joeqsmith]

About the lockup.  I was running some longer sweeps and I think what may be happening is I do something that tells the software to do something besides sweeping.

To clarify, this means the hangup happened as you were changing some settings? Do you remember which ones?
This narrows it down quite a bit - likely a inter-process communication issue. Hopefully we can resolve this by next release.

Sadly I do not but I can believe that my sweep times were in the order of several minutes.   When I was playing around writing my own code for it, I never flush Windows buffers.  It was easy to get my software in a mode where the scope had I expect, already dumped it's data but my software would be looking for other responses.   This is what made me wonder if you don't have a similar condition.   I think I have seen it happen three times.  Rare.

[joeqsmith]

For those of you interested in this product, if there is something specific you would like to see included in my review, consider this my open invitation to put in your requests.   Even if it isn't something I can demonstrate, maybe the Signal Path channel will consider it.   

[joeqsmith]

While attempting to make some very long, high resolution sweeps, I managed to get the scope to lockup to the point where exiting the software and reloading, it does not find the unit.     My guess is a power cycle is going to recover it.   I think I have had it hang in such a way that required a power cycle twice now.