NanoVNA Custom Software

[joeqsmith]

I tried installing your couple of files.  I placed cstartup.s in the compilers/IAR directory and the hal_i2s_lld.c in both SPIV1&2.   It still errors out.

hal_i2s_lld.c needs to be replaced just in the folder ChibiOS\os\hal\ports\STM32\LLD\SPIv2\hal_i2s_lld.c

But actually, these are minor changes, and they should not affect build.

The error depends on tool chain

/roaming/xpacks/@gnu-mcu-eclipse/arm-none-eabi-gcc/8.2.1-1.7.1/.content/bin/../lib/gcc/arm-none-eabi/8.2.1/../../../../arm-none-eabi/bin/ld.exe: region `flash0' overflowed by 584 bytes
collect2.exe: error: ld returned 1 exit status
make: *** [ChibiOS/os/common/startup/ARMCMx/compilers/GCC/rules.mk:243: build/ch.elf] Error 1

[radiolistener]

I could send more than one command while waiting for the Nano to respond.   I could not do this with the data commands.    At the time, it would not crash the Nano but it appeared to just not receive the commands.

You can send several commands, but it depends on buffer size in the NanoVNA. If you flooding it with commands, it will be lost.

[radiolistener]

/roaming/xpacks/@gnu-mcu-eclipse/arm-none-eabi-gcc/8.2.1-1.7.1/.content/bin/../lib/gcc/arm-none-eabi/8.2.1/../../../../arm-none-eabi/bin/ld.exe: region `flash0' overflowed by 584 bytes
collect2.exe: error: ld returned 1 exit status
make: *** [ChibiOS/os/common/startup/ARMCMx/compilers/GCC/rules.mk:243: build/ch.elf] Error 1

it seems that the code doesn't fit into STM32F072CB. First check that your tool chain uses 128 kB flash for this chip. The second you can play with optimization options. If it doesn't help, you can remove some commands. For example scan command duplicating sweep functionality, so you can comment it with #if 0 directive. Also remove it from the command list array.

This is known issue. The controller STM32F072CB has too small flash memory - just 128 kB. And you can find that other people also fighting with this limitation for NanoVNA.

This is why NanoVNA V2 uses more powerful controller, it has more memory, so it can handle more things

[radiolistener]

I had asked you about your regression tests that you mention but you chose not to discuss for what ever reason.  I am still open to discussing and plan to continue to add to the complexity of my scripts.

I don't have automated regression tests. I do it manually. Just compare what is changed in the code and then trying to test if these changes works properly and there is no regression. That's what I mean when I told you about regression tests

[ogden]

You both are doing great job to improve nanovna, yet chat about build and source code specifics better be done using private messages. No offense, just saying.

[joeqsmith]

/roaming/xpacks/@gnu-mcu-eclipse/arm-none-eabi-gcc/8.2.1-1.7.1/.content/bin/../lib/gcc/arm-none-eabi/8.2.1/../../../../arm-none-eabi/bin/ld.exe: region `flash0' overflowed by 584 bytes
collect2.exe: error: ld returned 1 exit status
make: *** [ChibiOS/os/common/startup/ARMCMx/compilers/GCC/rules.mk:243: build/ch.elf] Error 1

it seems that the code doesn't fit into STM32F072CB. First check that your tool chain uses 128 kB flash for this chip. The second you can play with optimization options. If it doesn't help, you can remove some commands. For example scan command duplicating sweep functionality, so you can comment it with #if 0 directive. Also remove it from the command list array.

This is known issue. The controller STM32F072CB has too small flash memory - just 128 kB. And you can find that other people also fighting with this limitation for NanoVNA.

This is why NanoVNA V2 uses more powerful controller, it has more memory, so it can handle more things

Yes, it seems to be setup for 128K.     Are you building with same files you have archived?     Comparing the LD file with others, any idea why flash7 is set to 32K?    Do they really need this much space for calibrations?   

I can disable your new commands and of course it will fit.   Another option is to enable the size optimization.   Setting this will easily allow it to build what you have and fit.   I suspect the performance hit is too much and why they are not using it.   I'll try running it built this way through the regression tests.

Assuming that you are building with all the files you have uploaded, what version of the tools are using?   Hard to believe that compiler would have changed this much but maybe.  It may explain why I wasn't able to build the other code I had downloaded. 


As for stacking commands (sending more than one without waiting for a response), the way I have my software currently structured, if I am sending a command that will change the frequency settings, I will go ahead and send the frequencies command with it.   In this case, I know all the set commands (start, stop, span....) are very fast and have virtually no payload.   So far this has not caused a problem.   It's the only case though where I continue to stack them.   

On the regression test,  this makes more sense.  I've started adding an automated report generation to document the testing so I can easily compare the results later on. 

[joeqsmith]

While enabling the optimizer does allow me to build with all of the commands you have selected and fit, once loaded into the Nano, it will not run.  Some text is displayed on the screen but the Smith Chart for example does not get displayed.   

I'm interested in seeing what your tool chain is showing for usages. 

Removing your last commands with the optimizer set to the defaults allows it to fit and also seems to pass my basic regression tests.   An odd side effect is that I no longer see the delay when the commands are switched like I show in the earlier plots. 

One of the test cases I added was to set the start frequency to 1500 and the stop to 0.   My old analog VNA has no way to scan backwards.  Looking at my old HP3589A, it behaves similar to the Nano which basically reports it will scan from 10KHz to 10KHz with 101 samples.     With my Signal Hound, when trying to set the stop lower than the stop, it will coerce the it to the maximum value.   Any thought on if you will change how this works?   

[radiolistener]

An odd side effect is that I no longer see the delay when the commands are switched like I show in the earlier plots. 

That's interesting. Is it better or worse compared to this hex file?

One of the test cases I added was to set the start frequency to 1500 and the stop to 0.   My old analog VNA has no way to scan backwards.  Looking at my old HP3589A, it behaves similar to the Nano which basically reports it will scan from 10KHz to 10KHz with 101 samples.     With my Signal Hound, when trying to set the stop lower than the stop, it will coerce the it to the maximum value.   Any thought on if you will change how this works?

Do you mean something like sweep from start=100 MHz to stop=50 MHz? It is possible, but I'm not sure if it really needed.

[ogden]

What is actual bottleneck why PC (NanoVNA Saver) scan rate is much slower than on-board display? Nearly two times or so, even when PC software is running, thus primary and on-board display scans is not that important anymore? There even could be option to disable onboard display while USB software running (PC software command, screen_off/on). (NanoVNA-Q-2019-10-15-e427dbb, NanoVNA Saver 0.1.1) BTW how PC USB scan data is made? It is separate scan which is not displayed on nanovna display? If so - is it possible to just readout buffer of completed scan?

[radiolistener]

how PC USB scan data is made? It is separate scan which is not displayed on nanovna display? If so - is it possible to just readout buffer of completed scan?

The device performs sweep and render in a loop. The PC can request the current data of the sweep through USB and NanoVNA will send the last sweep data to PC.

[ogden]

how PC USB scan data is made? It is separate scan which is not displayed on nanovna display? If so - is it possible to just readout buffer of completed scan?

The device performs sweep and render in a loop. The PC can request the current data of the sweep through USB and NanoVNA will send the last sweep data to PC.

That means that scan is not bottleneck. Excellent. USB have enough bandwidth, MCU have bandwidth as well - because it does update it's screen fast during PC software scan. It means PC SW update speed *can* be improved. Somebody just have to look at it because it is important while using average, especially knowing that it can be *twice* as fast.

Do you release hex and/or dfu releases as well? Please.

[radiolistener]

That means that scan is not bottleneck. Excellent. USB have enough bandwidth

that's not the case, it uses old USB peripherals which can work in USB FS mode and the speed is limited with small amount of buffer memory and MCU speed. But it is good enough for communication. I didn't measured the maximum performance, but think it is about 50-100 kB/s.

MCU have bandwidth as well - because it does update it's screen fast during PC software scan.
It means PC SW update speed *can* be improved. Somebody just have to look at it because it is important while using average, especially knowing that it can be *twice* as fast.

Do you release hex and/or dfu releases as well? Please.

MCU cannot communicate with PC during sweep, because it will affect measurement.

Communication speed really can be improved, but I'm not sure that this is really needed.

There are several ways to improve performance:
1) optimize communication from PC side (don't send extra requests, don't do long delays, etc)
2) use binary format communications, but it will break compatibility with existing software (NanoVNASharp, Saver, etc)

As for average, it is better to perform it on MCU side. For example you can use scanraw command from NanoVNA-Q firmware. It allows to setup average from 1x to 1000x and it will be performed on MCU side. But it returns raw gamma data with no correction (with no calibration apply, the same as you see with reset calibration), so you're needs to apply your own calibration.

[ogden]

That means that scan is not bottleneck. Excellent. USB have enough bandwidth

that's not the case, it uses old USB peripherals which can work in USB FS mode

Yes, I did mean exactly that - USB FS of stm32 which is running at 72MHz(?) have enough bandwidth for this application.

MCU cannot communicate with PC during sweep, because it will affect measurement.

Well, it does not need to show scan on it's display two times faster than PC is getting scan data as well. While connected to PC, display can be off. That would also fall into "will not affect measurement", if not then "waste less time" definitely.

For example you can use scanraw command from NanoVNA-Q firmware.

No. I can do many things but what i Want to do is - *use* nanovna, not modify or develop it. What I see - PC software is slow, slower than onboard firmware and I know that hardware is more than capable.

[5q5r]

Hello Ogden,
there were some changes to the official firmware which necessitated a slowdown be put in to all data reads. As of firmware 0.2.0 and newer, and NanoVNA-Saver 0.1.2 and newer (but use 0.1.3, it's just out now), the new "scan" command is used. This disables the screen, runs a single sweep of the requested range, waits for it to complete, and then the application fetches the data.

I still think it feels slow, also compared to early experiments. But it's how I'm told by the maker of the firmware that I should request data via serial, and thus it's what I make the application do. I would be very happy if someone were to try to optimize the code on the NanoVNA ;-)

[ogden]

Thanx, tried 0.1.3 and do not see any improvements in areas that IMHO are actually important. Same weird behavior - nanovna makes two scans but on PC only one is displayed. That does not makes sense because cooms interface is CDC! As soon as nano is finished scan, it can PUSH out data. Application shall jus sit and wait for next scan data to come in. Firmware and software guys shall start to talk each other. Also seems like averaging did not work in 0.1.1 and still does not in 0.1.3. Noise peak-peak amplitude does not go down . Anybody else think so?

[5q5r]

Hello Ogden,
I can confirm that averaging certainly does work. Whether it gives you the results you expect, or whether it works with your particular firmware I can't promise, of course :-) What firmware revision are you running?

On the "shall start to talk to each other": I certainly have had conversations with edy555 about the device and interaction with NanoVNA-Saver, and he has specified using this particularly interface, ie. using the "scan" command in newer firmware. What speed it goes at isn't something I can affect :-)

[ogden]

I can confirm that averaging certainly does work.

Firmware mentioned in this post:
https://www.eevblog.com/forum/rf-microwave/nanovna-custom-software/msg2739418/#msg2739418

Tested using s11 noise floor which is most "noisy". Look at the picture. Those dips can't be there after 20x average. No way. Not to mention that picture does not look any different than single sweep w/o average. If you are further interested - let's go PM. I can give input as long as it does not take big chunk out of my spare time.

[5q5r]

Whether those dips can be there or not depends on your calibration, I would say. If there were opposite peaks at the time of calibrating the NanoVNA itself, it would over-compensate on those particular frequencies, and consistently produce such dips.

Attached are two sweeps of 900-1500MHz (lots of noise!) - one without and one with 20x averaging. I would say it looks like it's working.




The levels you're looking at when looking at the noise floor are quite small. Even a tiny variation in the magnitude of S11 causes a large jump in measured return loss.

[ogden]

Even a tiny variation in the magnitude of S11 causes a large jump in measured return loss.

Agreed. Large jump will trash all the average and spike will stay. Yet I marked *dips*. Whatever. If you say it works - fine.

If there were opposite peaks at the time of calibrating the NanoVNA itself, it would over-compensate on those particular frequencies, and consistently produce such dips.

Oh, ok. It means that I did find problem with firmware/calibration, not PC software

[5q5r]

Possibly - though to confirm, you should probably do multiple sweeps with high averaging. If the dips (or peaks) stay in place, it's a "true" error, if they move around, maybe the averaging isn't good enough to catch it, or you have indeed found a bug that I just haven't recognized ;-)

I generally run 10 segments and 25x averaging (using truncated mean, so 25 averages dropping 5 readings per average) when doing calibration. It takes a few minutes per sweep at that point, but it does give a very nice and stable in-application calibration which I can save afterwards. Then again, if I change any cables or connectors, I have to start over ... 

[ogden]

Possibly - though to confirm, you should probably do multiple sweeps with high averaging. If the dips (or peaks) stay in place, it's a "true" error, if they move around, maybe the averaging isn't good enough to catch it, or you have indeed found a bug that I just haven't recognized ;-)

Everything is fine, there is no bugs. Phew. Three 20x averages of 20dB reflection, they all look basically the same. Conclusion: to get meaningful averaging results, same level of averaging for calibration data is necessary. Now averaging just reveals calibration noise.

[joeqsmith]

....

I didn't measured the maximum performance, but think it is about 50-100 kB/s.

....
Communication speed really can be improved, but I'm not sure that this is really needed.

There are several ways to improve performance:
1) optimize communication from PC side (don't send extra requests, don't do long delays, etc)
2) use binary format communications, but it will break compatibility with existing software (NanoVNASharp, Saver, etc)

Oddly, I'm seeing no where close to 50-100 kB/s (400kbs-800kbs) but then again, it could be something I am doing wrong.    I don't think this is the problem with why it's so slow.   Without the Nano connected to the PC and just watching it, it will scan about once a second.   This is using the last image you uploaded.     

On the PC I am seeing 325467 Bytes in 14.04 seconds or 185kbs.  This is plenty fast to keep up with how slow the Nano scans.   If I really push the data rates, basically asking for the next data set right after is finished sending a data set, without giving it some delay, the performance suffers.   

Attached, I am showing the data rate portion of my regression test.   I send out 50 frequencies commands which are very fast to respond as the payload is so small.  This is followed by 50 data  0 and 50 data 1 packets.      I will wait up to 2 seconds for the Nano to respond with a valid packet before timing out.   I had tried to disable the LCD at one time but it had no effect on the speed.  Maybe this has changed. 

I've posted this a few times and mentioned it during my video but as long as we are on the topic of speed,  unlike the software that was supplied with the Nano, I only request the frequencies when I have changed them.   I also only collect data 0 for S11.   Their software will collect data 0, data 1 and the frequencies for each sweep.   I suspect they need a slightly longer delay  for their Auto (1.2 seconds).   It's a very small difference and IMO, both are very slow.   

But hey, it's $50.  I am just happy to see how stable this image you uploaded is!   I have been running a fair number of tests on it over the last few days and not once has it white screened, locked up, or sent bad data.   

I have not added any support for your averaged scan or the normal scan.  I did try sending the commands just to get a feel how they would work.

[ogden]

If I really push the data rates, basically asking for the next data set right after is finished sending a data set, without giving it some delay, the performance suffers.

Problem even have a name: multitasking race condition. PC software have to request data but time window while it is available, is very small. As we see, PC manages to miss each other scan data set. Needed change is quite "simple" - leave existing "polling" approach alone and introduce new command for continuous scan and "push" for nano. In short nano while said mode is initiated: 1) performs scan 2) optionally displays it on the built-in LCD 3) sends scan data to PC unconditionally and only when USB transfer fully done, continue with next scan.

But hey, it's $50.

Yes, you can say so when you are talking about tiny screen and fonts, awkward and fragile input "thingie", lack of proper case but such excuse is not OK in case of poor software *design* that clearly can be improved.

[radiolistener]

Problem even have a name: multitasking race condition. PC software have to request data but time window while it is available, is very small.

No, there is no race condition. There is just delay while NanoVNA is busy. It cannot respond immediately, because it may be busy. If PC request performed when NanoVNA is busy, it will needs to wait. NanoVNA may perform sweep or rendering. PC command cannot interrupt this operation, so it needs to wait until current operation will be completed.

NanoVNA cannot respond in parallel, because data command needs to access to measurement data, which is under update by sweep. It needs to share the same memory with measurement data, because there is not enough memory to store a copy. Measurement data for single channel requires 101*4 = 404 bytes. There is no memory even for stack margins, so you're needs just to share the same memory for different task.

Since data is modified by sweep task and modification is not complete yet, the data command needs to wait when the sweep will be completed. This is the reason for delay.

[ogden]

Problem even have a name: multitasking race condition. PC software have to request data but time window while it is available, is very small.

No, there is no race condition. There is just delay while NanoVNA is busy. It cannot respond immediately, because it may be busy. If PC request performed when NanoVNA is busy, it will needs to wait.

You do not make any sense. Why PC does get only each 2nd scan then? I am afraid that you need to update your knowledge about "race condition". Meaning is quite broad BTW.

Since data is modified by sweep task and modification is not complete yet, the data command needs to wait when the sweep will be completed. This is the reason for delay.

Yes. It means that PC have small time window. I said it already.  Yet you somehow manage to reply "no you are wrong, let me say the same just using another words"  

[edit] Why don't you comment my suggestion of PC communications redesign? Two PC software guys are reading, they can potentially change software side accordingly, seems you are more than capable to make such seemingly small firmware change.