Anyone else use Jupyter Lab/Notebooks?

[mendip_discovery]

I am gradually getting myself set up to do some automated tasks using GPIB to talk to some of my test gear thanks to the RaspberryPi Setup. One of the things being used with this is Jupyter Labs. I am a bit of a noob with Python, but I am playing with it and Jupyter seems to be a good setup for doing some tests, talking about how that test is done and then showing the results.

So far I have only got part way into explaining some formulas/calculations I already do such as using a sine bar to generate angles. I use Excel for a lot of this but I would like to use something that allows me to share the efforts I have done to work out mathematical problems.

I plan to get it to work with a 34401 and 2 x Transmilles to do mostly automated crosschecks.

I noticed that Sandia PSL's Suncal allows you to build measurement uncertainties into a Jupyter Notebook which is kind of fun. There is also an Allen Time Deviation tool for when I get myself into the timenut stuff.

But I am wondering what others have used Jupyter for?

[miro123]

I'm also moved from using Python to Jupiter notebook voor most of my calculations /engineering notes.
For automated measurements I still use my old and trusty python.

The major problem that I have with python & jupiter notebook is portability and backward comparability. Those problem rises as script size increase
e.g.
 - upgrade from pythin v2.7 to 3.x requires manaitince in almost all my scripts.
 - the problem was lower when I moved from 3.3 to 3.6 and even lower from 3.6 to 3.7
 - Preinstall packages - e.g numpy, panda scipy - The best solution for me is to make  pip install scripts and virtual python environment for each of them
 

[alm]

Like miro123, I use notebooks for analysis but scripts for acquisition, since for me acquisitions are often long running and repetitive (like do a series of measurement for a number of parts), so I run them headless, and have them write data to CSV. But for more ad-hoc measurements I could easily see me using something like python-ivi from a notebook. But I still prefer writing all raw data to files for later analysis.

I use notebooks for analysis, but copy-pasting code if you want to do the same analysis multiple times gets old really quickly, so for anything non-trivial that I use more then once, like aggregating data in a particular way, I'll create functions. Initially I define the function in a cell near the top of the notebook, and as soon as I start having multiple notebooks using it, I put it in a Python module that I can import in a notebook. Putting it in a module also makes things more testable and easier to maintain than in notebooks.

For analysis I represent all data in Pandas DataFrames, which allow easy aggregation and basic statistics. Then I use SciPy for more advanced statistics and curve fitting, and Matplotlib for plotting. This is all pretty well integrated, but occasionally a line of numpy code is necessary because something expects a Numpy array of particular dimensions instead of a dataframe. I would say the learning curve is pretty steep.

The notebook I use to analyze data from 'foo' devices foo1 and foo2 might then end up reading something like:

Code: [Select]

foo1_data = pd.read_csv(...)
foo2_data = pd.read_csv(...)
foo1_data = add_temperature_humidity_pressure(foo1_data)
foo2_data = add_temperature_humidity_pressure(foo1_data)

show_counts(foo1_data)
foo1_fit_results = plot_and_fit_foo_data(foo1_data)
plot_fit_results(foo1_fit_results)

show_counts(foo2_data)
foo2_fit_results = plot_and_fit_foo_data(foo2_data)
plot_fit_results(foo2_fit_results)


[tszaboo]

Yes, I use it for a bunch of stuff. Though I run it locally, or in google colab.
Its good for some documentation, everyone reading it will understand the code enough to see what it does.

[zrq]

I frequently use jupyter notebooks daily, for serious work and also for home lab. Mostly quick and dirty data analysis.
I'm actually not that satisfied with it for two reasons. First is, in the notebooks, one can run the code not in order, this may seems to be convenient for quick experimenting, but it will be much more difficult to understand if one want to revisit the code after a few days. Also, jupyter is based on IPython, which does not work with multiprocessing. It is very frustrating when one have some embarrassingly parallel code, like curve fitting of thousands of peaks, and want to use all CPU cores.

[alm]

The Python code may be single threaded, but the heavy lifting will normally be done by C or Fortran code wrapped by Numpy and Scipy, and these can be made to run multi threaded.

But I doubt most test equipment will produce the volumes of data that require this.

[tszaboo]

I frequently use jupyter notebooks daily, for serious work and also for home lab. Mostly quick and dirty data analysis.
I'm actually not that satisfied with it for two reasons. First is, in the notebooks, one can run the code not in order, this may seems to be convenient for quick experimenting, but it will be much more difficult to understand if one want to revisit the code after a few days. Also, jupyter is based on IPython, which does not work with multiprocessing. It is very frustrating when one have some embarrassingly parallel code, like curve fitting of thousands of peaks, and want to use all CPU cores.

Here is a tutorial on how to run it on AWS.
When someone tells me that C is better because its faaster, just run your code on a 1000 server AWS instance, changing like 2 lines of python code.

[RoGeorge]

But I am wondering what others have used Jupyter for?

Yes, can be useful, and the Notebooks stile can be used for many other languages than Python.  So far I've used Jupyter with Python only.

Some example controlling an LXI power supply to trace the I_DS vs. V_DS of a MOSFET

https://www.eevblog.com/forum/testgear/rigol-dp832-power-supply-as-automated-curve-tracer/



or to solve puzzles 

IDK, is the PIN = 3112?
The denominator seems to be just bait.   

Other times I use Python without Jupyter Lab, it depends.

I found Jupyter easier when it's something to tinker, experiment, or learn.  When it comes to long term tools and programs, it is better to use the language itself without Jupyter, because Notebooks might not run a few months later after many updates, and Notebooks are too easy to alter by mistake.

Once something is working in a Notebook, move it to a standalone program if you want that to still work a few months later.

[alm]

Here is a tutorial

Where?

[tszaboo]

Here is a tutorial

Where?

You know, on that link that I forgot to paste.
https://docs.ray.io/en/latest/ray-core/using-ray-with-jupyter.html

[zrq]

I frequently use jupyter notebooks daily, for serious work and also for home lab. Mostly quick and dirty data analysis.
I'm actually not that satisfied with it for two reasons. First is, in the notebooks, one can run the code not in order, this may seems to be convenient for quick experimenting, but it will be much more difficult to understand if one want to revisit the code after a few days. Also, jupyter is based on IPython, which does not work with multiprocessing. It is very frustrating when one have some embarrassingly parallel code, like curve fitting of thousands of peaks, and want to use all CPU cores.

Here is a tutorial on how to run it on AWS.
When someone tells me that C is better because its faaster, just run your code on a 1000 server AWS instance, changing like 2 lines of python code.

That's quite a detour as the same can be done in matlab just by changing for to parfor, and then I can utilize all my 10 cores.
Python is OK for quick and dirty data analysis (and of course many other things).Its grammar is more friendly than matlab, but (especially with IPython) it gave me much more frustration moments than matlab because bugs and quirks.

[RoGeorge]

Forgot to say, VSCode is one of the best IDE for either Python or Jupyter, free and open source.
An even better choice might be Codium, which is a VScode without the Microsoft tracking and annoyances.

[mendip_discovery]

I'm not overly worried about performance, just getting my head around the tools. Though it did remind me of a YT video from Matt Parker,



But getting back to the subject. Getting the feeling it is better to collect data using python directly, and then do the analysis with Jupyter. I have noticed there has been a lot of work put into Jupyter over the past few years and that is often a good cause for things to fail. But that is a common feature for most open-source stuff I have used over the years.

[RoGeorge]

At first I was confused with Jupyter, but that is nothing but an IDE (Integrated Development Environment), except that instead of keeping many editor instances opened each with its own piece of code, in Jupyter you have many cells on the same page, each cell with its own piece of code in it.

Less windows opened on the screen.  That's the main gimmick.  A new code cell instead of another Python IDE window, and that's Jupyter.  Fancy editor that brings some comfort, and can be used with one or more languages.

Has many nice features though, I found it good especially for learning and/or trial and error approaches.  Not good for software development, not good to store or exchange code, has many traps regarding reproducibility of the results, and manual execution order while remembering the results from running other cells out of order can lead to wrong results.