Software to interactively plot a large number of X-Y points (>100k)

[jeremy]

Hi all,

I've got some logged data (temperature, humidity, resistance, etc) which I'd like to be able to view and zoom in/out of in an interactive manner. Number of points is somewhere between 100k and 500k. Does anyone know of any software that lets you view this sort of data (with appropriate downsampling), and also lets you zoom in on any interesting bits? I really like the PicoLog software, but it only works with their proprietary sensors, not random data from disk.

I can reprocess the data into CSV or some other format if necessary.

[RoGeorge]

Would SPICE (e.g. LTSpice, QUCS) charting capabilities be enough?
If yes, you can define a CSV voltage source, then chart it.

Later edit:
This one was doing nice charts, too, but didn't tried the latest versions.
https://hackaday.io/project/5334-serialplot-realtime-plotting-software

[eliocor]

DPLOT: https://www.dplot.com/index.htm 

ultrafast even with LOTS of data!

[Selectech]

KST.

https://kst-plot.kde.org/

I use it for large data sets from sensors collected by embedded MCU applications.

[ramon]

RRDtool + Javascript RRD   

Go here   ->    http://javascriptrrd.sourceforge.net/

Do not just look the pictures ... click on the images.
Click + move the mouse to do zoom into the graphs.  (I guess that this is maybe what you want).

This tools have been used for years for network traffic and server/cpu stats (usually), but actually they can 'eat' any other kind of data series too. They require a big learning curve.

[Karel]

EDFbrowser can import CSV/ASCII.

https://www.teuniz.net/edfbrowser/

[HighVoltage]

For statistical analysis of large data sets I use MINITAB
http://www.minitab.com/en-us/

[MarkL]

I do all my plotting and statistical crunching in Octave.  It's the open equivalent of Matlab:

  http://octave.org

A quick test of a scatter plot of 500k random (x,y) points takes about 3 seconds on a 2.5GHz processor.  You can then pan and zoom in/out.  I don't know if that's fast enough for you compared to some of the other packages mentioned, but I don't usually have that many data points to worry about.

It has CSV and other import formats.

[rhb]

I use gnuplot for such things.  No connection to  Stallman's Gnu project.  I've done a very large fraction of my data analysis work with it.  The Marquardt-Levenberg solver in it is the best I've ever used.  It does need a fast graphics card for manipulating large cloud points.

My experience with large datasets (i.e. TB) is that color XY plots and XYZ movies are the most useful.  Points get obscured by other points if you do clouds in XYZ.  In particular probability density functions (PDFs)  are very valuable.  Most large databases are full of garbage.  Theyreally stand out in a PDF of the dataset.

There is a bunch of code I wrote for this in the Seismic Unix repository on Github that John Stockwell has set up.  I've also got a code I've never published which computes spatial statistical attributes and filtering on TB size data sets.  Sorry, I can't tell you the names of the programs in Seismic Unix.  Stockwell changed the names and stripped my copyright from the codes.  So I quit doing support after 10-15 years.

[fki82]

Can you pan and zoom with any of the mentioned tools interactively?

[rhb]

Yes,  gnuplot provides pan and zoom.   SU may just offer zoom.  I can't recall as it's been a few years since I used it last.

[Karel]

Can you pan and zoom with any of the mentioned tools interactively?

Yes, EDFbrowser can pan, zoom in/out, has filters and FFT.

[BFX]

Grafana is the best for your purpose.

https://grafana.com

[KaneTW]

I use Grafana for presenting interactive time-series data

For offline data, my usual choice of CAS (Mathematica) does about 1mn points in 3.5sec.

[pwlps]

I'm using Igor Pro for all my data analysis. The software is designed to manipulate very large data sets and is really fast. On my i5-3470 plotting 10M points takes less than 1 second.  You can pan and zoom using the mouse.  You can control all basic functions either with menus or from the command line.  What I like is that when using the menus the corresponding commands are generated and go into the history record so it is easy to recall them from there,  you can also grab them from the history window and copy to a macro if you want to automate a sequence without learning the details of the programming language. Igor Pro forum is very active and it's easy to get help there.