Datalogging with Arduino

[gildasd]

I'm trying to get my head around using an Arduino for my Bachelor's Thesis project (called Mémoire in Belgium).
I've NEVER done anything with an microcontroller, so I'm a tiny bit nervous on splurging my hard earned student cash...
I'm not looking for someone to do this for me, I'm looking for help at avoiding the NOOB pitfalls.

This summer, I've decided to start to learn basic programming for the Arduino and build a basic setup.

I need to datalog quite a few things, the method is not the objective, so I can use off the shelf stuff.
My ultimate target will be to log once every second 4 analogue sources (pressure gauges, temp, humidity) and one digital (wind speed) 24/7 for 3 months.
Open source is better, because I am meant to pay for this (can't use that old copy of Excel they gave me at work in 2003) and provide a detailed BOM.
For this test I'll just be logging 1 analogue and 1 digital source. keeping stupid simple.

I've been reading stuff on Arduino.cc and going over tutorials...
... and for the initiated, it's slightly confusing.

To run this test, this is my simplified BOM:
1) Arduino board
2) USB SD card shield.
3) My sensors.
4) Small OEM 5V power supply
5) Real time controller?
6) Open office Calc.

To be able to replace anything in a few hours, in case of a major magic smoke moment, I'll be sourcing locally (and I can't put "some dude in China on Ebay" in my BOM).
I have a shop here in Gent (home) and a couple in Antwerp (Antwerp), so I can get stuff in the time it takes me to get there Monday to Saturday.

1) For the board my choices are limited to; Leonardo, UNO rev3, NANO (328) and M0.
I quite like the Nano due to it's form factor and 8 analogue inputs (I can have redundant sensors).
But none of these have an SD card reader...
2) The only SD shield I can get easily here is  a Wireless SD shield...
I obviously won't be using the wireless: http://www.gotron.be/media/files/Downloads/A000065.pdf
Or would this Adafruid SD breakout board be compatible? http://www.gotron.be/en/bouwkits/adafruit/adashield/micro-sd-card-breakout-board.html
3) Sensors: for the test I'll just wire a photo-resistor (anaogue) and a 555 timer circuit with another photo-resistor (digital; quantity of 1 and 0's, per unit of time, go up and down during the day), just crap to generate a signal.
4) 5V, 1A, plug style mains power supply.
5) An RTC: do I need this to get the Arduino to save data at regular intervals?
6) Open Office CALC is a simple spreadsheet, I'll have to write a script to transform the raw data into usable figures.
When I fooled around with GPS data, I would do this with a TCX file. It seems it's not that complicated to get an Arduino to do something similar: https://www.arduino.cc/en/Tutorial/Datalogger
The trick seems to open and close file each time. I'll have to code a bit to get it to open a new file every day, so if there is random corruption, I'll only lose a day of data.
I'll be retrieving the data via the USB port once a day.

Another option would be ditch most of this and use an Adafruit Datalogger: http://www.gotron.be/en/bouwkits/adafruit/adashield/adafruit-assembled-data-logging-shield-for-arduino.html

Feel free to comment, even if you think I'm doing it all wrong and should use Raspberry Pi instead (my only other realistic option)...


 

[Thor-Arne]

I'd suggest:

1. Arduino UNO
2. SD-RTC shield
3. 7-12V PSU
4. Sensors

The Arduino has a great support community, UNO being among the most popular boards.
The PSU must be more since it have to be more than the dropout of the internal regulator.

[Ian.M]

The sole point of using something like an Arduino rather than a Raspberry PI or other full featured Linux SBC is to minimise space/weight  and power consumption.  The software development effort required will certainly be greater on the Arduino platform.

Assuming the backup power issues to ensure continuous logging steer you towards the Arduino (as providing longterm battery backup to a Pi is a real bitch due to its much higher current consumption and restricted supply voltage range), you'd be an idiot not to get the Adafruit datalogger shield.  RTC, full size SD card, and a significantly larger prototyping area + example code specifically for data logging from Adafruit, what's not to like?

You'll simplify your life by writing the data directly to a file in CSV format which any spreadsheet and most databases can open directly and swapping SD cards daily rather than messing around trying to get USB access to the SD card in the logger working.  Use a push button to request SD card removal (so your sketch can close files etc.) and add an indicator LED for 'Safe to remove card'.  You can probably buffer a minute's or half a minute's data in RAM (which is a good idea anyway to reduce the number of SD card writes  and reduce the risk of card corruption) so provided you swap cards promptly, you wouldn't even loose any data.   I would suggest rotating log files on an hourly basis, it reduces the risk of data loss and makes file sizes more managgeable for post-processing.

You can feed an Arduino board's regulator direct from a 12V lead acid battery and keep the battery charged with a cheap float charger or even a solar panel if operating outdoors at a remote site.   Even a small SLA battery could power your board and sensors for days, giving you plenty of time to sort out any power interruptions without loosing data.

[ez24]

6) Open office Calc.

long time ago this gave me problems (sorry I forgot what) so I now use

LibreOffice Calc

just an alternative

[gildasd]

Thanks for all the answers...

Just writing stuff down for public consumption helps crystallise ideas into something realistic.
By the end of typing, I was already thinking along these lines...  Your answers helped confirm my choices; so I think I'll go:
1. Arduino UNO
2. Adafruit data logging shield
3. 7-12V PSU (with a backup battery on the "real" project) - even if the whole thing will use only about 100mA...
4. Sensors

This seems to offer the shortest and safest development pipeline.

[SL4P]

As usual, there are helpful and not-so-helpful comments flying all over the place.
A 2GB SD card (typical default support for Arduino) holds a lot of sample data.
As mentioned,I f you sample your inputs and timestamp them to file, you'll get a decent chunk of logging.
The power requirements are something you just have to manage, but there are several approaches for a three month trial.

So let's look at the storage...
4 x analog (2 byte / 10 bit) samples = 8 bytes
1 x digital count value (long int) =  4 bytes
- and the (long int) timestamp = total of 16 bytes per sample period

So extrapolating that out to 2GB,
yields a sample rate of <very fast>, probably more than you need.

[Ian.M]

Lets do the maths properly on a CSV file.
Each 10 bit ADC reading needs  four decimal digits.  Each unsigned long int needs ten decimal digits.  Each value except the last needs a comma after it.  Finally you need a <CR><LF> end of line code.  So that is 5*4+11*2+1 which equals 43 bytes per record, and, logging once per second is 3600 records per hour, or 154800 bytes.
One 2GB SD card would be good for over 18 months worth of data, but it would still be a good idea to swap cards at least weekly to reduce the risk of data loss.

Obviously, the Arduino's MCU has limited RAM so you should convert to decimal ASCII when you write to the file rather than when the sensor readings are buffered in RAM.

[SL4P]

Good point, I was counting for binary data storage...
I'm guessing you don't trust SD cards!  😉

[ez24]

Adafruit data logging shield

good luck

discontinued at Adafruit and out of stock at Maker

[retrolefty]

Adafruit data logging shield

good luck

discontinued at Adafruit and out of stock at Maker

 Are you sure. Adafruit presently showing in stock?
http://www.adafruit.com/products/1141

[Ian.M]

Good point, I was counting for binary data storage...
I'm guessing you don't trust SD cards!  😉

Its not that I don't trust them, though its well known that like other FLASH block devices they don't do well with frequent small writes,  its that I prefer to make pessimistic design decisions as I know that I if I do, I am likely to be pleasantly surprised by the final real world performance/reliability.

[ez24]

 

I forgot to check the details of this

http://www.adafruit.com/products/243

which shows this

https://www.adafruit.com/product/1141

[Kappes Buur]

You could check out http://www.instructables.com/ for an Arduino based weatherstation
to get some idea on how to bring your idea to fruition.

[gildasd]

Lets do the maths properly on a CSV file.
Each 10 bit ADC reading needs  four decimal digits.  Each unsigned long int needs ten decimal digits.  Each value except the last needs a comma after it.  Finally you need a <CR><LF> end of line code.  So that is 5*4+11*2+1 which equals 43 bytes per record, and, logging once per second is 3600 records per hour, or 154800 bytes.
One 2GB SD card would be good for over 18 months worth of data, but it would still be a good idea to swap cards at least weekly to reduce the risk of data loss.

Obviously, the Arduino's MCU has limited RAM so you should convert to decimal ASCII when you write to the file rather than when the sensor readings are buffered in RAM.

Never thought of it that way
Even adding a header and footer every hour, it's still 3x more than I'll ever need. You saved me buying a 16Gb SD card!

[gildasd]

Good point, I was counting for binary data storage...
I'm guessing you don't trust SD cards!  😉

Its not that I don't trust them, though its well known that like other FLASH block devices they don't do well with frequent small writes,  its that I prefer to make pessimistic design decisions as I know that I if I do, I am likely to be pleasantly surprised by the final real world performance/reliability.

Depending on what Arduino I can get today (forgot about the mega block party called Gentse Feeste...), the idea is to write every minute, ten minutes or hour.
The internal mem is there for something !

[gildasd]

Adafruit data logging shield

good luck

discontinued at Adafruit and out of stock at Maker

 Are you sure. Adafruit presently showing in stock?
http://www.adafruit.com/products/1141

There are still a few of the older green version in stock, but I'll be getting the new blue version.

[gildasd]

You could check out http://www.instructables.com/ for an Arduino based weatherstation
to get some idea on how to bring your idea to fruition.

I've seen that, but I am resisting reading it until I breadboard something simple...

(Sorry about the multiple posts; I'm answering from my bed with my IPhone 4. My first full week working on industrial electric motors makes me fully apreciate an extra few minutes on my back...)

[Psi]

Data-logging for a long period adds extra complexity over a typical MCU projects.

ie, how do you test if something is going to work correctly for 3 months without actually running it for 3 months?

If getting the data is very important you could have 2 Arduinos logging the sensor data together. Start the 2nd one maybe 1 week after the first.
If any coding bugs causes the 1st arduino to stop working the 2nd will still be running for a week and hopefully you will be checking it regularly and can restart the 1st one. eg..

These are the sort of things you need to consider when data logging for months.

[gildasd]

Data-logging for a long period adds extra complexity over a typical MCU projects.

ie, how do you test if something is going to work correctly for 3 months without actually running it for 3 months?

If getting the data is very important you could have 2 Arduinos logging the sensor data together. Start the 2nd one maybe 1 week after the first.
If any coding bugs causes the 1st arduino to stop working the 2nd will still be running for a week and hopefully you will be checking it regularly and can restart the 1st one. eg..

These are the sort of things you need to consider when data logging for months.

That's why I intend to so a 3 month test now 

[Ian.M]

If you are as pessimistic as another student I collaborated with on a long term datalogger project, you will duplicate the sensors as well with independent connections.   Admittedly he was measuring ground and buried sample temperatures with Dallas 1-wire sensors, which significantly increased the risk of sensor failure and he needed months of uninterrupted data for his thesis.  We went lo-tech brute force for the power supply:  a couple of good condition 12V car batteries from a scrap yard + a 9V Alkaline 6xAA backup battery bank to allow the car battery to be swapped out for charging without loosing power to the logger.  Its worth logging main battery voltage so you can spot any developing problems before total failure occurs.

However, if the logger software is properly designed there will be very few things that can crash it short of hardware failure or external EMI.  The hourly log file rotation keeps file sizes manageable so you aren't likely to run into any filesystem bugs there.   It would be advisable to either rotate log file folders on a weekly basis so you don't run into problems with the number of files per folder, or quick format the card that is being swapped into the logger to wipe previous logs.

Software testing may need to include manipulating the RTC and SD card filesystem to artificially advance the system state to where it would be after days or months of continuous logging.

One hack to the Adafruit that would be worth looking at would be to lift the EN pin (3) of the 3.3V regulator for the SD card, and wire it to an Arduino I/O pin so SD card power is under software control.  Being able to cycle power to the SD card increases the chances of recovering from a 'hung' card if it fails on write.

Log any fatal errors to internal EEPROM so you can attempt a post-mortem analysis in more detail than "Its dead, Jim ...".

[gildasd]

If you are as pessimistic as another student I collaborated with on a long term datalogger project, you will duplicate the sensors as well with independent connections.   Admittedly he was measuring ground and buried sample temperatures with Dallas 1-wire sensors, which significantly increased the risk of sensor failure and he needed months of uninterrupted data for his thesis.  We went lo-tech brute force for the power supply:  a couple of good condition 12V car batteries from a scrap yard + a 9V Alkaline 6xAA backup battery bank to allow the car battery to be swapped out for charging without loosing power to the logger.  Its worth logging main battery voltage so you can spot any developing problems before total failure occurs.

However, if the logger software is properly designed there will be very few things that can crash it short of hardware failure or external EMI.  The hourly log file rotation keeps file sizes manageable so you aren't likely to run into any filesystem bugs there.   It would be advisable to either rotate log file folders on a weekly basis so you don't run into problems with the number of files per folder, or quick format the card that is being swapped into the logger to wipe previous logs.

Software testing may need to include manipulating the RTC and SD card filesystem to artificially advance the system state to where it would be after days or months of continuous logging.

One hack to the Adafruit that would be worth looking at would be to lift the EN pin (3) of the 3.3V regulator for the SD card, and wire it to an Arduino I/O pin so SD card power is under software control.  Being able to cycle power to the SD card increases the chances of recovering from a 'hung' card if it fails on write.

Log any fatal errors to internal EEPROM so you can attempt a post-mortem analysis in more detail than "Its dead, Jim ...".

Cheers mate!

I am pessimistic, I must have a 12 hour turnaround from a seagull eating it and then getting struck by lighting.

The project will reside on the roof of my uni and use mains. However, I'll probably add a small battery pack (few hours worth) because there is a major building site one lock and bridge away... So excrement will hit the ceiling mounted air moving device if I don't prepare for the brown mist.

I had not thought of the folder rotation, but that makes perfect sense: cameras (and other stuff) use this.

Just came back from the shop, the Arduino UNO box is tiny, you can't fit many matches in there.
Right now, I'll concentrate on getting "helloworld" noob tribulations done and dusted before moving back to datalogging in earnest.

[Ian.M]

The *MINIMUM* battery backup pack to consider for that sort of application is 100 hours duration.  If power fails shortly after your daily logger check and its the day before a weekend that has an adjacent public holiday, its quite likely that you may not discover the problem till the daily logger check on the next work day.

That's the attraction of a massive car battery - a couple of hundred ampere hours of reserve means you really don't have to worry much about sensor and MCU power management as long as you can keep the mean current down to a few tens of mA.  It also serves to stop your logger blowing off the roof during storms or being stolen by seagulls! (And your Belgian gulls - at least the ones around the Oostende fishmarket - are the largest and most agressive ones I have ever met!)

[JacquesBBB]

I  have been doing a similar long time  data logging project, but with the additional  difficulty that
I wanted everything to run on  a single LIPO battery for months.
In  this case,  5 V arduino is not good, and it is better to design  from scratch using 3.3 V  8 Mhz
Atmega328, in order to save all possible power, in the lines of what is done
by the jeelab http://jeelabs.org/.

But if you are on the main, your solution of using plain UNO should be OK.

For the SD cards, I put two of them,  with a switch between the two with leds that shows the one which is active.
This allows to remove the inactive one without any   loss of data.
Even if I take away the SD card for several hours.

Even if you are on the main, you need a backup power as small breakdown can occur. In my case, I suspect that this led
to some corruption of  the SD cards.

Beware also of the precision of the RTC.
The one in the Adafruit shield  is the DS1307. This  is not very precise, with possible drift of a few mn per month.
 Depending on the  time accuracy you need,  you will need to use the DS3231 , or even
some GPS timing for ultimate precision, with no drift.

But the best is to start with what you have, and let it evolve afterwards.
As it was already said, doubling  the logger is very useful, as you can improve one while continuing to
log data.
 

[Ian.M]

For meteorological purposes, sample timestamping sub-minute  absolute accuracy isn't likely to be important as long as the sampling rate is reasonably accurate.  If you log the timestamp when the 'request card removal' button is pressed, and carefully manually record the time you do so each day, you will be able to compensate for long-term clock drift when post-processing the data without increasing the hardware cost of your logger.   OTOH GPSes for RC plane/quadcopter appications are reasonably affordable, so if you need precision timing I'd add on one of those and correct the clock to it frequently.

[gildasd]

I'll be running two test jigs:

One "optimised" and one reference. As long as they drift the same, no big deal. Maybe I'll need to reset them both once a week?

As for battery life, my Uni is in Antwerp Harbour, that has a dedicated backup power-plant, so even if ISIS drives a semi full of fertiliser and bad intentions into the refinery's gates, we should get a blackout of minutes - in theory - 1 hour max in real life.