Yet another cheap thermal imager incoming.. Seek Thermal

[efahrenholz]

The next thing that needs to be done is to unlock that firmware. I'm curious to see how they process the frames off the sensor.

[mikeselectricstuff]

The next thing that needs to be done is to unlock that firmware. I'm curious to see how they process the frames off the sensor.

ISTR someone said they'd found a firmware image in the .apk - should be fairly easy to determine if this is ARM code, or is obfuscated/encrypted.
At some point I'll read the flash - there may be stuff like cal data in there.
BTW I did establish that the test pads on the PCB do not connect directly to the flash - probably JTAG or SWD.
   

[marshallh]

I have the firmware image loaded in IDA. But I'm too shit at reversing to understand anything besides what's evidenced by the debug strings.

[Andrew Seltzman]

Has anyone made a USB API (something similar to the winusbdotnet program written by sgstair) in the form of a dll file that can be called from matlab?

[miguelvp]

First file is with the camera facing down during startup for 20 frames.
2nd file is with the camera facing the router during startup for 20 frames.

They were send at <10Hz ?
Do you have any timing hint what frame rate in this raw data could be?
It could be helpfull to estimate when those events took place, so for example delays between calibration frames and normal-do they happen at the same constant frame rate and delay between each those frames?
Such time stamp information when each frame took place could help investigate this raw data...so suggested to output each frame with time stamp info.
OpenCV has builtin functions for calculating such timing, but one have to be carefull and check if task scheduling in OS doesn't affect those time stamp at higher frequencies   

No timestamp sorry, I'll work on adding a time_t timestamp before each frame, what do you prefer, before capturing, after capturing, or both?
i.e: begintime-frame-endtime-begintime-frame-endtime...

And you are funny about the higher frequencies

[miguelvp]

Has anyone made a USB API (something similar to the winusbdotnet program written by sgstair) in the form of a dll file that can be called from matlab?

No but it's trivial to open a file and dump the raw data there to be later analyzed.
That's what I did and fed the raw file to Visiquest for analysis.

[eneuro]

I'll work on adding a time_t timestamp before each frame, what do you prefer, before capturing, after capturing, or both?

I used to use this code to compute how much time some processing is made and when  (of course using  C/C++ OpenCV):

Code: [Select]

// http://docs.opencv.org/modules/core/doc/utility_and_system_functions_and_macros.html
        double t= (double)getTickCount()/getTickFrequency();

        calf.convertTo(cal256, CV_8U, 128 );
        imgf.convertTo(img256, CV_8U, 128 );
        outf.convertTo(out256, CV_8U, ((double)128/25.0) );

        double dt= ((double)getTickCount())/getTickFrequency() -t;

        fprintf(stderr,"%s: Timestamp: %0.9f [s]  Processing time: %0.9f [s]\n",
                argv0, t, dt );
It outputs nice time in seconds how when from system powerup something happened and processing time:

./seek_thermal_opencv_test: Timestamp: 14736.406878773 s  Processing time: 0.000353242] s   

Of course in case of "Sick" Thermal dongle maybe 9 digits not needed, but it doesn't matter if we type 9 or 3 digits in output format 

Back to where put those timestamps probably the best option could be leave raw data untouched  for example in *.raw files and this timing info add to second text file *.idx and in parallel with catching those frames via USB put text in such easy format ready even to plot using gnuplot , etc Facerouter20frames.idx in attachment.

This is the way I deal with this nice raw data now-by outputting additional description text files and images.
I haven't got this timestamp info to try look how it changed in time...

Offset in bytes where frame begins (first byte offset 0x0 of course) is fine while if we have start than it is easy calculate where ends by image size or by tracking those 0x0000 words hexagon pattern.
There is another advantage ussing such kind of metadata text file-add support for those FITS image file format so staring from raw data such index table can be included, as well as other tables with added configuration data or parameters send from sensor via USB if its protocol will be known, so I'm writting sts2img converter with 16bit PNG output by default, but have FITS image format with tables in mind 

[miguelvp]

@eneuro, here you go, timestamps for all the first 20 frames,

Time stamp is 64 bit in hundreds of nanoseconds (DateTime.Ticks) I computed the deltas for the delays.

On the loop I take the timestamp, then capture the frame, and take the next timestamp before writing it to the drive and then loop again. So sometimes there is a delay while the frame is written to disc between frames, usually 1 ms sometimes a couple.

I also put the delta times that took to capture the frame.

Code: [Select]

Frame 1 ID 4
08d1c9758415a0a4
08d1c97584209d4d <- 72.0041 ms

Frame 2 ID 9
08d1c97584209d4d
08d1c9758425584f <- 31.0018 ms

Frame 3 ID 8
08d1c9758425584f
08d1c97584384456 <- 124.0071 ms

Frame 4 ID 7
08d1c97584384456
08d1c9758441ba5a <- 62.0036 ms

Frame 5 ID 10
08d1c9758441ba5a
08d1c9758446755b <- 31.0017 ms

Frame 6 ID 5
08d1c9758446755b
08d1c975844b305d <- 31.0018 ms

Frame 7 ID 1
08d1c975844b305d
08d1c9758488d684 <- 404.0231 ms

Frame 8 ID 3
08d1c9758488d684
08d1c97584a07d8d <- 155.0089 ms

<- 4.0002 ms File Write delay

Frame 9 ID 3
08d1c97584a119cf
08d1c97584b36994 <- 120.0069 ms

 <- 2.0001 ms File Write delay

Frame 10 ID 3
08d1c97584b3b7b5
08d1c97584c67cab <- 123.0070 ms

 <- 1.0001 ms File Write delay

Frame 11 ID 6
08d1c97584c6a3bc
08d1c97584e2deb6 <- 185.0106 ms

Frame 12 ID 1
08d1c97584e2deb6
08d1c97584ff67d1 <- 187.0107 ms

Frame 13 ID 3
08d1c97584ff67d1
08d1c97585170ed9 <- 155.0088 ms

 <- 2.0001 ms File Write delay

Frame 14 ID 3
08d1c97585175cfa
08d1c9758529fae0 <- 122.0070 ms

 <- 1.0001 ms File Write delay

Frame 15 ID 3
08d1c975852a21f1
08d1c975853ce6e7 <- 123.0070 ms

 <- 1.0001 ms File Write delay

Frame 16 ID 3
08d1c975853d0df8
08d1c9758554b520 <- 155.0120 ms

 <- 0.9969 ms File Write delay

Frame 17 ID 3
08d1c9758554dc11
08d1c975856c5c09 <- 154.0088 ms

 <- 1.0001 ms File Write delay

Frame 18 ID 6
08d1c975856c831a
08d1c9758588be13 <- 185.0105 ms

 <- 1.0001 ms File Write delay

Frame 19 ID 1
08d1c9758588e524
08d1c97585a5472e <- 186.0106 ms

Frame 20 ID 3
08d1c97585a5472e
08d1c97585bcee37 <- 155.0089 ms

Edit: No I didn't use the query performance timer based on the cpu frequency, but for this purpose DateTime.Ticks is good enough. We are not talking in measuring very high frequency data after all. And just maybe, DateTime might use the query performance timer for timing purposes, but I didn't dig into the details of the implementation.

If you want to convert it to actual time, I have them printed in 64 bit hex numbers and they represent the
"number of 100-nanosecond intervals that have elapsed since 12:00:00 midnight, January 1, 0001"

So it comes out to the 312 day of 2014 at 2:15 pm more or less, but the deltas have more pertinent information

[eneuro]

@eneuro, here you go, timestamps for all the first 20 frames,

Thx, I hope someone else also will try to look into this raw data so will benefit from this timing info  too
This is for frames from this file Facerouter20frames.dat I guess?
I've choosen this first for analysis while it should be easier to see anything, but noisy flat surface 

BTW: If someone needs to know how Git better for this software related projects this is nice 500 pages book to read after hours pdf: ProGit 2nd edition recomended by SourceForge.net 

[miguelvp]

It was for the face down, but should be about the same .I'll try later on to make one with the data included in the data or on a separate file.

[miguelvp]

@eneuro, Ok I made a new set with the camera held steady pointing at the router with a "third hand tool"

The zip contains a data.dat with the raw data and a data.txt with the timestamp metadata, times are pretty close to what I posted before, but this one corresponds to the actual data in the zip file. The program formats and outputs the txt file at the same time as the raw data.

I'm on the border of 1000KB upload limit so I can't add many more frames, maybe I can squeeze one more frame, but not worth it.

Code: [Select]

Frame 1 ID 4
8D1C9A479478DFD
8D1C9A479526396 <- 71.0041ms

Frame 2 ID 9
8D1C9A479526396
8D1C9A479571E98 <- 31.0018ms

Frame 3 ID 8
8D1C9A479571E98
8D1C9A4796A0A9F <- 124.0071ms

Frame 4 ID 7
8D1C9A4796A0A9F
8D1C9A4797380A2 <- 62.0035ms

Frame 5 ID 10
8D1C9A4797380A2
8D1C9A479783BA4 <- 31.0018ms

Frame 6 ID 5
8D1C9A479783BA4
8D1C9A4797CF6A6 <- 31.0018ms

Frame 7 ID 1
8D1C9A4797CF6A6
8D1C9A479BA9CCD <- 404.0231ms

Frame 8 ID 3
8D1C9A479BA9CCD
8D1C9A479D243D5 <- 155.0088ms

 <- 3.0002ms File Write delay

Frame 9 ID 3
8D1C9A479D2B907
8D1C9A479E9EADE <- 152.0087ms

 <- 2.0001ms File Write delay

Frame 10 ID 3
8D1C9A479EA38FF
8D1C9A479FCFDF6 <- 123.0071ms

 <- 1.0000ms File Write delay

Frame 11 ID 6
8D1C9A479FD2506
8D1C9A47A196000 <- 185.0106ms

Frame 12 ID 1
8D1C9A47A196000
8D1C9A47A35E91B <- 187.0107ms

Frame 13 ID 3
8D1C9A47A35E91B
8D1C9A47A4D9024 <- 155.0089ms

 <- 2.0001ms File Write delay

Frame 14 ID 3
8D1C9A47A4DDE45
8D1C9A47A607C2B <- 122.0070ms

 <- 1.0000ms File Write delay

Frame 15 ID 3
8D1C9A47A60A33B
8D1C9A47A736831 <- 123.0070ms

 <- 1.0001ms File Write delay

Frame 16 ID 3
8D1C9A47A738F42
8D1C9A47A865438 <- 123.0070ms

 <- 2.0002ms File Write delay

Frame 17 ID 3
8D1C9A47A86A25A
8D1C9A47A996750 <- 123.0070ms

 <- 1.0000ms File Write delay

Frame 18 ID 6
8D1C9A47A998E60
8D1C9A47AB5C95A <- 185.0106ms

Frame 19 ID 1
8D1C9A47AB5C95A
8D1C9A47AD25275 <- 187.0107ms

Frame 20 ID 3
8D1C9A47AD25275
8D1C9A47AE9F97E <- 155.0089ms


[jaybeez]

Look what these guys achieved with a 16x4 sensor

http://forum.xda-developers.com/showthread.php?t=2165354

www.mardaso.com

[Fraser]

It looks similar to another mobile phone thermal camera ...IR Blue.

http://www.rhworkshop.com/pages/product-ir-blue-dm

Its open source and there is a built or DIY version available.

http://www.rhworkshop.com/products/ir-blue-dm-kit

In fact if you look at the EU supplier (MarDaSo) , it would appear that the unit you detailed is in fact the IR-Blue in the prototype stages   

http://firsensor.com/Store/

Very limited capability though.

If you want to see clever use of a low resolution sensor take a look at the design by Honeywell licenced to the I.S. Snapshot camera. A 120x1 pixel thermoelectric array is scanned across the output of the thermal camera lens and a 120x120 pixel radiometric image is produced. The sensor array is calibrated against a black body at the beginning of every picture ! The big down side is that it is useful only for static targetas as the scan time is 1.5 seconds. I bought a lovely example named the Thermasnap under the Inframetrics brand name a month ago for GBP250 and, IMHO, worth every penny as its images are very good and accurate considering the somewhat agricultural technology used within. I believe the imaging sled and servo are actually the same as, or very similar to, that used in a floppy disk drive head sled.

Imagine what could be achieved using the same technology as the Thermasnap but with an X-Y sled moving a SEEK microbolometer....super high resolution of static targets 

Patent for the Honeywell design is here:

US5675149

http://www.google.com/patents/US5675149

[callipso]

There once was a design of a makeshift TIC with 1x1 (the regular IR thermometer thermopile) and X-Y servo powered fixture. The scan time was around 1.5 minutes, but the imagery was hi-res and beautiful.

Another thing popped up in my ebay alert - a thermal line scanner. Those were used in aviation before FLIR became the norm, so called pushbroom thermal imaging, or S(ideways)LIR. Nowadays these are used in industry for checking of the temperatures of continuous stream of film material (as far as I understood)

but that we're OTing

[jaybeez]

I was looking at it more from the software side, with his fusion of thermal and visual images. mostly a visual image with thermal overlayed, almost the opposite of what flir is doing with msx. somehow he has a solution for the location of different cameras on different phones.

[eneuro]

@miguelvp, Thx for another set of router images-I think next week after hours will be able finish sts2img conversion software and connect to my image processing Seek Thermal sensor raw data engine and output something usefull, while my 16bit Iron table works very well and already was tested last week and this parametric thermal Iron LUT has size of 65536 (16bit) not 256 (8bit) 

I was looking at it more from the software side, with his fusion of thermal and visual images. mostly a visual image with thermal overlayed, almost the opposite of what flir is doing with msx.

MSX is only strange name given by FLIR to simple few image processing tasks
I think and just testing this from software point of view in OpenCV that with current OpenCV capabilities easy task to make such fusion and will show it soon by producing FITS image with contures and tables of measurement data in the form of datasheet.
Using FITS images with contures output probably we could also avoid those doggy Flir's MSX fussion patent while FITS images are dated as I know back to 80's and this FITS format has backward compatibility for archiving purposes to be able read those old images made many years ago

If to this  MELEXIS thermopile array we add contures from visual (not this visual overlay)

than of course this image will look much better, and using FITS format simply we do not change thermal image at all, only adding contures from visual, so quite easy to connect this array to any modern device via USB, and messing with Bluetooth is not the way I want to go in my solution based on this automotive grade MLX90620 Series Wide (60x15) 16 x 4 Array 3 V 9 mA Thermal Array Sensor - TO-39 .

Even 1x1 MLX90614   Digital, plug & play, infrared thermometer in a TO-ca can be very usefull and in another project using this with.... something extra to make IR eye  at cost below 10$

BTW: It is interesting question if FITS thermal image format with conturs which is part of its specyfication, but not sure how long could violate this Flir's MSX patent claims, but I smell "bullshit" there and only Flir's attempt to make competitors life harder, while FITS format is used in imaginery industry for ages, so maybe this is easy way to avoid this MSX doggy technology which doesn't look as much impressive when we add simple contures   
Just lokking for licensing information and specyfication of FITS format and it susage in commercial applications while this is something new for me but easy to create such files. I think it is good output format to email someone results of thermal inspection, and Flir's MSX is not needed no more 

[callipso]

I finally figured it out.

datascheet

German (or thereabouts).

[efahrenholz]

I just read something the other day about patent trolls being hit with a new law that prevents "ideas" from being patented. So this heavily rolls on the fine line of software. Unless you stole code or your process matches the process of a piece of software, you can't be sued. Idea's are no longer protected, only material products. Imaginary bits ans bytes aren't really solid material things...so this should prove interesting.

[slackaz]

I just read something the other day about patent trolls being hit with a new law that prevents "ideas" from being patented. So this heavily rolls on the fine line of software. Unless you stole code or your process matches the process of a piece of software, you can't be sued. Idea's are no longer protected, only material products. Imaginary bits ans bytes aren't really solid material things...so this should prove interesting.

Ideas are not a statutory class and cannot be protected. But when you turn that idea into something, such as a machine, a system, a method of manufacture, or a process of doing something, then you are in the area of what patents cover. The government has not created any new (past year) law to curtail trolls and/or ideas from being patented. I think what you are speaking of is the Alice decision which reinforces the concept that one cannot patent an abstract idea such as a law of nature or fundamental truth. While the decision was said to not change the interpretation of whether something constitutes an abstract idea and whether a claimed invention is "more than" the abstract idea, the reality has been the opposite. If a set of claims is simply computer code being executed on a generic computer (think cpu+memory+display) then at the moment that seems to raise a red flag as being nothing more than an abstract idea. We will have wait and see how future decisions from the Supreme Court and future laws from Congress shift the pendulum.

I have not seen the FLIR patent, nor will I comment on it, but I will say that when you introduce other things into the claims such as sensors and things not found in a typical computer (again, think cpu+memory+display) then it currently seems you start to edge away from the abstract idea cliff.

[miguelvp]

So I've been adding some more information to display the max and min values of both the sensor raw frame and the calibration raw frame, ignoring values under 2000 and over 49152 (Edit: I'll try to adjust the upper range because it's in the border with the roll up, so it might be that it's a couple of rogue pixels doing this).



It all starts with some sane numbers, min 3300 max around 10000, but then it starts climbing on the max value (I'm pretty sure it's overall because the image doesn't seem to change much but then again it could be a couple of pixels that get out of wack and the calibration takes care of that.

It increases around 300 per flag event more or less.



It does climb all the way to 49100 both at the sensor and the flag events, and then it rolls back to the 10000 range.

Then it stays at the 10000 range for a while I thought it was stable but nope it doesn't stay there forever.

Not pictured: After a while the min goes down to around 2000 then back up to 3300 and eventually the max starts climbing up again.

I'm thinking this is more a firmware thing than the sensors themselves.

I'm going to add a capture frame button to get the current sensor and calibration frames to see if I can figure out how the actual image changes overtime. That drift is making it hard to come with a simple capture reference scheme because of the range change, so this has to be solved first before attempting to clean up the image.

Edit: I did another run and it was stable at 10000 max and 3300 min until I placed a hot source in the field of view (lighter) then that triggered the drift in the firmware or wherever that is happening. I also bumped the upper limit to 50K (51200) and it went past the 49100 mark up to 50995 then rolled to 9416, so the rogue data (probably just a couple of pixels) are at fault.

Will implement the save individual frames to see what pixels are doing it, since the resulting image subtracts the overranged values from the calibration image it doesn't affect the result.

[eneuro]

So I've been adding some more information to display the max and min values of both the sensor raw frame and the calibration raw frame, ignoring values under 2000 and over 49152

Seek Thermal sensor has 14bit output or more bits while @ 14bits maximum unsigned value is... 16383 ONLY so 50k value is 3 times bigger than 14 bit maximum, but Ok we'll see what I get from this raw data and compare

Anyway watched again this Seek Thermal gradient issue teardown by @Mike and made those screenshots for some investigation, while it is cleary visible that this shutter looks smaller than sensor external top dimensions and it looks like without lens holders its end position is not limited and one sensor edge formed some kind of triangle not covering this sensor uniformly (selected shutter edges and made it more visible by adding more green).


Additionaly lens holder (its bottom view which lands above sensor) is white and metalic with hole where shutter moves, so when PCB MPU makes alittle more heat there than this shutter will mix air close to sensor when is pushed for calibration frame and it looks like so many ways to move air at higher temperature than metalic lens holder into it and while this shutter at the edges creates such strange triangle cuts, maybe this affects temperature and that is why it is not uniform and we land with those strange gradients? 

Again, in Flir 4 this shutter completelly cuts IR from outside in front of lens-there in Seek Thermal I do not know-I do not like this shutter design at all while we have close to sensor asymetry and hole where external heat can go inside at higher rate than via metalic lens holder which has much higher thermal capacity 

BTW: Isn't a little bit noisy sound present during those calibratons event, while it looks like shutter might hit walls of lens holder or it only lokks like this from this perspective those images were made and it could look better when were made perpendicular to sensor surface? 

What if aka Flir 4 nice shutter were used in front and lens holder filled to avoid external air flow over sensor and oryginal shutter removed?

Note: This lens holder one edge close to sensor was milled by @Mike, so in oryginal Seek Thermal this hole is smaller 
UpdateL Captured there this milled white corner from first @Mike video teardown:


Now, when we have oryginal lens before removal this critical part, it is clear that heat around lens tries to go inside above sensors when shutter is opened, BUT... of course 1/3 or 1/2 of sensor height is masked by this white metal part which @Mike milled in his first teardown video, while huge part of sensor side is EXPOSED to external heat REGARDLESS of shutter is inside/outside lens holder-die cleary visible form this perspective-so of course sensor corner not masked gets more heat and we have our TEMPERATURE GRADIENT NOT MYTHBUSTED, but REAL and it HAVE to happen in such design 


Marked in this subimage from above side of sensor and possible effect of masking one half of shorter side by white metal lens holder and constant warming exposed sensor to air througth hole where shutter moves   
MPU is on other side of PCB, but close to enclosure so everything around heats up, but only one corner of the sensor is exposed all the time to this heat 

[miguelvp]

I did change the code to only keep from 2000 to 16383 and make the rest 0, but I still want to know what pixels are drifting because they will create problems even within the 14 bit range for cleaning up the image.

[efahrenholz]

Look at the inside of the lens holder. There's a huge flat mass that the lens sits in just above. The paint has a slight reflective quality to it, besides it being white. The majority of issues people are having occurs when the camera has been running for awhile, a gradient appears from the edge. The sensor has been shown to reach 38°C in thermal photos. That's fairly warm, enough to reflect off the flat bottom of a white lens holder. Most of the lenses out there aren't aligned exactly to the one before it. The glue shrinks as it cures, which could send it in any direction. This explains why the gradient manifests itself in different areas for some people.

[Fraser]

I concur that the FFC shutter is a weak point in the design. It is certainly very agricultural  when compared to others that I have seen. Most others are very thin metal or plastic with a decent actuator that snaps them across the signal path and covers the microbolometer.

There is still the matter of the regularity with which the FFC is required for a decent image. Once the unit has been running for a period of time I would expect temperature equalisation in the microbolometer and less regular FFC events. In the case of the SEEK camera the microbolometer appears VERY unstable and I can't see why this should be. More research into this is needed. A 12um microbolometer is a technical challenge and it is interesting that they have been around for several years, yet few if any manufacturers have used them in TIC's. There may be issues with such microbolometers of which we are currently unaware.

I remember when CCD resolution kept increasing. As the pixels got smaller and closer together, there were unwelcome interactions between pixels that degraded the image that was produced. That lead to some 12MP cameras producing images inferior to an 8MP camera.

[eneuro]

That's fairly warm, enough to reflect off the flat bottom of a white lens holder.

Do you mean this PCB part marked it with a more red in previous post?
There were no glue I guess and small air gap must be there.
lets check it again on @Mike teardown