Yet another cheap thermal imager incoming.. Seek Thermal

[efahrenholz]

Bktemp, try what frenky did. Just cut and paste his code in, that should put the nail on this. Removing the banding first, then run the code frenky put up. Post an image and let us see how much better this sensor looks
 
You can't average out noise if it doesn't change on each frame. This works traditionally on astrophotography because the noise changes on each frame, so the real data can get averaged in. In our case, we have to do a neighbor comparison to find the bad pixels give them the average of their neighbors.

[miguelvp]

... build a loop that scans each pixels and compares it to its 4 neighbors. So find the average of x+1,x-1,y+1,y-1, divide that in to a variable. Subtract value of current pixel from the variable hosting the average, flip the sign so its positive, and replace the pixels value with the average if it is over or under the average by 10 (or however agressive you choose it to be.)

Already done that:
https://www.eevblog.com/forum/testgear/yet-another-cheap-thermal-imager-incoming/msg538423/?topicseen#msg538423

I went a step futher by checking which pair (horizontal or vertical) has smallest difference in color.
(This is to keep details on vertical and horizontal edges)
The average value of this pair is the new pixel value.

Yeah, and for targeting just the thermistors we already did that using as many orthogonal neighbors as available:
https://www.eevblog.com/forum/testgear/yet-another-cheap-thermal-imager-incoming/msg538322/#msg538322

But frenky's cleans the inside one as well as minor non responsive pixels. So I adapted my code to only clean up the top/bottom/left edges and adapted his code to clean the inside ones, instead of just targeting the patent/thermistor pixels.

[ricksastro]

Bktemp, try what I said. You can't average out noise if it doesn't change on each frame. This works traditionally on astrophotography because the noise changes on each frame, so the real data can get averaged in. In our case, we have to do a neighbor comparison to find the bad pixels give them the average of their neighbors.

That's true in practical terms here, and also in astrophotography where some fixed type of noise can still remain even after calibration.   That's why some guiding software bumps the tracking a few pixels in between each image to allow averaging (or better yet statistical Sigma rejection).

Like I said, it's probably of less consequence here for a typically handheld device that needs higher refresh speeds.

[bktemp]

Bktemp, try what frenky did. Just cut and paste his code in, that should put the nail on this. Removing the banding first, then run the code frenky put up. Post an image and let us see how much better this sensor looks
 
You can't average out noise if it doesn't change on each frame. This works traditionally on astrophotography because the noise changes on each frame, so the real data can get averaged in. In our case, we have to do a neighbor comparison to find the bad pixels give them the average of their neighbors.

I do not try to avarage it out. I am trying to generate a black level reference image and subtract it from all images. This works well, as long as there are no big thermal differences. The image is almost noise free, without replacing any pixels. The nonlinearity is the main problem. I can tweak the gain to compensate either hot or cold areas, but not both.

The cold background is almost noise free, but the hot areas have now inverse noise. frenky's noise reduction will be the next step to remove the dead pixels.

[efahrenholz]

@bktemp,

I'm not sure I am following you very well. It truthfully looks like the image noise was inverted completely. I can see black pixels in the background that suggest this.

[bktemp]

Enough image processing for today. The results:
- The 207th value of each line can be used to compensate the horizontal noise, but there is some linearization needed before this works for a wide temperature range
- It is possible to remove the static noise without loosing details by subtracting a dark noise frame, but same problem as before: linearization is needed to be able to simply subtract data
Maybe somebody with more in depth knowledge of thermal imaging cameras can use this to write some useable code that works without manual adjustments for each frame.

The processing steps:
From left to right:
- input data
- reference frame subtracted
- black pixels removed
- horizontal bars compensated
- avaraged noise image subtracted (does not work very well except for background, therefore not used for further processing)
- avaraged noise image subtracted, but with some more tweaking. It should be much better with a true image of a flat surface or using a larger number of reference frames.
- frenky's filter applied
- another noise reduction filter (maybe a bit too strong)
- palette applied

[eneuro]

Not sure if those are bright pixels or subtracted darker pixels from the calibration frame.   Regardless, you should be able to do a bad pixel map of the patent pixels and those and median them out.

Why do you suggest to use median for those 2143 hexagon sensor pattern pixels?
I've used average but made it in tricky way, so it do not include dead looking black ones if any 
Of course made sparse matrix of those hexagon patterns, so only 2143 iters needed to find those sensor pixels and fill them with desired color.

If someone plays with OpenCV this code nicely opens 16bit grayscale PNG images

Code: [Select]

Mat img = imread(fin, CV_LOAD_IMAGE_GRAYSCALE| CV_LOAD_IMAGE_ANYDEPTH );


[efahrenholz]

@bktemp

That image looks good up to right before that heavy filter you applied. I'm curious, are these images already averages of 5 frames or are they individual frames? If so the noise could be filtered with frame stacking, but so far it looks really good!

[eneuro]

The processing steps:
...
- reference frame subtracted
- black pixels removed
...

Just thinking of FFT on reference frame and some kind of averaging aproximation before using this calibration data.

I think, removal/updating those hexagon sensor black pixels should be the first step before any other operations with sensor data regardless it is calibration frame or image data 

[bktemp]

@bktemp

That image looks good up to right before that heavy filter you applied.

It was this filter:
http://www.ipol.im/pub/art/2011/bcm_nlm/
It is quite slow, but does amazing work. But it is not really suited for heavy noise. If the strength is too high, it will smooth the image, if it is too low, it will simply keep the noise without any processing.

I'm curious, are these images already averages of 5 frames or are they individual frames? If so the noise could be filtered with frame stacking, but so far it looks really good!

I have used the recorded USB data from marshallh:
https://www.eevblog.com/forum/testgear/yet-another-cheap-thermal-imager-incoming/msg533801/#msg533801
Therefore they are the nearest raw data you can get without modifying the firmware. From mike's video the camera seems to be avaraging frames internally.
But additional avaraging should work, especially when using images processes with multiple reference frames. After each reference frame, the noise seems to change a little bit.

[frenky]

@eneuro:
And after that line tuning with pixel 207 should be done on both frames before subtracting cal frame from image frame.

[Fry-kun]

Funny you should ask, I've been poking at this for a couple of days. I've written a Python program which uses PyUSB to capture calibration and Image Frames from the Seek Imager.

Awesome, gave it a quick shot. A few notes for reference (in case others want to play with it):
* works with PyUSB 1.0.0 beta2. Fedora 20 comes with 1.0.0 alpha3 which doesn't work
* line 18 "import Image" is more portable as "from PIL import Image" (didn't work for me until I changed that

Got a few simple images so far, yay!

[ricksastro]

Not sure if those are bright pixels or subtracted darker pixels from the calibration frame.   Regardless, you should be able to do a bad pixel map of the patent pixels and those and median them out.

Why do you suggest to use median for those 2143 hexagon sensor pattern pixels?

I used the term generically in reference to the neighbor pixel method presented here.

[eneuro]

@frenky
Just computed those sensor hexagon pattern pixels count based on created before sparse matrix of those black pixels:

seek_thermal_test: Seek Thermal sensor hexagon pattern (black) pixel count in row: 14 14 14 13 14 14 14 13 14 14 14 13 14 14 13 14 14 14 13 14 14 14 13 14 14 14 13 14 14 13 14 14 14 13 14 14 14 13 14 14 14 13 14 14 13 14 14 14 13 14 14 14 13 14 14 14 13 14 14 13 14 14 14 13 14 14 14 13 14 14 14 13 14 14 13 14 14 14 13 14 14 14 13 14 14 14 13 14 14 13 14 14 14 13 14 14 14 13 14 14 14 13 14 14 13 14 14 14 13 14 14 14 13 14 14 14 13 14 14 13 14 14 14 13 14 14 14 13 14 14 14 13 14 14 13 14 14 14 13 14 14 14 13 14 14 14 13 14 14 13 14 14 14 13 14 14

Probably it could be better apply this line tunning even before updating those hexagon pattern black pixels, while processing its neighbor with not corrected pixels in rows up and down will introduce incorrect values to processed row pattern pixels inside.

So, the main concern now is to find the best way to apply those corrections from 207th column.

Anyway looking for good description of such  microbolometer with zone thermistors, but not too much about it found so far only a few patents and history of infared detectors:

http://en.wikipedia.org/wiki/Bolometer

patent: Use of vanadium oxide in microbolometer sensors

pdf: History-of-infrared-detectors

[bktemp]

Probably it could be better apply this line tunning even before updating those hexagon pattern black pixels, while processing its neighbor with not corrected pixels in rows up and down will introduce incorrect values to processed row pattern pixels inside.

It makes sense. I have already done that in my example unintentionally, by doing all subtractions at one place.
Removing the hexagon pattern should be the last step before applying the noise filter.

By the way, the easiest way to go through all hexagon pattern pixels:

Code: [Select]

x = 10;     // first black pixel is at x=10
for (y=0; y<156; y++)
{   for (; x<206; x+=15)
    {   ...
    }
    x-=208+8;
    if (x<0)
        x+=15;
}


[frenky]

I agree that line tuning should go before updating those hexagon pattern black pixels.

@bktemp
Very nice code for locating hexagon pixels.

@all:
Btw great work guys...

[eneuro]

By the way, the easiest way to go through all hexagon pattern pixels:

Yep, but anyway we need sensor hexagon pattern pixels matrix, for example when dealing with rows stats, so wrote it once and now it was easy too in OpenCV to generate some row stats from router calibration and image frames using this 

Code: [Select]

Mat hex= seek_thermal_black_dots_matrix();
...
      for(i=0; i<h; i+=k ) { // i
        val0= cor.at<ushort>(i, n); // 207th column             
        sum= 0;
        m= 0;
        for(j=0; j<n; j++ ) {
                val= cor.at<ushort>(i,j);

                if(  hex.at<uchar>(i,j)>0 ) {
                        // Skip hexagon pattern pixel
                } else {
                        sum+= val;
                        m++;
                }
        }
        // Non zero pixels average
        if(m>0 ) {
                avg= sum/m;
        } else {
                avg= 0;
        }
        // Stats
        fprintf(f,"%03u %0.3f %0.3f %03u %03u \n",
                        (i+1), avg, val0, m, n );

        } // i
With above code I've got such nice stats of each row:

Code: [Select]

row    avg   207th    count   width
001 7947.229 5533.000 192 206
002 8029.047 5188.000 192 206
....
154 8198.523 5180.000 193 206
155 7953.286 4873.000 192 206
156 7892.047 5184.000 192 206
When we plot such pixels row averages vs this additional 207th column we get something like this for router raw data (router_image1c.png and router_image2i.png) provided earlier by @ricksastro:


It is time to apply this additional 207th column data somehow with this plot in mind 

Note: It will be interesting do the same for those two @ricksastro thermal frames of flat surface (flatpaper_image1c.png and flatpaper_image2i.png).

[efahrenholz]

@bktemp,

Do you have a git with your source code? There's quite  bit of fragmented source code on the topic now, perhaps someone could host it with all the updated source. Maybe some of us could work on an interface, tools, palettes etc

[frenky]

I was testing today this line tuning with 207th values.
I works great on low temp background, but on higher temp areas it adds light stripes.
So it probably should only be applied to the low temp areas...

[miguelvp]

You just need to find the magic number or range to do a nice ramp to/from

[bktemp]

I was testing today this line tuning with 207th values.
I works great on low temp background, but on higher temp areas it adds light stripes.
So it probably should only be applied to the low temp areas...

If you increase the gain applied to the 207th value it works for hot areas, but then it leaves stripes on dark areas.
When trying to remove the static noise you will get the same problem.
I have tried to use the data value to modify the gain before adding the 207th value:
pixel=pixel + (pixel+offset)/gain * 207th value
This works a bit betten than just adding 207th value / gain, but it is still not good enough to be useable.
Therefore I would convert the data to linear temperature scale before any further processing, but I have no idea how to do it.

@efahrenholz
My code is bodged together from several other programs. I just wanted to play a little bit with the raw data and did not care about good coding practices.
Better use the version from sgstair and add the dark line compensation there. He has written a much better code.

[eneuro]

Therefore I would convert the data to linear temperature scale before any further processing, but I have no idea how to do it.

I have some idea how to tweak those temperatures, but no physical device for the moment 
If someone have IR thermometer and ambient temperature thermometer than usual flatiron could be heated in quite cold room (eg. 0*C 10*C 20*C 30*C) to quite high temperature, than disconnect from mains and one could make series of thermal sensor raw data frames and record also iron temperature measured by IR laser thermometer.
So, experiment is easy, and flatiron is good heat source so nice suspect to watch, for example put close to it another metal part, for example copper PCB template to see how is heated from distance througth air 

Make scan of USB sensor frame with shutter (calibration frame -mark this image with -cal- in its name), than quickly make normal image frame (name it with -img-) and read iron temperature-probably even multimeter sensor could be fine to estimate this temperature and get absolute value.
Wait 5 minutes, take another two shots read temperature, and so on-if automated each 1 minute delays could be better to see any gradients efects if any too.
Probbaly the easiest way to do it is of course add delays to our custom software and do all those measurements automatically,
just by connecting I2C thermometer to PC 
But ok, even using manual one if making those frames shots will be programmed, than GMT clock could be used and timestamps added to each image, so one could easy control only multimeter/thermometer and log iron temperature changes in time.
Than those manual measurements could be approximated (I can do it for you in maxima) in time and fusion with PC clock time made to ensure we have estimate absolute iron temperatures, when those frames were taken.
Additionaly in the same way ambient tempartaure close to Seek Thermal sensor could be tracked and added, but the most important is to have real temperature sensor visible )eg. in the middle of the flatiron, so it could be easy to tweak those pixel data close to its position on thermal image.
With automated software and OpenCV it is easy to write those USB raw sensor data images 208x156 as 16bit grayscale PNG to  them and test different image processing aproaches by this forum members, which do not have this cam, but could test those thermal images and comment how different alghorithms affects output images.
Simply, some kind of test images could be created, so people could use the same raw sensor data and than outputs each of them creates and attaches there in this forum thread could be easy to verify and it could be possible to compare between different implementations on different machines (Windows,Android, Linux, etc.).

So, who will sacrifice 15 minutes and flatiron for the Apollo 11: Frase_de_Neil_Armstrong   make those thermal shots with tiimestamps, zip with README text file with manual readings from iron temperature and provide link to do homework by other EEVBlog members

[miguelvp]

Last week I forgot to ask a co-worker for the IR thermometer but I didn't this week, he forgot 2 days in a row, hopefully he remembers today or tomorrow

But I have my own set of experiments. I wan't to see how accurate the central spot for the Seek is both at boot time and after being on for a while.

[eneuro]

I wan't to see how accurate the central spot for the Seek is both at boot time and after being on for a while.

It could be interesting another teardown of Seek Thermal for @mikeselectricstuff where he could test how accurate are those temperature spots shown in Android app in comparision to real temperature measurement made on heated object-for example using classic multimeter thermometer and maybe additionally IR thermometer with lase pointer.
I didn't saw such vblog so far, so it could be interesting to see what is scientific quality of temperature this Android app shows on customers screens 
Do they (Seek Thermal) provide some kind of datascheet with expected spots temperature measurement accuracy?
Is it even available any official Seek Thermal manual with such technical information?

Nope, google doesn't provide any link to manual 
Google play: Seek Thermal shows latest software version:

Code: [Select]

October 30, 2014
Size
7.6M
Installs
1,000 - 5,000
Current Version
1.4.0.2
Requires Android
4.3 and up
Maybe will try download it later under Android and see if maybe inside is Seek Thermal users manual? 

No, even on its product page no way to find any manual 

This is all technical specs availabe on this page Seek Thermal Camera - ANDROID ? 

Code: [Select]

Technical Specs:
    mUSB Thermal Camera for Android devices
    Works on most Android devices running 4.3 or higher that support USB on the go. See Device List for Compatibility
    True Thermal Sensor
    206 x 156 Array
    32,136 Thermal Pixels
    12um Pixel Pitch
    Vanadium Oxide Microbolometer
    Chalcogenide Lens
    36° Field of View
    Magnesium Housing
    Long Wave Infrared 7.2 - 13 Microns
    -40C to 330C Detection
    < 9Hz
    Includes Protective Carrying Case
    Model: UW-AAA
Very interesting is this < 9 Hz output frequency - does it mean it could be 1 Hz after some app updates? 
Only temperature range provided: -40C to 330C Detection
It is 

[mikeselectricstuff]

The Seek website has a glaring absence of any specs on accuracy or NETD - if there's no spec, people can't complain it doesn't meet it!