Running Seek thermal cameras (& others?) on a Raspberry Pi

[IwuzBornanerd]

 I have been meaning to start this thread once I get my new SDL2 code ready for release but I decided to start it now because I recently learned that a graduate student (@kiranjayaraj) has been using my old Python code to control a Seek in his research.  He has asked several questions about the code via PM & it occurred to me that the discussion could benefit others who use the program, so this thread can provide a forum for such discussion.

I didn't intend to "support" the program and had even abandoned the Python code because it was incapable of running surveillance while displaying the images at even 1x scale, but knowing that someone has been using that program for such a serious purpose has motivated me to make some relatively easy  updates to the code that people who have used it would probably like to have, so I will post the new code here.  It is also possible that some people might not want to use my new SDL2 code on a Pi because it has to be run in console mode (no X-windows).

I made the topic general enough that we can post links to other software that runs on a Pi.  For instance I know from another thread that people are using this code on GIthub: https://github.com/maartenvds/libseek-thermal which is more of a "bare-bones" program.

The main updates to the Python code have to do with the color palette.  The program now reads in palettes from ASCII file lists of RGB values.  The file names are listed in the config file, so people can create & use their own palettes.  I have included 6 palette files with the program.  Hopefully @frenky doesn't mind that I stole a couple palettes from his software package.   There is also a setting for autoranging the temperature use of the palettes wherein the program finds the min & max temperature in each frame and applies the entire palette to that range.  This is, I guess, the common way of doing things and I kind of like it for the smooth palettes but I still like my distinct color palettes without the autoranging.  I have also added the pixel bias correction which makes for cleaner images.  The effect of this was not noticeable with the original Seeks but now that they are much less noisy this correction makes a significant difference when the camera temperature is even just a few degrees different from the temperature at which the "user cal" was done.  To accomplish this the program now saves the shutter data to a file when the "user cal" is done.  There are other things I could or should do but I'll toss this out there now & see how much interest there is otherwise I'll just keep coding and never get it released.    At least with this thread there is a place to comment or complain about the program.

When the palette is in autorange mode the program will display markers at the min & max scene temperatures.  These will update every frame and can appear erratic in some instances (see attached sample video), so I also provided the ability to turn off the min/max marker display.

I have changed the program file names so that compiling the new version (of pixelmath) won't mess up any earlier versions installed on the system.  I also woke up to the fact that there are things I should have changed in the setup.py file instead of leaving the boilerplate stuff there, and I changed some things in that file as well as the file name.  Still, I suggest unzipping the archive to a new directory of your choice.  Once that is done, follow the instructions in the Readme.html file.

I have successfully run my updated version of the Python code on my Pi2B running Raspbian Stretch and on my Pi Zero(W) with a 3.5" touchscreen running Raspbian Jessie (January 2017 version).  I have not gotten Stretch to work with that 3.5" screen but have not spent much time on that.

Attachments below are:
The revised program
Video showing "erratic" markers during palette autorange
Screenshot with "Iron" palette autoranged
Screenshot with my hand-made palette at 5 shades per degree (I like the extra isotherms I get with that palette. )

[Berni]

I just made a similar thing for the Thermal Expert on the RaspberryPi using C++, but its not quite fully featured yet. It just displays the image and that's it.

[tonykids]

Just did a quick review of your code ,it seems that you did  a lot of research on different frame IDs
I get an average of 100 frames to make a cal frame and it does better than use one frame,both on compact XR and pro.

[IwuzBornanerd]

I just made a similar thing for the Thermal Expert on the RaspberryPi using C++, but its not quite fully featured yet. It just displays the image and that's it.

Thank you for posting that.  Did you use their SDK?  That was the subject of @kiranjayaraj's initial post (https://www.eevblog.com/forum/thermal-imaging/thermal-expert-raspberry-pi/) and he was having difficulty making progress with it.  I offered to help with the Pi aspects but I don't have a TE camera so I can't be much help with that.

If you are willing to publish your code when ready, feel free to post it on this thread or provide a link to wherever you do post it.  The reason I put "& others?" in my title here was that I thought that people might run into similar problems with the Pi regardless of the brand of camera so this topic could be a sort of gathering place for such information.  Maybe that's not a great idea, but we will see.

[IwuzBornanerd]

Just did a quick review of your code ,it seems that you did  a lot of research on different frame IDs
I get an average of 100 frames to make a cal frame and it does better than use one frame,both on compact XR and pro.

Thanks for the thumbs up. 

Looking at the old file directory I see I have averaged up to 100 frames, and 50, and 30 but I don't recall how much difference I noticed, but beware of averaging too many because if you get frames from 2 different shutter periods you will be averaging in some bias change differences and this will mess up your cal factors.  I don't know if the camera ever takes than 60 frames in one shutter period, but I just looked at my 20  minute warmup file & the highest it got to was 60 in that time period.  If the camera is very well warmed up & stable there might only be a few pixels that have changed bias, but I would not bet on any number.  I have observed where pixels toggle back & forth between 2 bias settings every single shutter frame at a certain temperature.  I also wonder if averaging is such a good idea because the image tends to degrade between shutter frames.  Maybe having the non-averaged noise in the cal factors is worse than having the degradation though.

With the lower noise of the newer cameras we really need more precision in those cal factors (in order to get fractional degree accuracy) and that is one of the things I did not yet put in the Python code.  The original camera had 3 digit percentage numbers in frame 10 but the newer ones have 4 (5?) digits that are not simple percentage values and I accommodate that in my SDL2 code.  But merely having more digits there is not truly enough unless we have enough digits in the "diffs" between the external uniform temperature surface & the shutter frame values.  And that requires a temperature difference of something like 60-80 degrees Fahrenheit (double the difference between shutter & outside ambient).  I have not found a good way to get that yet.  If I use something out of the freezer it looks like device temperature gradients show up. 

[Berni]

I used the Linux driver that i got from Thermal Expert via email. For some reason they don't put the drivers on the website. Not quite sure what the deal is there, are the driver not supposed to be available to anyone?

One would think that a thermal camera appears like a webcam, but this is NOT the case, its a fully custom device with a custom driver.

Anyway the technical details of it are that the drivers come as an archive file contains a few *.h C headers and a large *.run file for 32 or 64bit. This run file is actually a shell script that does apt-get on a few USB libraries and then moves the large block of binary mess at the end of the run file into a .so file and copies it into a directory (I think in /etc/). This is a precompiled binary shared object (Equivalent of a DLL in windows). The easiest way to use this .so library is to write a C++ program and link it in and include the provided .h files. This expects you to have knowledge of C compilers and how to write makefiles. Additionally you might sometimes need to set the $LD_LIBRARY_PATH to the right place for things to work(I think the driver sometimes has problems finding the usb library).

Once you have all this you should be able to call the functions from the .h file to make the camera do things. The included doxygen documentation helps with what to do and shows some example code. Using the camera consists of a set of commands that finds the camera(I assume there can be multiple cameras connected) and then initializes it. This initialization part takes a few seconds due to for some reason reading the cameras internal flash really slowly. Once that is done you can call a function that gives you one frame of the image in the form of a float array. The values of the floats are the Celsius temperature for each pixel. For video you simply call this function as fast as possible (And it takes long enough to execute that get you the usual non military frame rate). The other two useful functions are the calibrate and dead pixel correction. You put the included lens cap on the camera and call calibrate to do a flat field calibration and then optionally call the dead pixel correction that maps out and hides any bad pixels. The correction will be included the next time you read a new frame. And that is pretty much everything the driver will let you do (Other functions don't do anything useful).

From here you could create a C++ wrapper app to make it work in Pyhron or <inset language of choice>, or develop the rest of your app in C++.

In my case i opted for doing my whole app in C++ and use OpenGL(using raylib) to draw directly to the screen (Doesn't care if there is X11 running or not, it just steals the frame buffer from it). My goal is to give it similar functionality as one of those standalone FLIR gun shaped thermal cameras plus a few features i think would be cool to have.

[IwuzBornanerd]

Thanks for that information; it should be helpful, but I'll let @kiranjayaraj speak for himself.

I'll have to look into that raylib you used.  I don't recall seeing that before.  SDL2 uses OpenGl also but on the PiZero even that took too long to write to the 3.5" LCD so I ended up using the videocore dispmanx functions to write to the frame buffer.  It also helped to put that in a separate thread since it uses the separate video core.  I don't recall if I tried putting the SDL2 rendering functions in a separate thread.  That might have helped if OpenGL via SDL2 uses the video core...

[Berni]

With hardware accelerated graphics you sometimes have to experiment a bit to find what works fast. The GPU has a certain way of doing things in a way it prefers and all the layers of software abstraction do too, but in general you usually want to send as few commands to the GPU as possible since the whole path has some overhead. So trying to directly manipulate pixels is usually the worst way.

The way i did it was able to run at 60fps (RPi seams to force Vsync so its capped to 60), this is way faster than needed for a thermal camera. My process was to use a for loop to turn the float temperature data into pixels via a color lookup table. The pixels ended up arranged in a bitmap in RAM and then this bitmap is sent over to the GPU to update a texture in GRAM so that the GPU has access to it, then it simply draws the texture to the screen. This could be potentially speed up even further by writing a shader script for the GPU to do the lookup by itself and we only give it a float array of temperature data, but that's significantly more work and the speed is not needed anyway(The CPU has plenty of time to do the lookup anyway).

In my app i still have to move the camera reading process into a separate thread as the Thermal Expert library blocks inside until the frame is done. Yet you have to call it quickly again to get it to start capturing a new frame as soon as possible. This means doing other stuff in between can jitter the cameras frame rate while at the same time leaving the app waiting in a blocked state for most of the time.

User interfaces in general tend to be a pain to make in just raw C++, but with OpenGL you can make some pretty impressive looking things that run very fast if you know what you are doing.

[richnormand]

Nice job IwuzBornanerd.
I'll give it a run tonight with my Seek Thermal.
Cheers.

[grizewald]

Thanks for posting this IwuzBornanerd!

I just recently bought a Seek Thermal XR and this code makes a great starting point for running the camera on a Pi 3 B+.

I have it working with the latest Stretch on the Pi and it delivers a solid 8 fps on the more powerful hardware. The first thing I'll be looking at is changing the screen so that I can actually use the buttons on the 3.5" touch screen. They're a bit small for fingers at the moment.  I've not used Python before, but then again, I've used so many languages over my career that another one won't hurt.

One language which is my forte is C. When I was tidying up the C module so that I could read it a bit better, a bug jumped out at me. In the shuttercal function, you have a test in two places where you examine each value in the diffs array to see if the program should use that value or the previous one. There's a mistake in that code where you use & and | instead of && and ||. Doing a bitwise operation when you mean to do a logical comparison isn't going to work very well and may explain some of the artefacts I saw under certain conditions.

The corrected code reads:

Code: [Select]

        if (((abs(diffs[ix]) < 6) && (noraster > 0)) ||
            ((badpxls[ix] < 10.5) && (abs(diffs[ix]) > 6) && (pixelcorr > 0)))

Further down in the function, in the clause bounded by "if (bwcolor > 0)", similar code appears, but I suspect this code doesn't do what you wanted it to. It examines a variable "diff" and that is always 1000, as per the initialisation at the start of the function. You might like to have a look at that part to see what is really should be testing!

I'm only just getting into exactly how the different frames work and how the corrections and calibrations are done. If I find anything else, I'll let you know. Once I have my modified version working how I want it, I'll post the changed source files here.

[IwuzBornanerd]

Thank you for finding && pointing that out.    But did you miss the instance right after the line "//calculate scaled (corrected) diffs"?  I think the error comes from having the code originally in Python which did not want the 2 &s.  Really, though, if the comparisons evaluate to 1 or 0 (Isn't that what true & false actually are?) is it any different (I'm no C expert)?  It does work quite well.  Toggle the z key and you see the "patent pixels" come & go.

The "if (bwcolor > 0)" case is defunct & I probably should have removed it since I quit using even before my original release.

Keep looking!  I have a vague memory of finding some sort of calculation error while working on my SDL2 code last year & I lost that code in December when I broke a uSD card.  And I have no idea what the error was in order to recover the correction.  The errors you found were still in that SDL2 code too so I did not find them after I pulled the C module into the all-C code.

[grizewald]

I seem to have corrected the single & in that line without even thinking about it!

You're right that it probably works either way, but it does make assumptions about the way the compiler handles the intermediate boolean values from the comparisons, which is always a bad idea. False is always zero, but true is "not zero", so what you get by bitwise combinations of logical comparisons may, or may not be what you expect.

Also, in the test: badpxls[ix] < 10.5, you're comparing an unsigned char (0 - 255) to a float, which isn't really useful as there will never be a fraction in the badpxls array. Changing it to < 11 or <= 10 would make more sense. That's the thing about C - you have to be aware of what types are being used and what the effects of any implicit type conversions might be when it comes to loss of precision or sign. Normally, the compiler will try to do something sensible when you mix types, but it's always best to handle types explicitly and avoid puzzling behaviour.

If you're using & to mean a logical and in Python, then there are similar problems in the Python code. In Python, logical and and or are the words themselves and bitwise and and or are single & and | characters.

That being said, I'm certainly getting some good images from the code! I'll keep you posted if I find any errors.

[IwuzBornanerd]

Also, in the test: badpxls[ix] < 10.5, you're comparing an unsigned char (0 - 255) to a float, which isn't really useful as there will never be a fraction in the badpxls array. Changing it to < 11 or <= 10 would make more sense.

I believe my thinking when I did that was as a way to avoid the = sign, but I don't know why I cared since that number is not all that critical.  It's only there to keep from having ridiculous or zero scaling values on pixels that are not actual image pixels (or maybe dead pixels).

If you're using & to mean a logical and in Python, then there are similar problems in the Python code. In Python, logical and and or are the words themselves and bitwise and and or are single & and | characters.

Now that is just plain funny!  You'd think there was a rudimentary tutorial somewhere in which I would have seen that but I don't remember one.  "And" of all things; whodathunk that?  If I try something and it gets the desired result I move on.  While I can write stuff that works I'm not knowledgeable enough (or perhaps even serious enough) about software to write real good "proper" code, which is why I keep typing disclaimers.  I don't recommend  it for non-nerds. 

And I DO appreciate your nerdy responses, @grizewald. 

[tmorar56]

I get link error when I try to make your code on my Raspberry Pi 3 B+. I think I installed the libraries, usb, X11, gl, etc... Could you tell me the Linux commands I need to use in order to solve the link errors. See attached output. Thank you very much for your help.

[IwuzBornanerd]

I get link error when I try to make your code on my Raspberry Pi 3 B+. I think I installed the libraries, usb, X11, gl, etc... Could you tell me the Linux commands I need to use in order to solve the link errors. See attached output. Thank you very much for your help.

To whom are you addressing your question?  The code I posted above is Python with a C module, no "main.cpp" and no function named ReadRawFrame, and nothing to make other than the c module which has no USB or display stuff in it.  And I don't see where anyone else posted code. 

But as for the errors, my best guess is that you need libusb version 1.0 or higher and the "-dev" packages for libusb, openGl & maybe xserver-xorg.  These contain header files & library files needed for compiling.  Try entering
    sudo apt-get install libusb-1.0-0-dev
and if it does not say already the newest version, then you probably don't have it installed at all. 

If the above clears out the USB errors then you should check the package  manager for the -dev versions of the GL & X packages.  I can't guess which ones you would need so you could just install all of them that you find.

[gtizz2]

Hello everyone,
about 10 days ago, I purchased a seek thermal XR for a personal project to automatic fever detection. I therefore need to get the thermal image and the temperature from my seekcam through my script in python. After several searches, I was able to find the software of IwuzBornanerd, a real godsend because, although it needs little improvement, it is very complete 
I downloaded the zip package and followed the instructions in the readme file. The result was the following:
(see attachement image named seek_beforeor click here https://drive.google.com/file/d/1By5krN2ZqLUzs0hL5dGUtv5CmPIX_Jc4/view?usp=sharing)
Really horrible.

However, sifting through the various threads of this blog I came to the discussion:
https://www.eevblog.com/forum/thermal-imaging/seek-compact-xr-lemon-or-not/25/
I applied the suggestions explained in the various messages including
Reply # 22, Reply # 24.
And setting the following values in the config:
ForC = C
ambientT = 20
pixelaT = 6191
diffshouldbe = -487

The result was this:
see attachement image named seek_after or click here https://drive.google.com/file/d/1MDYu_EH7C2gUGo_RqDzrHd6WuVAOPt5e/view?usp=sharing
FANTASTIC. The image clearly improved. 


But there is still some problem with the accuracy of the temperatures and the staggered pixel line.
Moreover, I would need to understand how to get temperatures. I managed to understand that they are present in the tamfx22 array. But I can't understand their content and how to get it if I wanted to take it from a specific point in the image. Could you help me?

Thanks in advance.

[IwuzBornanerd]

Okay, glad you got some improvement.  The thread you referenced is where I first learned that the new non-pro units had a new curve in frame9 and there are more changes related to that which I gave you in my PM.  But I see that there was an error in that info. so I'll re-post it here while also trying to accommodate both old and new cameras like I tried to do in that thread.  As I said in my PM, I don't have a new camera with which to test this code so you or someone else here will have to work with me and test it for me.

Please press the "L" key with an image on camera & post the pixellistSceneXXXX.txt file that the program writes so I can examine it.  It would be good if you can tell me what the min & max temperatures in the scene actually are also.

As I explained in my PM the tamfx22 array contains numbers representing the Temperature Above Minus Forty, times approximately 22 for Fahrenheit (thus the name of the array).  The elements of the array are one 16 bit value per pixel starting at upper left and going left-right, top-bottom.  The temperature at a given pixel is given by the value in that pixel number element as (at approx line 750 in py file):

Code: [Select]

pixeltemp = float(tamfx22[2*mousepixel] + 256*tamfx22[2*mousepixel +1])/tscale - 40
Edited 8-1-2020 to remove code containing errors so as to avoid confusion for future readers.

Also, if you want to detect fever you probably should review this thread:
https://www.eevblog.com/forum/thermal-imaging/virus-outbreak-in-wuhan-chine-thermal-cameras-likely-to-be-deployed-again/

[gtizz2]

Hello, IwuzBornanerd
sorry for for the late reply, I have been busy with work.
Anyway..  I restored the project in original state (I downloaded the software from this post), and in primary step I launched the software for 10/20 min (some time the software closed itself without messages) e after 20min I pressed "L" key.  (in attachment there is a zip file named: "original-pre-cal" contains a file pixellistScenexxx.txt and an image of screen. (very bad quality)

Subsequently, I had clicked on "Cal" for calibrate the camera and after substitute calfile to pixelcal and newrefile to refframe, I have press the "L" key. In attachment there is a zip file named "original-post-cal" with the pixellistScenexxx.txt and an image of screen. (very bad quality)


After that, I had apply the modify that you asked. But the software go in infinite loop because in the condition of "while" in 554 line code Under "if status = 9" there isn’t a increment of index variable and I didn’t know what is the increment statement. Precisely:

“
while index < 16430:
                if( (diffcurve[index][0] + 256*diffcurve[index][1]) < 1 ):
                  diffcurve[index][0] = diffcurve[index-1][0]
                  diffcurve[index][1] = diffcurve[index-1][1]
“
So I wait the statement missed.

Moreover, I have increment the time in label_image.after(1, show_frame) and I notated that the temperature decrease. Why this behavior?

PS: my seek cam is a Seek Compact XR.

Thank you  in advance.

[IwuzBornanerd]

Edited 8-1-2020 to remove my apparently failed attempt to correct & clarify the code I deleted above...And then I posted my revised program file in reply #26.

Thank you for the files; I am still studying them, but posted the above so you can see if it helps any of the holes while I look at the files.  The delay in your response allowed me to start re-thinking how to figure the temperatures, but as time wore on I stopped thinking about that and moved on...

BUT ALSO, it  looks like your camera is rather ill.  It appears to have many dead pixels & for some reason the bottom row of the shutter frame is all zeros but the image frame is not.  I post an image showing the frames from your camera on the left vs. one of my cameras on the right.



What you see on the left top is a bunch of randomly located white pixels and on the right is a pattern of "patent pixels" and a few dead pixels.  The original sensor had every 15th pixel blocked out or dead by design in order to avoid infringing a FLIR patent.  Judging by the data from your camera those pixels are gone, which will make many people happy.

I don't understand why the "raw image frame" in the second row looks skewed, but the resultant image is okay so I'll ignore that for now.

Have you run your camera with the Seek app?  And if so, did you get good images?

[IwuzBornanerd]

I was staring at that rasterized image & looking at the code & it dawned on me why the image pixel list results in that rasterized image!  In the code segment below (starts about line 720), change all instances of "ret9" to "stripbfr".  ret9 does not have the extraneous columns removed from the image data!

Code: [Select]

            if( (listfile >0)  & (firstcalframe > 0) ): #Again, try to get the first frame after cal frame
               s = "pixellistScene" + str(nimage) + ".txt"
               f2 = open(s, 'w')
               s = "pxl#\tcal\timage\tdiff\tcorrdiff\tTAMF" + "\n"
               f2.write(s)
               ix = 0
               while ix < imgw*imgh:
                 s = str(ix) + "\t" + str( (calbfr[2*ix+1]<<8) + calbfr[2*ix]) +"\t"  +str( (ret9[2*ix+1]<<8) + ret9[2*ix]) + "\t" + str( ( (ret9[2*ix+1]<<8) + ret9[2*ix]) -( (calbfr[2*ix+1]<<8) + calbfr[2*ix]) ) + "\t" + str( diffs[ix] ) + "\t" + str(tamfx22[2*ix] + 256*tamfx22[2*ix +1]) +"\n"
                 f2.write(s)
                 ix=ix+1
               f2.close()
               firstcalframe = 0
               listfile = -1  #only write one file

That will make the image list look more like the shutter list.  I just tested it with my camera. 

You can post a new list file after making that change if you like.

[josmera]

Hi gtizz2 and IwuzBornanerd, I have the same project as you gtizz2 fever detection with seek and raspi. But I'm stuck, I followed the read-me as well but I have a lot of issues, with libraries and finally with python 3 and some changes can import the libraries, but now I have a issue with pixelmathA library I have this message when I can run the code:

pi@raspberrypi:~/Desktop/Rpi2SeekRevA $ python3 Rpi2SeekA.py
Traceback (most recent call last):
  File "Rpi2SeekA.py", line 48, in <module>
    import pixelmathA
ImportError: /usr/local/lib/python3.7/dist-packages/pixelmathA.cpython-37m-arm-linux-gnueabihf.so: undefined symbol: Py_InitModule4

If you can send any kind help I will be grateful with you for the rest of my life. Thanks a lot, and sorry for my mistake.

[gtizz2]

Hello josmera,
I also have started with python 3.x and I had your same problem. After a little research, I found this post:
https://www.eevblog.com/forum/thermal-imaging/seek-thermal-camera-with-raspberry-pi/
(Reply #17 and Reply #18) that refers to this solution:
https://stackoverflow.com/questions/28305731/compiler-cant-find-py-initmodule-is-it-deprecated-and-if-so-what-should-i

I have tried this modify but without success, maybe I done some mistakes. So, if you have success, let me know please.

[gtizz2]

Hello IwuzBornanerd,
I assume that increment is index=index+1 right?
And…
that code will replace all the code under “if status = 9” (so, ambientIndex, sensrref, curvbase go away) or only the part where there is “while index < 11316”?
Anyway, I have replaced only part where there is while index and increase index=index+1.


In attachements you will find:
- pixelSceneXXX with changes of you last messages  (named: pixellistScene180-fix.txt);
- image of your software with your fix (frame-fix.png);
- pixelSceneXXX with changes in my initial message where the image is enough clear (named: pixellistScene303-myChanges.txt);
- image obtained with official seek android app (named: seek_android_app.jpeg).

Finally, the question about the decrease of framerate decrease the temperature?
Thank you

[IwuzBornanerd]

Also modified 8-1-2020 to remove my apparently failed attempts to clarify requested changes.

Finally, the question about the decrease of framerate decrease the temperature?
Thank you

Makes no sense to me.

Was there supposed to be a hand or other recognizable object in the images for those pixel lists?

Have you made other changes to the py file or the C module?

[IwuzBornanerd]

... I have a lot of issues, with libraries and finally with python 3 and some changes can import the libraries, but now I have a issue with pixelmathA library I have this message when I can run the code:

The original code I started with from fry-kun on github said "[Python] 3+ may work".  I never tried it with Python 3 and certainly have not tried it with my expansions to fry-kun's code.  I am not a Python expert.  I hate Python.  It was extremely frustrating to use (perhaps mostly due to pathetic documentation).  Is there a reason you need to use Python 3?  Python 2.7 is included in Raspbian Buster.  I have now removed the "3+ may work" from my source file.

Pardon me, I need some sleep badly.