Flir E4 Thermal imaging camera teardown

[Rasz]

and fraud if one claims their hardware performs at E8 levels when it's a hacked E4

Wait what?

Doing some perusing and stumbled on this...How did you test the frame rate?

One way to measure would be to view a vane on a variable speed motor and see at what RPM you get a stroboscopic "freeze"
Or maybe even a sig-gen driving a thin wire or small resistor to create a fluctuating heat source

You said in the teardown that fpga stream is 50(or 60 dont remember) Hz. That can only mean throttling happens inside the app on the SoC.
I assume flir was smart about it and uses adhering to 9Hz limit as a pretext/excuse to bump sensitivity with frame averaging.
I dont know Windows ce at all, is there something like Device manager there to look up if standard windows DirectShow capture API is exposed? List loaded drivers to see if some camera/capture driver is loaded?
Google says M$ has
 Wince600\private\test\multimedia\directx\dshow\camera\cameraapp
for testing capture under wince, not that this program will be very useful (needs a mouse to operate)
One way might be killing main flir app and loading something that would capture using windows api.

Would it be smart for flir to include an option for disabling 9hz limit at all in the software? One would imagine their lawyers made it clear to developers to never ever do it, just like patent lawyers hamper development by prohibiting designers from looking up solutions in existing patents. Too much liability.

[mikeselectricstuff]

and fraud if one claims their hardware performs at E8 levels when it's a hacked E4

Wait what?

Doing some perusing and stumbled on this...How did you test the frame rate?

One way to measure would be to view a vane on a variable speed motor and see at what RPM you get a stroboscopic "freeze"
Or maybe even a sig-gen driving a thin wire or small resistor to create a fluctuating heat source

You said in the teardown that fpga stream is 50(or 60 dont remember) Hz. That can only mean throttling happens inside the app on the SoC.

No - it is most likely done in the FPGA. There may or may not be a way to control the downsampling rate. We don't know.

I assume flir was smart about it and uses adhering to 9Hz limit as a pretext/excuse to bump sensitivity with frame averaging.

9Hz is all about ITAR & other export regs. However I think it's highly likely they use multiple frame averaging to increase signal-to-noise ratio to allow use of a smaller lens. The lenses on the 60fps Ex0 range look significantly bigger

[mrflibble]

You said in the teardown that fpga stream is 50(or 60 dont remember) Hz. That can only mean throttling happens inside the app on the SoC.

The fpga gets a 60 Hz stream with ADC readings from the sensor. The fpga then does funky stuff such as image processing, and if it feels like it decimating frames. The frames then go over a totally different interface (camera interface) from fpga to the i.MX257.

So the fpga is fully capable of throttling things well before the application in Win CE gets even close to the data. Unfortunately.

[Rasz]

No - it is most likely done in the FPGA. There may or may not be a way to control the downsampling rate. We don't know.

sorry, bad memory + to lazy to rewatch
ok i think i remember now, you couldnt find exposed fpga-soc connection (multi layer board).

that leaves checking out wince standard capture api on the table

[Taucher]

If you look closer you will see that this function returns a value 1/0 depending on the printed value.
Then go look where that function is called and you will notice that depending on that value, it will write either 9 or 30 to a variable, which is then used to trigger a write to a FPGA register. The patch just makes the function return a constant rather that the real result.
Cheers, Sylvain

nice once *kudos*

Just did another 240FPS slomo-measurement: clearly some 100ms between frame updates
I think it could be fun to take a look at rset .image.contadj.frequency  ... which will reject anything >10 ... second check?
Well... at least .image.framegrab.swburst.frequency looks nicely with that hex

It's interesting how many times one will find 0x1E and its doubled value 0x3C in direct neighbourhood... 00111E34, 00117264 ...
... heh, I really need to get into ASM again... my guess is your ASM beats mine by far

[amyk]

Would it be smart for flir to include an option for disabling 9hz limit at all in the software? One would imagine their lawyers made it clear to developers to never ever do it, just like patent lawyers hamper development by prohibiting designers from looking up solutions in existing patents. Too much liability.

I'm not familiar with what ITAR says but it's possible that there has to be a level of difficulty to it, i.e. changing the framerate is not a documented easily accessible user option.
Even Flir has some interesting wording on their website about it (from http://www.flir.com/cvs/cores/knowledgebase/index.cfm?CFTREEITEMKEY=326&view=35781 ) :

Is an export license required for Tau and Quark thermal imaging cameras?
Note:  This answer also applies to the legacy Photon cameras produced by FLIR.
Yes, except if the video frame rate is factory set to be less than 9 frames per second. We refer to this configuration as "slow video".

To me, that almost sounds like "we set it to 9fps just so you can export them without license",

[Pinkus]

If you look closer you will see that this function returns a value 1/0 depending on the printed value.
Then go look where that function is called and you will notice that depending on that value, it will write either 9 or 30 to a variable, which is then used to trigger a write to a FPGA register. The patch just makes the function return a constant rather that the real result.
Cheers, Sylvain

nice once *kudos*

Just did another 240FPS slomo-measurement: clearly some 100ms between frame updates
I think it could be fun to take a look at rset .image.contadj.frequency  ... which will reject anything >10 ... second check?
Well... at least .image.framegrab.swburst.frequency looks nicely with that hex

It's interesting how many times one will find 0x1E and its doubled value 0x3C in direct neighbourhood... 00111E34, 00117264 ...
... heh, I really need to get into ASM again... my guess is your ASM beats mine by far

Maybe it is a language thing, but I am confused right now:
I thought, modifing the 4 bytes at 0x001016ec from "05 00 a0 e1" to "01 00 a0 e3" will enable 30Hz but in your posting above you say 100ms between each frame = ~9 Hz. Did I miss something or did I understood something wrong? So no 30Hz by changing appcore.exe?

[mikeselectricstuff]

Is an export license required for Tau and Quark thermal imaging cameras?
Note:  This answer also applies to the legacy Photon cameras produced by FLIR.
Yes, except if the video frame rate is factory set to be less than 9 frames per second. We refer to this configuration as "slow video".

To me, that almost sounds like "we set it to 9fps just so you can export them without license",

Yes, exactly.
Export licenses are expensive and time-consuming, so the need for one significantly adds to end-user cost.

E2V developed a product specifically around non-US parts to make it outside the scope of ITAR regs

e2v has launched a new range of Argus® Thermal Imaging Cameras, designed specifically for security and defence markets. The cameras feature the latest thermal imaging technology and are manufactured using European sourced components, making them available to the wider export market and not subject to the USA International Traffic in Arms Regulations (ITAR) controls (UK export controls applies).

[Taucher]

Maybe it is a language thing, but I am confused right now:
I thought, modifing the 4 bytes at 0x001016ec from "05 00 a0 e1" to "01 00 a0 e3" will enable 30Hz but in your posting above you say 100ms between each frame = ~9 Hz. Did I miss something or did I understood something wrong? So no 30Hz by changing appcore.exe?

That hex will enable something and rls will report 30 for some values that were formerly at 9. but nontheless the effective framerate will stay under 10Hz.
Anyway the Ex is quick enough so one does not have to wait too long for a new image ... I don't have a big urge to investigate that further

[tinhead]

Reprogramming the flash is not so simple due to the issue of bad-block mapping and ECC - a straight image from one unit would almost certaonly not work on another. It would need to be read & parsed with the knowledge of how WinCE manages the flash, and then reconstructed.

this is why dump with OOB bloks must be made, you can always restore such dump (as long your flash have no bad blocks, if there any you have to enable relocation).

Easiest way would probably be to connect it up & mount on another WinCE system.

oh well, this is always an option

[mikeselectricstuff]

Reprogramming the flash is not so simple due to the issue of bad-block mapping and ECC - a straight image from one unit would almost certaonly not work on another. It would need to be read & parsed with the knowledge of how WinCE manages the flash, and then reconstructed.

this is why dump with OOB bloks must be made, you can always restore such dump (as long your flash have no bad blocks, if there any you have to enable relocation).

In my limited experience of driving NAND flash at low level, there are always some bad blocks.

[ViciousPest]

Reprogramming the flash is not so simple due to the issue of bad-block mapping and ECC - a straight image from one unit would almost certaonly not work on another. It would need to be read & parsed with the knowledge of how WinCE manages the flash, and then reconstructed.

this is why dump with OOB bloks must be made, you can always restore such dump (as long your flash have no bad blocks, if there any you have to enable relocation).

In my limited experience of driving NAND flash at low level, there are always some bad blocks.

Might be a irrelevant. Does anybody know if this is managed or unmanaged NAND? Datasheet link?

[JockeT]

Looks a lot like dust on the sensor.

Bit out of my depth here because I probably know a lot less about TICs than most people on here but could this be proved by flicking the lens cover to closed  and letting the sensor reach equilibrium?

Would the spot then disappear? (or at least fade a bit?)

Yep, this kind of worked. At least well enough to convince me to take it apart and clean the sensor.
Spot is now completely gone. I marked the lens in an inconspicious spot to be able to get infinity focus back easily, but it really wasn't that simple despite the effort. It seems moving the mark by less then a millimeter has a pretty significant impact on focus, it took me a couple of rounds before i was satisfied. The close focus capability was really neat though, i will definitely have to make a focus adjustment tool.

Anyway, thanks to you and mike for the aid in diagnosis.

[Pinkus]

Anyway the Ex is quick enough so one does not have to wait too long for a new image ... I don't have a big urge to investigate that further

That's true - I am not complaining about the frame rate. Moving through the menus could be a bit faster if you ask me (though I have the feeling it is slower now after the menu-hack), thats why I was asking.

[mamalala]

Anyway the Ex is quick enough so one does not have to wait too long for a new image ... I don't have a big urge to investigate that further

That's true - I am not complaining about the frame rate. Moving through the menus could be a bit faster if you ask me (though I have the feeling it is slower now after the menu-hack), thats why I was asking.

Just a random thought: Could it be possible that there is some kind of "global framerate" thing going on, which makes the UI slower as well in the 9fps units? Does anyone know if it is that sluggish on a 30fps unit as well?

The few embedded QT things that i know are fairly responsive, that's why i'm wondering why it is so slow on these units. Something like a throttle in the QT message/event loop maybe.

Greetings,

Chris

[Taucher]

Anyway the Ex is quick enough so one does not have to wait too long for a new image ... I don't have a big urge to investigate that further

That's true - I am not complaining about the frame rate. Moving through the menus could be a bit faster if you ask me (though I have the feeling it is slower now after the menu-hack), thats why I was asking.

That can actually depend on active measurement options - especially the delta modes appear to me to be implemented a bit below optimum performance - if the measurement frequency was increased by rset, then that aspect will also reduce responsiveness... I'm pretty happy with 5fps there.

All I did was re-add/enable some entries for the functions that were already compiled into the application

EDIT/PS: of cause there's always the option to revert back to original state and compare by measurement

[Taucher]

Just a random thought: Could it be possible that there is some kind of "global framerate" thing going on, which makes the UI slower as well in the 9fps units? Does anyone know if it is that sluggish on a 30fps unit as well?

The few embedded QT things that i know are fairly responsive, that's why i'm wondering why it is so slow on these units. Something like a throttle in the QT message/event loop maybe.

It's a bit more complicated...
Here a rough estimation how that thing works: the sensor delivers raw data to the FPGA, which pre-processes it according to it's .bin programming file - the raw image data is beeing corrected for all sorts of distortions, errors, calibration etc... if required re-scaled and noised ... and then copied into an overlay memory-segment where the display can present it (either directly or via CPU). In addition there are some other features getting overlayed - like the OSD, temperatures etc.

So to get the optimum performance, all components in the chain have to work perfectly:
- sensor(s): capture at high rate [check]
- fpga: churn out (composit) images at hight rate [unknown]
- cpu: process images at high rate [maybe/uknown]
- display: refresh often enough not to skip frames and not to be faster than own optical reaction time

I somehow doubt that every image is beeing piped through QT - I think that's used just for the UI and the sluggishs, scripted menu overlays - hence my bet would be on that fpga.bin

[mikeselectricstuff]

Just a random thought: Could it be possible that there is some kind of "global framerate" thing going on, which makes the UI slower as well in the 9fps units? Does anyone know if it is that sluggish on a 30fps unit as well?

The few embedded QT things that i know are fairly responsive, that's why i'm wondering why it is so slow on these units. Something like a throttle in the QT message/event loop maybe.

It's a bit more complicated...
Here a rough estimation how that thing works: the sensor delivers raw data to the FPGA, which pre-processes it according to it's .bin programming file - the raw image data is beeing corrected for all sorts of distortions, errors, calibration etc... if required re-scaled and noised ... and then copied into an overlay memory-segment where the display can present it (either directly or via CPU). In addition there are some other features getting overlayed - like the OSD, temperatures etc.

So to get the optimum performance, all components in the chain have to work perfectly:
- sensor(s): capture at high rate [check]
- fpga: churn out (composit) images at hight rate [unknown]
- cpu: process images at high rate [maybe/uknown]
- display: refresh often enough not to skip frames and not to be faster than own optical reaction time

I somehow doubt that every image is beeing piped through QT - I think that's used just for the UI and the sluggishs, scripted menu overlays - hence my bet would be on that fpga.bin

The image is clearly not being processed at all by the UI as it appears before winCE has booted. There may be somelow-level  code handling image data but I'd expect most of this to be done by DMA hardware. bear in mind a SoC with a camera interface will be designed to handle much higher rates than 320x240x9fps, so it is reasonable to assume it can deal with most of it entirely in hardware.
 I would expect the palette data is sent by the application code to the FPGA to do the colour mapping. I think the FPGA probably also handles the merging of the visible camera as the cam is connected to the FPGA
All the UI will be doing is overlying the onscreen menus and icons. ISTR seeing that the Soc chip has some hardware support for overlays.
 

[Fraser]

An Apology.

I would like to apologise to the members of this thread who were upset or inconvenienced by the bulk deletion of my posts. Sometimes you have to do as you are told and this was such an occasion.
I now know where my 'line' is drawn so, no future deletions should occur.
P.S I regularly edit my postings as I am a bad speller 

I am not able to post anything technical in this specific thread but then, as some said, I added nothing to the mix anyway 

I can advise that I did not delete the UK supplier recommendation as that would have been unkind.
My PASS order arrived this morning with 1.19.8. and a free soft case.

UPDATE: S/N 6390 65xx

Life goes on, and I live to fight another day.



Now back to your regular programming.........

[bookaboo]

No need to apologise.
From anything I read you were only being helpful, we all know how it can be when employers etc. don't want certain things discussed.

[mamalala]

Sorry, i didn't express myself clearly, i think.

No, i'm not talking about the image processing or somesuch done through QT, or the thermal image being piped through it. What i meant was some kind of callback that will update the display, and which then would also call the QT sheduler to process the UI elements, causing the UI to become sluggish since it would be updated at the same framerate then, instead of being free-running.

Like some display/UI process that sleeps for 100ms, then transfers the thermal image into the display buffer, and then calls the UI, having it overlay the result, repeating that over and over again. QT allows the app to be run in it's own event loop, or to call the event handling manually in your own loop.

I'm simply wondering why it is so sluggish. Even though it builds UI stuff dynamically, it shouldn't be that friggin slow. Unless it is very badly written/implemented, that is. But then, my experiencee is on embedded Linux platforms only, maybe WinCE has some (negative) influence there as well.

Hope this is clearer now. And of course this is all speculation.

Greetings,

Chris

[Frost]

Today I have received my cam, it was on backorder for a week now.
If I noticed the calibration Date 08.11.2013 (last friday),
on the included calibration letter, it was a shock.

The supply chain must be extreamly short at the moment, so it seems
the cam was sent out direct from the production facility in estonia.

But the firmware version is 1.19.8, so I prepared a patch file
installed the tool software and tried the to upload the file into the cam.
It worked like a charm the picture quality is superb now.

So many thanks to all the people who made this possible,
especially to Mike and Taucher.

I am really happy to own such a great tool (for men ) now.

[gotvolts]

TL;DR :
12 Nov 2013 - There is a single second-hand report of a new firmware update 1.20.x that prevents the hack. This has not been confirmed either way and we do not know if this version is in the distribution chain, as it was reporteldy found on a unit returned after being sent for repair. If you're considering ordering, you may want to either do it soon having confirmed with the supplier that firmware version is 1.19.8 or earlier, or hang on a few days, as several pepple here have units on order  so things should become clearer soon. One user has received a camera showing as calibrated on Nov 9th with 1.19.8, and there have yet to be any more indications of a later version in the wild.

Do you know if it is possible to update your firmware to an earlier version if you have purchased a unit with the updated firmware?  If thats the case, then is the earlier firmware even available or can be downloaded from a working unit?

[mikeselectricstuff]

TL;DR :
12 Nov 2013 - There is a single second-hand report of a new firmware update 1.20.x that prevents the hack. This has not been confirmed either way and we do not know if this version is in the distribution chain, as it was reporteldy found on a unit returned after being sent for repair. If you're considering ordering, you may want to either do it soon having confirmed with the supplier that firmware version is 1.19.8 or earlier, or hang on a few days, as several pepple here have units on order  so things should become clearer soon. One user has received a camera showing as calibrated on Nov 9th with 1.19.8, and there have yet to be any more indications of a later version in the wild.

Do you know if it is possible to update your firmware to an earlier version if you have purchased a unit with the updated firmware?  If thats the case, then is the earlier firmware even available or can be downloaded from a working unit?

The only firmware which is "available" as a proper upgrade package is 1.18.7 It is not known whether there are any issues downgrading. It may be possible to build a 1.19.8 upgrade package from a unit with this version, but I'd consider it risky and not worth doing as we are not aware of any benefit of this version. Remember that if something happens that crashes the boot process before applauch.dat, this could easily lock out all currently known routes back in to fix things. 
There is a potential fall-back mechanism via the WinCE bootloader, but it is not known whether or not we have sufficient information to be able to recover a unit via this path. Any comments form WinCE experts would be welcome on this subject.
 (BTW has anyone checked if they fixed the no time/date on file information issue in 1.19.8?) 
If a later version appears and if it resists other current lines of attack, then atttempting to downgrade would be worth a try. However if the purpose of an update is to prevent hacking, it is highly likely they would have made a change to the FW update process at the camera end to prevent this. As the FW update process appears to use an unusual/nonstandard route through the USB system, it would be relatively easy for them to change without causing them other problems in production etc.. They would probably also need to change the PC update tool, but they'd only need to release this if & when they released a further update significant enough that users would want/need to do a field update to.

[JockeT]

No time or date on my 1.19.8 unit.
Only Emissivity, Reflected temperature, Thermal resolution, Digital resolution, and lens field of view.