FLIR Ex chassis ETS320 - interesting 'fault' . May apply to E4 E5 E6 E8

[Fraser]

I have seen an interesting 'fault' on my FLIR Ex chassis based ETS320 and thought I would share it here in case anyone else sees similar on their camera. The 'Ex' chassis is used in the E4, E5, E6, E8 and ETS320.

Some quick background on my ETS320 for those who have not seen my ETS320 improvements thread...

I purchased a partially dismantled ETS320 that was sold as 'Spares or Repair' on eBay. The chassis module of the unit had been removed for reasons unknown and due to the crazy design of the unit, it could not be re-assembled by the owner. It was disposed of as working  it dismantled. It was also described as new which I found interesting. The unit was manufactured in March 2017 and is running firmware 3.5 on Hardware 2.0L. It is basically a 2017 Ex camera chassis placed into a 'microscope like' housing for PCB inspection work.

I intend to upgrade the firmware to 3.9 but want to leave that until I am certain all is well with the unit. (Upgrading a faulty units firmware can be risky if it crashes mid update !)

During initial testing of the ETS320 I noted no unusual behaviour, but I was only carrying out cold boots and not a boot from standby. The ETS320 has two 'off' states. With a short press of the power button it goes into a sleep state and automatically goes to 'full off' after 48 hours of non use. A long press of the power button initiates a 'full off' condition. In the sleep state, a press of the power button returns the unit to full operation in a couple of seconds without a boot sequence visible. When in the 'full off' state, a press of the power button starts the cold boot sequence and takes the usual 20 seconds or so to boot.

I have not been using the ETS320 much as it is still in pieces awaiting my upgrades. It has been charged and used for a few minutes total run time.

During some further testing I set the unit to sleep and then woke it up again. I was surprised to see what looked like a Star field of twinkling pixels on the display ! I recognised the pattern as likely the dead pixels on the microbolometer. I initiated another sleep, followed by another wake up and the 'Star field' remained. A full reboot from the 'full off' state returned the camera to normal operation with not a single dead pixel visible. Tests proved the unit to be working perfectly. I once again initiated the sleep mode and then woke the camera. The 'Star field' returned ! Further testing showed the 'fault' to be repeatable but sometimes the camera would come out of sleep with a normal display. The length of time sat in the sleep state seemed to effect whether the 'Star field' appeared or not. Interesting once the 'Star Field appeared it could not be eradicated by warm boots. Only a cold boot provided a normal display. A cold boot refreshes all calibration and dead pixel data held in volatile memory.

Here is a précis of my thought processes.....

1. Camera boots from cold without issues. Part of the cold boot is reading of the Dead Pixel map and NUC tables.

2. After a cold boot the camera operates perfectly. Time and date present and accurate in RTC.

3. Camera enters sleep state from which it can awake in seconds. The processing systems within the camera must enter a low power state and maybe contune to retain the Dead Pixel Map and NUC data in powered memory, ready for use upon awaking.

4. When camera awakes from a short sleep, all is normal. When camera awakes from a longer sleep, the display shows a 'Star field' of twinkling pixels. This suggests a time related degradation or loss of valid data.

5. The RTC maintains correct data and time even when the 'Star Field' appears.

6. Sometimes significant corruption of the thermal display occurs instead of the 'Star field'. This suggests the possibility of a powered memory device holding image related data becoming corrupted.

7. The RTC Lithium cell voltage reads 3V3 which appears healthy. The cell is rechargeable but it is not known hen charge is applied to it. The camera may need to be in the on state to supply a charge to the cell, much like some laptops that use similar RTC power supplies.

8. There may be a memory area that holds key calibration and image data whilst the camera is in sleep mode. When cold booted the camera fills that memory area with fresh, valid data. If the memory is volatile it needs a power supply whilst the camera is sleeping. Could that power supply be inadequate for some reason and causing the corruption or loss of the data it is holding ?

9. Where could the volatile memory power come from ? There are two sources in the camera, the Main Ali-Ion cell, and the tiny rechargeable Lithium RTC cell. Which provides the power to the memory, if that is indeed the case, is not yet known. I have previously seen RTC cells powering configuration memory and the memory can become corrupted at a voltage higher than the RTC requires to maintain time and date.


The camera has been left running for several hours today in an effort to fullybchsrge both Lithium cells. I will carry out further testing with the camera self powered and powered from its external power supply. The results may reveal whether there is an issue with the cameras internal Lithium cells, or that the problem has been resolved by the long charge period charging the tiny Lithium RTC cell.

An interesting little 'fault' but not one that concerns me. I tend to cold boot my cameras so a warm boot issue is no great drama. I will try to identify the cause however.

Sorry, no pictures yet as I did not bother to take any to date. If the 'Star field' returns I will capture an image of it for others to use as a comparison if they suffer the same issue.

Fraser

[Fraser]

I forgot to say, this could be a firmware issue ! I have established that the cameras processing system is stable from a cold boot so I may upgrade to firmware 3.9 shortly. I would like to see whether the long charging period has removed the warm start issue though. I can also fit a new Lithium RTC cell if that is suspect as thankfully it is a common FDK ML621 rechargeable button cell in a PCB holder rather than soldered to the PCB.

If the problem has gone away, I am considering removing the Lithium RTC cell whilst the camera is in sleep mode to see whether it causes the problem again.

Fraser

[Fraser]

Result of early days testing.....

Test to determine approximately how long camera can sit in sleep mode before 'Star field' appears

Sequential tests -

Round 1

30 Seconds - OK
60 Seconds - OK
90 Seconds - OK
120 Seconds - FAIL - Star field present

Round 2

30 Seconds - FAIL - Star field present

Round 3

30 Seconds - FAIL - Star field present

Round 4

5 Seconds - FAIL - Star field present


So from this crude first test it appears to me that after approx 90 Seconds in standby something starts to 'lose its mind' and it does not recover its 90 Seconds capability after a cold boot. It has the appearance of a slow charging energy storage device that has lost most of its capacity and effectively become a capacitor with only 90 seconds of useful run time. That tiny Lithium cell is looking more guilty by the minute !

We shall see.

Fraser

[Fraser]

New Sony ML621 cell ordered. Even if not needed, it is a good spare part to hold 

Fraser

[Fraser]

Another round of testing completed. It created more questions than answers though.

First I shall upload some pictures captured on the unit showing the issue. As can be seen, the 'Star field' effect varies considerably !
This is ONLY occurs when the camera comes out of sleep mode. It is eradicate by a cold boot and the effect is not present at every warm boot. More on that in a minute. First the pictures.

Fraser

[Fraser]

OK, to the test results......

I spent a bit of time today repeatedly alternating the unit between sleep and on modes. At first I saw some of the typical 'Star field' images. Some were really 'busy' way more than would be expected for a missing Dead Pixel map ! The first two attached images (above) were what I thought was evidence of Dead Pixel Map loss, or corruption. The last three are something far more weird, such as corruption of the dead pixel map rather than loss of it.

A then carried out a formal test after having left the unit to rest for 30 minutes. the results follow:

Interval between Entering Sleep mode and being awoken = 15 Seconds
Test repeated 24 times with 30 seconds delay between each sleep/wake cycle
Result = No issues present.

Interval between Entering Sleep mode and being awoken = 30 Seconds
Test repeated 5 times with 30 seconds delay between each sleep/wake cycle
Result = No issues present.

Interval between Entering Sleep mode and being awoken = 60 Seconds
Test repeated 4 times with 30 seconds delay between each sleep/wake cycle
Result = No issues present.

Interval between Entering Sleep mode and being awoken = 100 Seconds
Test repeated 2 times with 30 seconds delay between each sleep/wake cycle
Result = No issues present.

Interval between Entering Sleep mode and being awoken = 110 Seconds
Test repeated 2 times with 30 seconds delay between each sleep/wake cycle
Result = No issues present.

Interval between Entering Sleep mode and being awoken = 120 Seconds
Test repeated 2 times with 30 seconds delay between each sleep/wake cycle
Result = No issues present.

The test continued with single tests at 135 Seconds, 150 Seconds, 165 Seconds, 180 Seconds, 240 Seconds, 300 Seconds. A final test was done at 15 minutes. None of the tests showed any signs of the 'Star field' effect or any other issues.

The whole of the above test was one continuous testing sequence from a single cold boot at the beginning. I could not make the unit show the Star field no matter how I tried.

In the end, I gave up and decided to install a nice fresh copy of Firmware 3.9 in case there is some weird stability issue with the version 3.5 that was loaded on the ETS320. I am not a fan of reaching for a firmware update before ensuring that a hardware fault is not present, but on this occasion I broke my own rule as I was always going to install version 3.9 and the actual processing engine of the camera appeared stable enough to not crash during the update process.

The update to Firmware version 3.9 went smoothly and without issues. I am now in the good position of knowing that the camera contains a nice fresh firmware and any instability has hopefully been remedied in the new firmware version. 

More on this matter later when I have carried out more testing 

All good fun 

Fraser

[tomas123]

I don't think, that is a missing bad pixel mapping after a soft start.

All images are different...

Looks like a missing FFC after a soft start.

Code: [Select]

exiftool -b -RawThermalImage FLIR0001.jpg | convert -define png:swap-bytes=on - -auto-level FLIR0001.png

[Fraser]

Hi tomas123,

Many thanks. Agreed, it is not dead pixels.  It initially looked like around 50 'dead pixels' , but it soon became clear that something more complex was happening. A missing FFC table would be corrected at the next FFC event though. Such did not occur and the FFC events did not change the pixel pattern.

As soon as the camera comes out of sleep, it initiates an FFC event. That was happening so at least the flag was not jammed.

The camera is now behaving itself perfectly with absolutely no repeat of the 'star field' image. This is a 'fault' that I am at a loss to explain. As stated, a cold starts never showed the 'star field', only a warm start from sleep mode. I have been testing the camera all day today and nothing unusual has occurred. A pity as I have no explanation for this weird effect. It appeared to clear just before I decided to carry out the firmware update so I cannot declare the update to be the solution.

I shall continue to test the camera regularly as it remains disassembled awaiting lens holder development.

Fraser

[tomas123]

Do I understand you correctly that the pixel error does not occur after the firmware update? 

[Fraser]

Hi Tomas123,

Well it is a bit more complicated than that. The camera was suffering regular occurrences of the 'star field' problem only when coming out of sleep mode. I left the unit powered and running for several hours in case the issue related to a discharged memory back up cell. These 5mAh cells take a tiny charge current so can take time to charge. I then tested the camera again and the same 'star field' symptom appeared. I thought that disproved my idea of a discharged memory battery. I left the unit running for an hour and tested it again, as detailed above. It performed flawlessly over an extensive test period and many, many test cycles. I could not explain this. It was tested from a single cold boot though. I began to wonder whether the issue occurs as the result of something happening during some cold boots, but not others. That then set the scene for problems with the sleep mode ?  Just blue sky thinking really.

I decided to update the firmware from 3.5 to 3.9 in case I was seeing some instability in the firmware that caused issues when coming out of sleep mode. I have seen such issues with sleep and hibernate modes on some Windows based PC's. It was a shot in the dark and I have no proof that it permanently solved the issue.

I began suspecting the firmware because a physical fault with the camera appeared unlikely in view of its ability to operate normally from a cold boot, with no symptoms of a fault. as you have stated, I began to discount a corrupted dead pixel map and an FFC event is carried out immediately after waking the camera from sleep. A physical fault would likely cause other issues with the camera ?

A stability problem in firmware can be a real nightmare, as others will attest. It can replicate physical faults within memory and sub systems when such faults do not exist in hardware.

All a bit of a mystery I regret to say.

I will advise if the problem returns. The chassis is getting a full strip down so any intermittent dry joints should get a good agitation, if such exist !

Fraser