parallax

[artag]

I like the effect of an edge-sharpened visual camera superimposed on a thermal image to make it easier to recognise the view (though increasing resolution is making that less important) but actual implementations suffer from parallax and need a usually manual correction to line up the images accurately. Adding an infra-red macro lens usually makes the visible image unusable - either the supplementary lens obscures the other sensor or the changed FOV makes the images unmatched.

It seems as though there's a few ways to fix this. My knowledge of optics isn't great so perhaps someone could tell me why these wouldn't work :

1. Perform autofocus on one or both images and use that to measure the distance (like an old-fashioned rangefinder). Then use the measured distance to correct the image overlap in software.

2. Use a semi-clear lens such as zinc selenide to cover both sensors. This will result in a  yellow-tinted image and a different focal length. Image colour is not a huge problem (it's often just monochrome for edge sharpening and maybe some colour correction is possible) and an approximation to focal length correction could be made by placing the sensors in different planes or even moving one of them.

3. Place a half-silvered, front-silvered mirror in front of the visible light lens at an angle. Reflect the IR sideways into the thermal sensor. It won't pass through any glass so will be subject only to the thermal lens. Part of the light will pass through the mirror and into the visible-light lens system. The two images will then have separated optical paths.

4. In an inversion of 3, have a flat plate mirror of germanium or chalcogenide glass which would reflect the visible image sideways and pass the IR image through. Maybe even have a plano-convex lens where the plano side acts as a mirror and the convex side focusses the IR.

 

[Bill W]

1. Perform autofocus
2. Use a semi-clear lens such as zinc selenide to cover both sensors.
3. Place a half-silvered, front-silvered mirror in front of the visible light lens
4. In an inversion of 3, have a flat plate mirror of germanium

1 would work - but only at the [scene] focal plane
2 won't work because you still have two sensors on separate optical axes.
3 & 4 would work.

There is also 5 - where manual focus of the thermal lens is used to converge the optical sensor axis mechanically at the desired distance

With relatively wide angle lenses, and being at the 'cheap' end of things, these are not going to be worthwhile commercially.  The old rule that spending too much on fixing a 'cheap' system means any buyer could simply buy the more expensive system that works properly in the first place.

You would also still get the phantom images issue of looking through glass - where 'MSX' tries to show a tree through the window in thermal.

Bill

[artag]

Thanks for the feedback.

I guess you're right that they wouldn't be cheap any more, though might be fun DIY.  The cheapest would be the zinc selenide lens, as you say it wouldn't be ideal but maybe puts the sensors closer together than the common solutions.

Is there any one doing a 'proper' system, albeit expensive ?  Maybe using some improved version of one of the above.

[CatalinaWOW]

You can only correct parallax if the subject has no depth.  You can sorta-kinda correct but there will always be pieces of the scene invisible to each camera when the scene has depth. 

The only fix for this is to have a common optical path from the scene.  It can be achieved by a partially reflecting mirror.  If the mirror is an interference filter the split can be nearly lossless with each sensor receiving nearly all of their in band photons.  Chief drawbacks are bulk and fiddly alignment + cost of the filter.

[Vipitis]

using a beam splitter and common optical path does kinda work. I have tried to do that by using a Germanium window. It did require both systems to focus individually, and was somewhat limited to a narrow fov.

[Phil1977]

No 6. Place 3 VIS-cams around the thermal imager, combine their pictures to a synthetic aperture picture which has the IR-Imager as a virtual optical axis.

Would probably mainly need a potent video processing unit.

[dietert1]

Our FLIR camera supports manual parallax adjustment, but only for distances above 0.3 m, where the parallax is already small as the distance between the two imagers is about 2 cm.
I think a beam splitter should work pretty well as one doesn't need a lot of light for the RGB imager.

Regards, Dieter

[Phil1977]

The problem with the beam splitter is that it works best if it is transparent for thermal IR - that means in best case you use Germanium which is quite expensive for larger pieces.

If you do it vice versa, if you e.g. use a gold plated glass that is highly reflective for thermal IR but transmits a small part of VIS, then you get a quite affordable system but the thermal IR reflector is always reducing your contrasts and makes it more difficult to get absolute temperature measurements than it anyhow is.

[artag]

What about dichroic glass ? That's used in halogen bulbs to allow IR to pass out the back whilst reflecting visible as a parabolic mirror. I don't know what wavelength it passes though, nor how much of the IR is absorbed by the glass.

I like Phil's virtual camera idea. I think something like this is done for wrap-around car reversing mirrors so probably doesn't need an insane amount of processing. Especially for something like PCB views where you've likely got a PC and maybe a GPU to do it. Maybe it could also provide depth imaging. However it might make focussing more difficult - 3 instead of 1.

 

[Phil1977]

The problem with all dichroic filters is that it´s very difficult to get it highly efficient and broadband. For the halogen bulbs it was for sure better than nothing, but for (precise) thermal imaging you have to keep the SNR for low temperatures really good. A few percent of self-emission would probably spoil the thermal image quality - the real thermal camera experts here can tell more about this for sure.

[CatalinaWOW]

What about dichroic glass ? That's used in halogen bulbs to allow IR to pass out the back whilst reflecting visible as a parabolic mirror. I don't know what wavelength it passes though, nor how much of the IR is absorbed by the glass.

I like Phil's virtual camera idea. I think something like this is done for wrap-around car reversing mirrors so probably doesn't need an insane amount of processing. Especially for something like PCB views where you've likely got a PC and maybe a GPU to do it. Maybe it could also provide depth imaging. However it might make focussing more difficult - 3 instead of 1.

 

There might also be possibilities with a pellicle, a thin film stretched over a frame to make it flat.  The minimal thickness minimizes absorbtion losses making materials like polyethylene feasible.  The downside is that I know no way to lay an interference filter down on polyethylene so you are limited to partially reflecting mirrors.

Another option to explore to improve the contrast is to cool elements visible to the sensor.  A little bit helps a lot, but staying above the dew point, and calibration for thermometry is more challenging. 

What you do depends on whether geometric accuracy or temperature accuracy is more important to you.  For my uses of thermal cameras looking at circuit cards even several degrees of error isn't too important, but there are applications that need (or at least really want) fractional degree accuracy.

[Phil1977]

I like Phil's virtual camera idea. I think something like this is done for wrap-around car reversing mirrors so probably doesn't need an insane amount of processing. Especially for something like PCB views where you've likely got a PC and maybe a GPU to do it. Maybe it could also provide depth imaging. However it might make focussing more difficult - 3 instead of 1.

 

The idea to look for the car-surround-view algorithms seems promising. The real "synthetic aperture" approach is mathematically quite demanding, but the cars proof that you can do this on quite a low level GPU. But I don't know if the car systems really can reconstruct an arbitrary 3-dimensional environment or if they are limited to a standard road topology.
The older systems I know of own driving experience just merge together two or more pictures after perspective correction - that´s really easy to do, but wouldn't help for the IR-cam.
I´ve seen in Niu and Tesla the systems where you can turn around the view around the car however you like, these seem to be more advanced - but how advanced is more advanced?

[artag]

I was imagining something like the software for stitching stills together into a panorama. But tbh it's probably driven by the user and just does the detailed work given a key match. That would be OK for initial alignment but not really for continuous use.

My needs aren't really very great. I'm just extrapolating from the slightly underwhelming cheap offerings and wondering what's possible.

I do have an idea in mind. 
I use an optical stereoscopic microscope for a lot of work. It has a long arm mount and produces a superb image but sometimes the need to use eyepieces is annoying.  A Vision Engineering unit is very expensive and takes a lot of space. A USB microscope tends to be laggy even though I have a conveniently placed monitor.
I can imagine a digital microscope setup (probably streaming over HDMI rather than USB) with a suitably floating bench support and the possibility of overlaying thermal data. It could be quite big as most of the optics would be  behind the LCD screen. I'm not sure if the lack of stereo vision would be a big handicap - the cheap digital microscopes I've tried have mostly suffered from small screen sizes and lag.

As CatalinaWOW says, the thermal image alone is pretty usable - the optical overlay helps locate the image but the thermal image has enough detail that it's pretty usable on its own. I've just doubled resolution by going from the Seek camera to a 256x192 TS001 which helps as well.

[Geoff-AU]

Our FLIR camera supports manual parallax adjustment, but only for distances above 0.3 m, where the parallax is already small as the distance between the two imagers is about 2 cm.

Mine too.  I emailed them asking for the slider to have more range, so that my $400 entry-level IR phone dongle could have better close-in performance for working on PCBs.

They replied, telling me to spend $4000 on a unit that has better specs.

Needless to say this will be my first and last FLIR device. 

Zinc Selenide lenses do work for addressing the parallax adjustment limits in my case (and way cheaper than the $4k FLIR).  But I haven't been sufficiently motivated to design a holder for it.

[Vipitis]

how about putting a mirror on some kind of actuator or wheel, sync it with the framerate of both cameras so it never reduces your thermal signal? and you then simply grab a short vis exposure of the diverted path for the outlines (maybe discard all other frames).

[CatalinaWOW]

how about putting a mirror on some kind of actuator or wheel, sync it with the framerate of both cameras so it never reduces your thermal signal? and you then simply grab a short vis exposure of the diverted path for the outlines (maybe discard all other frames).

Clever idea.  Shouldn't be expensive to implement, though quite a bit of brain juice will be expended.

[Nogtail]

I tried this a few years ago with a glass pane taken out of a picture frame that was somewhat reflective to IR. I placed it at a 45 degree angle as a beam splitter. I ended up having a couple of issues: Firstly, the glass I used wasn't anywhere near flat which was causing distortions to the reflected thermal image. Another problem was the IR image needed lens distortion correction to align with the visible image. Probably not too difficult, but I didn't have the time to spend on it. Using a mirror with the reflective coating removed might be an easy way of getting a fairly flat sheet of glass. Perhaps a large microscope slide would also work.

[artag]

how about putting a mirror on some kind of actuator or wheel, sync it with the framerate of both cameras so it never reduces your thermal signal? and you then simply grab a short vis exposure of the diverted path for the outlines (maybe discard all other frames).

Clever idea.  Shouldn't be expensive to implement, though quite a bit of brain juice will be expended.

Maybe combine it with the calibration shutter.
Didn't early cameras have a continuously rotating shutter rather than today's intermittent ones ?
Seems like a step backwards but in many ways a spinning shutter is better than a flip in/out one

[Phil1977]

Maybe one more pragmatic idea: Put a small VIS-cam in front of the IR-cam. Take the image, freeze it and swivel the VIS-cam away for thermography.

Or do the same with a removable VIS-mirror and fixed cams.

[Bill W]

Maybe combine it with the calibration shutter.
Didn't early cameras have a continuously rotating shutter rather than today's intermittent ones ?
Seems like a step backwards but in many ways a spinning shutter is better than a flip in/out one

Yes they had continuous rotating shutters / choppers, but that was because they used sensors that only responded to changes.

I never tried a bolometer and chopper as a 'shutterless' system - you'd get constant imagery if only 30 or 15 fps.  The required front area would always be a problem as a commercial proposition.

[artag]

Another possibility is to use the parallax as a feature, rather than a bug, in the manner of rangefinder cameras. It's quite hard to focus using a thermal image alone as heat bleed smudges the edges and the real features you see are likely due to local cooling or emissivity differences.

This is somewhat like the cross-coupling between display offset and focus controls, but with a different priority so that minimising parallax (manually or by cross-correlation of edges) optimises focus.