What is the best sensor for gesture controlled rotary knob?

[Georgy.Moshkin]

I am in search of sensor which is best suited to detect rotary movements of pointing finger. Goal is to achieve experience similar to mechanical rotary knob at sensing distances around 3cm. I have some success in detecting such gestures realtime using data from IR based gesture sensors, but there are also some limitations that difficult to deal with (mirroring, etc.), which require additional sensor or fusion with data from other type of sensor. Computer vision algorithms are great, but most camera modules do not work in low light conditions, use larger space and need more power for processing. If current approach fails, I'll try VL53L5CX.

[BadeBhaiya]

I think a capacitive touch/slider would be ideal for this. No moving parts so MTBF would be quite low.

Maybe I'm too obsessed with value engineering but using a camera or a VL53XX ToF sensor seems kinda wasteful?

[rstofer]

Microchip seems to have some ideas about touch pad recognition:

https://ww1.microchip.com/downloads/en/Appnotes/Touch-BuiltIn-Surf-GestRecogn-Using-PTC-DS00003433B.pdf
http://ww1.microchip.com/downloads/en/DeviceDoc/QTouch-Modular-Library-Peripheral-Touch-Controller-40001986B.pdf

[Georgy.Moshkin]

I think a capacitive touch/slider would be ideal for this. No moving parts so MTBF would be quite low.

Maybe I'm too obsessed with value engineering but using a camera or a VL53XX ToF sensor seems kinda wasteful?

Depending on where it would be used and selling amounts. And the other important aspect is how good the algorithm is. Yes, it is almost 10 times more expensive than infrared based gesture sensor, which is essentially is a four-pixel IR camera (gesture sensing part of APDS9960). Price and reliability may be a problem, but my top priority is usability and size. For my accessibility project I needed several high-quality parts - radar sensing, camera sensing, gesture sensing and feedback system. For a such system TOF price is acceptable, but adding it only makes sense if it adds a lot of convenience. There is some balance between component price, usability and overall product price. The problem with most current solutions is that people must perform some clunky large-amplitude movements and "calibrate" their bodies to make sensing work, often repeat gesture several times, even if using product every day and learned how to use it. I have gesture-controlled kitchen lamp, and it is ridiculous how long it takes to turn it on or adjust the brightness.

Microchip seems to have some ideas about touch pad recognition:

https://ww1.microchip.com/downloads/en/Appnotes/Touch-BuiltIn-Surf-GestRecogn-Using-PTC-DS00003433B.pdf
http://ww1.microchip.com/downloads/en/DeviceDoc/QTouch-Modular-Library-Peripheral-Touch-Controller-40001986B.pdf

Thanks, it gave me an idea of how evaluation board may look like if I continue development of custom IR based gesture sensing module.

[Georgy.Moshkin]

Are there any good examples of touchless rotary gesture implementation using infrared sensors similar to apds9960? I do not see a reason why hand movements could not be sensed with much better precision. I want to understand the competition. Most YouTube videos on gesture sensing demonstrate relatively poor performance even for basic gestures (up/down/left/right). I am thinking about creating a module with much better gesture sensing characteristics, gesture angle, speed, linear and rotational movement sensing, and then sell it. How do you think, is it a viable idea? (asking eevbloggers).

[ebastler]

Could you explain the overall use case and the intended arrangement a bit more?

I understand that your overarching use case is a sensing device for the blind or visually impaired. Why would you want to use a touch-less gesture interface in that scenario -- wouldn't a physical rotary encoder, joystick etc., which provides tactile feedback, be preferable? Where would the gesture detector be physically mounted; how far away from it would the user's hand be; what "field of view" and range of distances does the gesture sensor need to capture?

[Georgy.Moshkin]

Could you explain the overall use case and the intended arrangement a bit more?

I understand that your overarching use case is a sensing device for the blind or visually impaired. Why would you want to use a touch-less gesture interface in that scenario -- wouldn't a physical rotary encoder, joystick etc., which provides tactile feedback, be preferable? Where would the gesture detector be physically mounted; how far away from it would the user's hand be; what "field of view" and range of distances does the gesture sensor need to capture?

There was no clever idea behind that. I haven't planned any buttons, only on/off and maybe indoor/outdoor mode. Imagine if person needs to press some button urgently, having a white cane in one hand and a heavy bag in the other hand, there is a chance that something will be hit by a white cane while hand is moving to some buttons. My reasoning may disappoint you. I've bought relatively popular affordable sensors and modules, one of which was proximity sensor. It provided good data at very short distances. After some experimenting, I've made a not very reasonable decision to investigate gesture sensing, because data looked good, and I did not want to waste this function. Also, there is a deeper underlying problem. I was asked by people to show them stuff, to bring "the device downstairs". It made me feel kind of ashamed and angry at the same time, because there is nothing to show. And if I bring up time and money constraints, I'll probably hear some cliche answers I do not want to hear. To sum up, the main goal at this stage was to impress an investor, to show cool features, to make experience similar to a good toy. In other words, device should provide feedback about surrounding objects position and movements in a such a way, that sighted person would want to play with it and try to navigate without any prior training. After getting finances I was planning to exit my job and focus on making prototypes, pushing them to some association of the blind, do real tests, data recordings, receive feedbacks, go through prototyping iterations. Another aspect is that my motivation source is satisfaction from engineering and problem solving, I can't say that I care too much about disabled people, I can feel empathy, I can be terrified by idea of becoming blind.

Now I put some "clever" ideas.
Physical controls you've mentioned are much better for this task.  My initial view was to avoid any controls at all, just a simple on/off button. Controls limits user when both hands are busy with something already, or wearing mittens, gloves, etc.
Gesture controls may be useful if device is head-wearable which easily displaced when touched. With gestures, you can do some simple controls while wearing carrying some object and all fingertips are busy. No need for tactile feedback to find right button.
I've preferred cheap sensors and modules, bought several modules, all below 20 USD.
I've preferred sensors with low bandwidth information. I was planning on using cortex M7, and processing need to be not non-heavy low power consuming.
Radar: provides a lot of useful data over very low data. Very cheap if custom designed using FETs as resistive mixers.
Camera: provides a lot of data, but very high bandwidth. Plan was to heavily decimate the data, kind of ROI algorithm that throws away everything but few small rectangles, which then fed to angle extraction algorithm based on curve tracing. No time for FPGA at this stage
IR gesture sensor: a lot of data with a super low bandwidth. Can put several of these for almost 360 degree coverage.
Lidar: expensive and high bandwidth, I did not like demo videos of modules with data processing
Ultrasonic: not considering it

About rotary knob. As my optimism in accessibility device faded, I've generated more ideas. And few months ago, I've felt a great potential of gesture sensing for entertainment, industrial, hobby and more. Touchless vinyl scratch effects, DJ deck, musical instruments, guitar pedals. Working in gloves, touchless controls that can be used with thick gloves covered with dirt or sticky goo. Bluetooth/USB connected HID devices for personal computers, new ones or keyboard/mouse enhanced with gesture "window" for gaming or other purposes. Password input using finger micro-movement, alarm micro-gesture panic buttons embedded in the floor, etc.

[ebastler]

Thank you for the detailed account!

Nothing wrong with trying a promising technology, even when its potential integration in your overall design is not clear from the start. Designing a product is not strictly top-down, from market analysis to user requirements to specs to design. In reality there is often an interplay between top-down and bottom-up (technology driven) design choices, and a clever technical solution may provide a nice differentiator.

But it is important not to get carried away. Once you find that gesture recognition works well, and makes for an impressive investor demo -- take a step back and think about how it will be integrated in your final product. In your accessibility device, for example, I am not convinced that making gestures is really more convenient than pressing a button: If one is carrying their grocery shopping and a cane, gesturing may be difficult, accident-prone, and may look a bit ridiculous.

How about a little wireless remote controller integrated in the cane handle? Important to check with actual users though; I have no idea how a cane is typically held for best tactile feedback, and where a button, rotary encoder, mini joystick or such could be placed to be conveniently usable.

[Georgy.Moshkin]

You a right, I was carried away by experimenting on gesture detection and other algorithms. The only benefit is that now I have more experience with various sensors and algorithms. About best buttons arrangement: buttons may be there, but interaction probably must be minimum. Like an emergency flashlight which is automatically turned on when pulled from wall mounted bracket. No need to search for the button, only for a flashlight.
About touchless rotary knobs in general, I think that IR looks promising. Price is low, can use three sensors in triangle arrangement if needed, just to increase area where correct rotary movements can be estimated.