Automating a finger press on iPad screen

[tifkat]

I have a small project I am interested in exploring, to automate a finger press on a touch screen tablet/iPad.

If I was to purchase something like the silicon nib in this AliExpress sale:

https://a.aliexpress.com/_mrVjShP

and have it screwed into a 3D printed jig which was actuated by a suitable MCU, would it work as is, or does the nib need to have some electrical connection, say to ground, or to a capacitor?

How likely is this to be universally compatible with older or newer screens? Say we have iPad 9.7", and future plans might involve an upgrade to iPad Pro 2020 or Air 4th gen?

[Renate]

I know nothing about the Apple family.

In the Android world I know how and have implemented this three ways.

1) Android has the shell "adb shell input tap 100 200" command. You issue it over ADB.
This is actually pretty boring and slow because the default is a Java process that takes forever to spin up.
Maybe Apple has something similar?

2) Android uses Linux and it has the regular /dev/uinput system.
Just do an ioctl UI_DEV_CREATE to /dev/uinput and create your own input device of any type.
Create a process and have a pipe to take taps/text/whatever.

3) Use USB host mode. Make a custom USB HID device
On the side plugged into your device it's a keyboard/mouse/whatever.
If your µP has two USB interfaces, just connect the other to your host.
If not, use a serial interface and a USB UART to connect to host.

[Manul]

Yes, mechanical solution will work, if the tip has enough capacitance - if it is coupled to something big in size or to mains ground. It is probably the most reliable way (no false touch), except, of course, software solutions, which were already mentioned.

Yet, because this is an electronics forum, I will mention that you might be able to simulate touch electronically. In fact I tried this once for fun and it worked. You stick a conductive disc shape "electrode" on a screen, size of finger tip and switch it to ground by using small very low capacitance mosfet. It might need a little play, like for example, applying thin isolation layer on the bottom of electrode - bare metal might be not good. Electronic solution is more risky and migh generate false touch in some cases because of electrical noises.

[Fraser]

Modern projected capacitance touch screens, as found in iPads use a form of proximity sensing. Disruption of the projected field is detected and not an earthing or EM noise proximity process. It just requires a stylus with a conductive tip, carbon coated rubber etc, to disrupt the projected field enough to be detected by the touch detection matrix.

https://zytronic.co.uk/insights/article/how-does-projected-capacitive-touch-technology-work/

[Fraser]

This reminds me of a keep awake device I once saw that used a nodding toy 

Completely different requirement but an interesting mechanical solution to a ‘problem’ 

https://youtu.be/Lw243n4djCs

Fraser

[Fraser]

More ‘thinking outside the box’ 

https://www.reddit.com/r/lifehacks/comments/6d9d79/prevent_a_lockeddown_work_pc_from_sleeping/

Fraser

[Manul]

I remember doing experiment by puting AA battery on a touch screen with flat (minus) end down. It reacted to me touching the other end. So then I tried switching to ground with a small mosfet and it worked. It was all like 5 min experiment just for fun. Not sure how repeatable or reliable it is. I think it was iPhone 7 or 6.

[Renate]

Well, I apologize for only coming up with three ways to do this.

4) Reverse engineer the app to allow control programatically.

5) Use a capacitive contact like already suggested.
I use copper foil with an adhesive backing, about 1 cm x 1 cm.
Use CMOS inverters (or complimentary outputs of a µP) to drive two 1N4148 forward and reverse polarized.
I use a 1 k resistor in series with each diode on the driver board (in this case, a pusbutton & CD4093).
(It's hard to get a good photo.)

6) Use a solenoid and a soft tip.

[tifkat]

From what I've seen here, and others places, the nib would need to be connected to a common ground with the device.

I could make a charging dock which would also feature the MCU controlled nib which would be connected to a common 0v/GND with the charger output. I had considered a solenoid, but having little experience, am no sure on impact velocity, and probably would desire to avoid the noise.

I have actually considered a small geared motor driving a cam shaft to actuate the 'finger'

My main concern was electrical requirements for the silicon nib, which it turns out should be GND in common with the target device.

[Renate]

7) Build a custom Bluetooth HID device

You've got a lot of possibilities here. Why do you want to stick with a physical touch?