Tuesday, March 19, 2024

PC Control With Eye Movement For Physically Challenged

By Ashwini Kumar Sinha

Millions of people across the globe suffer from neuro conjugate disorder, or some other form of paralysis, which does not let them move any body part except their head and the eyes. To overcome this challenge, we can make a smart personal computer (PC) that empowers such physically challenged persons to control a computational system with the help of facial movements. It enables control using face and eye movements and works even in low light or dark conditions.

Fig. 1: Author’s eye movement data converted to coordinates
Fig. 1: Author’s eye movement data converted to coordinates

Coordinate movements in the PC =(1080,720) to (0,0)

where x=1080, y=720

mapx=(x), mapy=(y)

- Advertisement -
Fig. 2: Author testing the blink sensor
Fig. 2: Author testing the blink sensor

The solution is based on three different functions:

  1. Blinking detection of the right eye for system operation
  2. Detection of the eye movement using image processing
  3. Translation of the eye movement and blink to control graphic user interface (GUI) of the PC
Fig. 3: The author with his prototype eyeglasses
Fig. 3: The author with his prototype eyeglasses

The challenges

The above three functions come with several challenges, which can be handled with various steps for the device to work perfectly. These are:

Fig. 4: Author’s eye image processing
Fig. 4: Author’s eye image processing

Differentiating between natural and intentional eye blink

Our eyes involuntarily blink at regular intervals so that they remain moist and free of dust. To perform a left or right mouse click, here an eye will have to blink intentionally. However, there has to be some way to distinguish between involuntary and intentional eye blinks so that the device functions properly.

- Advertisement -
Fig. 5: Author testing the code
Fig. 5: Author testing the code

To do that, two sensors (one corresponding to each eye) can be integrated in such a way that when both of them detect an eye blink at the same time, it is considered a natural eye blink. If the blinking happens in only one eye (either left or right), it is considered as an intentional eye blink, resulting in a left or right mouse click.

Fig. 6: Code snippet
Fig. 6: Code snippet

Although this technique is effective, there is one problem. It increases the bill of material to several hundred rupees. Also, people having problem with one eye cannot use this solution.

Fig. 7: Code mapping eye movements to laptop GUI coordinates
Fig. 7: Code mapping eye movements to laptop GUI coordinates

Therefore, this solution has been modified with the inclusion of a code. Now, if an eye blinks within a second, it is considered as natural. But if an eye blinks for more than a second (say, four seconds), it is considered as intentional.

Detecting eye movement using image processing

For a mouse cursor to move, the human eye needs to refer to an object and move the mouse relative to it. Using eye movement, a physically challenged person can effortlessly operate a PC. But image processing of eye movement does not give accurate results, does not work in low light or completely dark conditions, and the entire process is quite difficult.

Thus, a light mounted on the sensor can be used to track eye movement whenever the person moves his/her head.

Fig. 8: Some more code
Fig. 8: Some more code

Translating eye movement and blink to control the PC

This solution detects eye blinks and moves the light mounted on the eyeglass. But it is useless without recording this movement on the PC’s GUI to obtain accurate movement of the mouse cursor. While it needs to cover the entire length and width of a PC monitor, a human head can move only up to a certain degree. To solve this problem, a small head movement needs to be converted into large pixel movement for the mouse cursor.

Prerequisites

To develop the basics, prepare the SD card with the latest Raspbian OS and check whether it has a pre-installed Python IDLE. Next, install the following Python libraries and modules for the project:

  • OpenCV
  • pynput
  • numpy
  • gpiozero

To do so, open the terminal and use the following commands:

sudo pip3 install python-opencv
sudo pip3 install numpy
sudo pip3 install gpiozero
sudo pip3 install pynput
sudo pip3 install nmap

After installing all the libraries, include the OpenCV official GitHub repository inside the Raspberry Pi using the following command:

git clone https://github.com/opencv/opencv

We are now ready to code.

Coding

You will need to modify the example code found in the OpenCV library folder and fuse your code in it to prepare the device. For that, open the OpenCV folder→platforms folder→python, and select the mouse.py code. Copy and paste it in a new text file named headcontrolGUI.py and save it.

Fig. 9: Code for controlling mouse using converted coordinates data
Fig. 9: Code for controlling mouse using converted coordinates data

You will need to get the eye blink sequence from the sensor for a mouse click. To get it, access the gpio pins and import the gpiozero module into the code to read the blink sensor data.

Also, import the pynput module to create a virtual mouse input for the GUI of Raspberry Pi OS.

Fig. 10: Eye blink converted into mouse click on GUI
Fig. 10: Eye blink converted into mouse click on GUI

Next, create an if condition to check the eye status. If the eye status=blink, then set four seconds to determine if it is an intentional blink or a natural one. If the blink is intentional, pass the pynput command to virtual mouse left or right click.

Fig. 11: Testing code and controlling GUI with an eye
Fig. 11: Testing code and controlling GUI with an eye

Construction and testing

Put the eyeblink sensor on right side of the eye glasses and wire it to a Raspberry Pi pin. Connect a camera to the RPi board and place it more than 15cm away from the middle of both the eyes, such that the entire face fits within the camera frame perfectly.

Wear the glasses and run the code. Make sure to mark the red light in the eye (as shown in Fig. 3) with a mouse to track. When successful, a person can operate a PC with eye movement. To move the mouse, move the eye in left, right, upward, and downward directions. To click the mouse, blink (shut) the eye for up to four seconds.

Download Source Code


Ashwini Kumar Sinha is a technology enthusiast at EFY

This article was reviewed and transferred from EFY June 2022 Issue and was published online here.

SHARE YOUR THOUGHTS & COMMENTS

Electronics News

Truly Innovative Tech

MOst Popular Videos

Electronics Components

Calculators