Been some time that I came across an interesting patent application but here’s an interesting one. Microsoft engineers have applied for two patents detailing the Architecture to control the PC using hand gestures. Earlier Microsoft showed controlling devices by muscle movements and has applied to patent a pressure sensitive mouse.

The basic architectural diagram shows a user can use voice, gesture, keyboard or mouse as inputs to a computer. Microsoft engineers talk about using both voice and gesture together to control various actions on the PC. A basic image shows the two cameras and a mic are placed in front of the user:

Hand Gesture PC Control

To determine when to consider movement as input the system relies on the volume area around it. The person who plans to interact with the computer has be within the engagement volume before the system kicks in and starts considering the movement as input. Here’s a flowchart that explains how the whole system is intended to work:

Flowchart To Control PC Using Hand Gesture

The patent gives examples of how voice and gesture input would work together. When you make a gesture the screen will recognize it and show you options on the screen:

Hand Gesture Options Microsoft Patent

The following is what you have to do and what the system will do:

  • You make a hand gesture
  • The cameras track your movement and show 4 options on the screen
  • You say ‘move’
  • The mic recognizes the word and depending on the direction you now move your hand the window will be moved.

That’s the basic gist of it. There are some more commands such as ‘Flick’ that’ll allow you to move a window from one monitor to another as shown in the image:

Move Windows Using Hand Gestures Microsoft Patent

One of the uses of this technology as shown in the patent application is while doing boardroom presentations. A user can flick through slides or demonstrate something using gestures:

Control Projector Using Hand Gesture Microsoft Patent

An image in the application is a what Natal can do. Two users interacting with their PCs that are connected over a network and there’s a third broadcasting the two screens:

Hand Gesture Microsoft Patent

Since Project Natal offers pretty much all of the same capabilities, Microsoft engineers bringing this to the PC would be quite interesting. According to them, this might help people who’re not comfortable using a pointing device.

Patent Application