Dreaming to navigate around your desktop while you’re typing without moving your hands to reach the mouse or touch screen? Or playing games without using any keys? Here’s how your dream to be realised by Microsoft Research.


Microsoft Research is the research division of Microsoft to study on different aspects through computing. They have 1,100 employers from different background. The idea is to collaborate with government and the industry to solve real world problem via technical innovation.  Not only algorithm, software and hardware development, the team also have researches for health, education and social sciences included too.  The Kinect for Xbox is probably one of the best thing they ever created. Despite started as gaming device, now it has been use widely in different areas such as baby monitors and even for military purpose in South Korea.


Coming back to the keyboard mentioned earlier. It is a project from the team to enhance user experience while using keyboard. It is a type-hover-swipe keyboard where you can literally type, hover and swipe over it to use it. Aside from normal typing, motion gestures above the keyboard can be sensed and your computer should response accordingly.


What they did is actually inserting rows of infra-red proximity sensors between the rows of keys. The matrix of sensor will be able sense the location of objects above them and the distance between object and the keyboard. By analysing the readings from the sensors, gesture such as swiping from left or pinching can be recognised and these gestures have been assigned with different actions. After the gesture being detected and identified, your computer will react accordingly, performing the task you want, such as zooming in, switching between programmes and scrolling down. These actions are simple but can be annoying when you have to do it repetitively during typing.


The idea seems simple but the technology behind more complex than what we think. The position of sensors and spacing between them need to be adjusted correctly. To detect the motion, the frame rate ( speed of sensor capturing the object location) has to be high enough.  There is a large set of motion gestures and differentiating each other will not be easy at all. Different gestures have to be repeated over and over again by different people to sample the raw readings from the sensors for each gesture. This will be recorded and then when a new gesture is detected, the algorithm developed will compare the new readings with the data recorded to decide the gesture. Studies have to be carried to assign actions to each gesture. It has to commonly used to lower the learning curve.

Head over the official page to watch the video and learn more about it. You can have a look at other interesting projects too. The prototype is very promising although still in early stage. The size of the keyboard is not any larger compared to normal one. I certainly like the idea and I hope we can see it in the market soon. I will grab one definitely.