Limiting the multi-touch vocabulary

On the classic Nintendo Entertainment System, and many other game systems, there was an extremely limited controller vocabulary.

• Four way direction-pad
• A
• B
• Select
• Start

The entire game library of NES, Gameboy, and several other game systems existed in a world with only these five controls. In fact, many games didn't even use all of them! I remember being surprised when a game uses the select button.

At first, it might seem that having so few controls would be a limitation and result in obscure and non-intuitive control schemes. On the contrary, game designers pay a huge amount of attention to usability and design their games so people can pick them up and use them without much thought. Otherwise, gamers would quickly lose patience and their games wouldn't sell.

With multi-touch systems, the possibilities of many touches and complicated gestures seems to blind designers and developers from the fact that you don't really need a rich vocabulary to navigate an application. There are already a few common gestures that are almost universally supported and used across all of the different multi-touch platforms:

• Tap
• Pan
• Pinch in/out
• Rotate

Using only these few gestures we could build hundreds of applications. In fact, most applications would only need Tap and Pan. (I'm assuming the keyboard would still be used, as necessary.)

Sure, in scenarios that need it, more complicated or custom gestures and manipulations could be used, just like how later game consoles added a few more buttons. But even the latest games use a few primary controls or buttons and the others are only used in special cases.

Before we get crazy, though, we should make an effort to really get the basic gestures right. Figure out what scenarios they should be used, and what metaphors work. Should Pinch In and Pinch Out be used for menu navigation? Does one or two finger rotation make more sense in which scenarios? The early game consoles had hardware and computational limitations that forced them to be creative and make intuitive interfaces. We shouldn't get caught up in all the possible gestures before getting the core set right.

The call to action here is when designing applications, think about the smallest set of multi-touch gestures that would accomplish the job. Can you design a more creative interface that only requires tap and pan? If so, then you might want to opt for that, since it requires the users to learn fewer gestures.

Hand Gesture diagrams adapted from Dan Saffer's graphics. Thanks for the contribution, Dan!

WPF 4 beta 2 Touch documentation

With the public availability of VS2010 beta 2, the MSDN documentation has also been updated.

Two pages that any touch developers should read are:

What's New in WPF 4: Touch and Manipulation

Walkthrough: Creating Your First Touch Application (Basic scatterview-like pan/rotate/stretch on a rectangle.)

It's pretty easy to get started. I actually wrote a program similar to the walkthrough (before finding the documentation) in about 30 minutes.

Multi-touch mice from Microsoft Research

There is article and video in the Seattle Times about some research into multi-touch mice done by Microsoft Research. There is a video that include four concept mice, each using a different concept and technology. This video is worth watching to see what kind of concepts they came up with.



Before watching the video, I was skeptical about the utility of a multi-touch mouse. Mostly I think that point of view was colored by the fact that I think multi-touch trackpads as being inferior to multi-touch displays. I assumed that a MT mouse would be pretty much the same, but after watching the video, I can see there is some advantage.

My problem with the MT trackpad is that it does not allow the NUI concept of direct manipulation. Sure, it is multi-touch and can recognize gestures, but those gestures are only a small incremental improvement to the scroll-sensitive side area of trackpads. With the trackpad is used to move the cursor, the finger movement is relative to the cursor. When the trackpad is used with a multi-touch gesture, it is also relative, but multi-touch gestures are much better using absolute positioning and direct manipulation. True Natural User Interfaces are not possible with MT trackpads. (Would the iPhone be as cool with a non-touch display and a separate MT trackpad and cursor?)

In the video, they showed that each concept multi-touch mouse has one (or more) regular mouse position sensor on the bottom but also a method to detect (many!) touches somewhere on the top. This is different from MT trackpads in that the cursor positioning is done with relative movement of the entire mouse, but multi-touch gestures are done using the fingers on the MT-sensitive surface. Some of the concept mice even responded to whole palm gestures and positioning.

The separation of the relative cursor positioning from the multi-touch gestures may be a solution to the issues I had with MT trackpads. In addition, it allows additional degrees of freedom so that you can rotate and stretch a picture using gestures while and panning by moving the mouse all at once, for example, whereas with the trackpad you would likely be limited to one type of manipulation at a time.

I can envision an ecosystem where multi-touch mice being accepted as a low-cost enabling technology that introduces people to Natural User Interfaces without a large investment. Those would be used in addition to or to complement multi-touch displays and larger Surface devices. It would benefit NUI adoption by allowing existing computers to use NUI software. While it would add to the complexity of planning interfaces across a larger variety of hardware capabilities, the potential for mass adoption may be more important.

Update: Picture of the mice courtesy of CrunchGear: