System Gestures vs. Manipulations

In my previous post, I mentioned that gestures need to be evaluated in context to determine what the user intends. We can break down gestures into two types: System Gestures and Manipulations.

A system gesture is a gesture (as previously described) that can be performed in any context and have the same meaning. For example, if a foreground document editor responds to a downward flick gesture to go scroll down regardless of where the flick started or ended on the screen, that would be a system gesture.


A manipulation is a gesture performed on or to an UI element on screen, or in NUI terms, an object. The object will only perform the action if the gesture falls at least in part within the its bounds. For example, if a 3-D carousel only scrolls if the user starts a drag gesture from within the visible elements and not just anywhere, that is a manipulation.

One core NUI concept for touch interfaces is direct interaction. This means that when you touch an object, that object should respond, not something else. Applying this to gestures, most interaction should use manipulations rather than system gestures. The reasoning is simple: if you use a system gesture over one object (say a downward flick over a non-active window) but another object responds (the foreground document editor instead of what the finger was over), then the user can easily become confused.

Manipulating objects directly is much more intuitive and much easier to discover than generic system gestures. Objects can have affordances (future topic) that help the user figure out what they can do with it. System gestures are only learned by reading instructions or by accident (or socially.)

Most system gestures are not really NUI worthy. They are actually a step backward into the era of rote learning of CLI commands, except with touch. Some of the first generation of touchscreen laptops seem pretty cool, but when you figure out you can only really use manipulations within a proprietary application, and everywhere else it just uses system gestures as shortcuts for common tasks, it doesn't seem nearly as cool. That's mostly a software problem, though, since most existing software is written for WIMP, not for touch.


There are a few exceptions. The application chooser on Microsoft Surface is basically a horizontal scrolling picture list. It scrolls when you drag anywhere on the screen. In fact, with the exception of tapping the corner hotspots, the application icons, or the "I'm done" button, if visible, "drag anywhere to scroll" is all you can do. Even though it is a system gesture, it was smart to use because new Surface users may still be learning that Surface responds to touch. They are much more likely to discover the touch responsiveness if 98% of the screen will scroll, and the rest responds to taps. In this case, the concept of scaffolding overrides direct interaction. (More future topics.)

Regardless, when thinking about gestural interfaces, we should design for manipulations in almost all cases.