NUIs reuse existing skills (updated NUI definition)

Last month I posted an excerpt from my book with a section that contained a definition of natural user interface.

A natural user interface is a user interface designed to use natural human behaviors for interacting directly with content.

It sparked some great conversation both in the comments of that post and some other blogs. Richard Monson-Haefel suggested on his blog that I consider changing the "use natural human behaviors" part to match how I talk about natural in terms of innate abilities and learned skills. Those discussions were derived from some of Bill Buxton's thoughts on NUI.

At the time, I was on the fence about this change. I wrote in a comment that I liked using the simpler vocabulary words since it will be understandable by more non-technical folks, and that I liked the term behavior for a rhetorical reasons since in the beginning of chapter two I had an interesting set of phrases that needed it to say "behavior" to make sense. (I know that isn't a very good reason and actually I since deleted those pages with that particular discussion.) So I was leaning towards my initial definition but I was keeping my mind open for more convincing arguments.

Well as it turned out, I was convinced by Bill Buxton at MIX10. He wasn't talking directly to me about this, but in his keynote he described how a saxaphone input device is a natural interface for someone who has years of saxaphone training, and he proceeded to use his electronic saxaphone (a Yamaha WX7 wind controller) to create flute and electric guitar sounds. Below is a screenshot, but if you want to watch his part of the keynote, skip to mark 96:45 of the MIX10 Day 2 Keynote video.

Bill Buxton uses a Yamaha WX-7 wind controller and his existing saxaphone skills to create music that sounds like a variety of non-saxaphone instruments.

When I heard him describe this, I immediately understood that my existing definition ("use natural human behaviors") excluded advanced NUIs designs for people who were already experts with advanced skills. I had already prepared my NUI presentation for day 3 including the original definition, so directly after the keynote I went and changed it. In my session video you can see I used a revised definition of NUI, and in the next revision of chapter 1 (now updated in the MEAP) I also updated the definition.

Here is the updated book excerpt:

There are several different ways to define the natural user interface. The easiest way to understand the natural user interface is to compare it to other type of interfaces such as the graphical user interface (GUI) and the command line interface (CLI). In order to do that, let's reveal the definition of NUI that I like to use.

A natural user interface is a user interface designed to reuse existing skills for interacting directly with content.

There are three important things that this definition tells us about natural user interfaces.

NUIs are designed

First, this definition tells us that natural user interfaces are designed, which means they require forethought and specific planning efforts in advance. Special care is required to make sure NUI interactions are appropriate for the user, the content, and the context. Nothing about NUIs should be thrown together or assembled haphazardly. We should acknowledge the role that designers have to play in creating NUI style interactions and make sure that the design process is given just as much priority as development.

NUIs reuse existing skills

Second, the phrase "reuse existing skills" helps us focus on how to create interfaces that are natural. Your users are experts in many skills that they have gained just because they are human. They have been practicing for years skills for human-human communication, both verbal and non-verbal, and human-environmental interaction. Computing power and input technology has progressed to a point where we can take advantage of these existing non-computing skills. NUIs do this by letting users interact with computers using intuitive actions such as touching, gesturing, and talking, and presenting interfaces that users can understand primarily through metaphors that draw from real-world experiences.

This is in contrast to GUI, which uses artificial interface elements such as windows, menus, and icons for output and pointing device such as a mouse for input, or the CLI, which is described as having text output and text input using a keyboard.

At first glance, the primary difference between these definitions is the input modality -- keyboard verses mouse verses touch. There is another subtle yet important difference: CLI and GUI are defined explicitly in terms of the input device, while NUI is defined in terms of the interaction style. Any type of interface technology can be used with NUI as long as the style of interaction focuses on reusing existing skills.

NUIs have direct interaction with content

Finally, think again about GUI, which by definition uses windows, menus, and icons as the primary interface elements. In contrast, the phrase "interacting directly with content" tells us that the focus of the interactions is on the content and directly interacting with it. This doesn't mean that the interface cannot have controls such as buttons or checkboxes when necessary. It only means that the controls should be secondary to the content, and direct manipulation of the content should be the primary interaction method.

I decided to say just "reuse existing skills" and not include the innate abilities part to keep it simpler and also because the thing that convinced me to change it was the fact that some NUIs may use advanced skills, so innate abilities may not always play into NUI. The core thing that did make it natural was reusing existing skills.

I do need to thank the commenters on my last blog, Ben and Laurent, and particularly Richard, who had also put himself out there trying to define NUI but still had great comments about my thoughts. Richard pointed me in the right direction and with some time and thought I realized that he was correct. It's you, my readers, who will help me make this book totally awesome.

Please keep the feedback coming and feel free to call me out if you think I could improve something.