Metaphors and OCGM

It really seems like OCGM has sparked a lot of discussion! Ron first described it two days ago, and already there have been several different bloggers posting their own ideas and responses to those blogs. Lynn has a good summary of the conversation so far, and the thread on nuigroup.com also has a summary of links. I think overall the reception has been positive. There is a lot of enthusiasm for NUIs, and OCGM is just one way for people to understand NUIs better.

There have been some comments about Objects, Containers, Gestures, and Manipulations that seem to be the result of not fully understanding OCGM yet. It is a new concept, and not much has been publicly discussed yet. I want to help people understand better by explaining a bit about metaphors and OCGM.

What is a Metaphor?

A metaphor is the understanding of a complex idea in terms of another, well-understood idea. Here is an example:

Most people (myself included) do not understand the complexities of the financial markets. When the economic meltdown really got going in 2008, the media struggled with understanding what was going on and how to explain it to the layman. One term that was heard in almost every story about banks was the Toxic Asset.

The phrase "Toxic Asset" refers to financial assets with some fault that means they cannot be sold for a reasonable price. The word toxic is used to describe specific attributes of these complex financial tools. There is nothing literally toxic about them -- they're contracts and legal documents, and touching them won't poison you. Toxic waste and these financial assets have certain similar properties, and the metaphor of a "toxic" helped people understand the complex idea (financial assets) in terms of a well understood idea (toxic waste.) If you had it, you want to get rid of it, but no one else wants to take it from you.

Metaphors are used literally all the time in every day language and activities and are core to the human experience. The word "core" in the previous sentence is a metaphor to help you understand the relationship between metaphors and the human experience. Metaphors are not actually inside of the human experience, in the way the core of a planet is inside the planet. But both the planet core and metaphors are important parts of the things they are metaphorically the core of. For more on this, I really recommend reading about Conceptual Metaphors and Conceptual Blending on Wikipedia.

What about metaphors and interfaces?

Computers are extremely complex things. The software running on the computer is also very complex. In order for non-computer geniuses to be productive at all, the human-computer interface has to help the user understand the complex parts of the software in terms of something the user understands.

Even though some people claim that NUIs will eliminate metaphors, that is not completely accurate. NUIs will still have metaphors (unless you want to interact directly with binary electrical pulses), but the NUI metaphors are based around a different core set of values. The NUI metaphors will be less complex than GUI metaphors. (There we go with Occam's Razor again!) This will result in easier to use applications and more enjoyable user experiences.

Even the command line interface had a few metaphors. It had the keyboard input, which to early users was similar to the typewriter. It had folders and files, which are familiar to office workers. It had the command prompt, commands, and responses, which are similar to the interaction with service industry workers: "What can I do for you today?"


A command line interface waiting at the prompt for your input.

We all know and have discussed ad nauseum that GUI's basic metaphors are Windows, Icons, Menus, and Pointers, so I will skip discussion of those.

Is OCGM a metaphor?

You can probably guess by now where I'm going with this. Objects, Containers, Gestures, and Manipulations are each metaphors, and they will help us understand and interact with software in a different way than before.

Aside from the interface, an application that transitions from GUI/WIMP to NUI/OCGM could be exactly the same. It can still have the same databases and same business logic and same web services and everything else. The new NUI interface will be designed using OCGM as the foundational metaphors. There will be other specific metaphors based upon the application content and interactions, but those metaphors build upon Objects, Containers, Gestures, and Manipulations.

What do Objects, Containers, Gestures, and Manipulations have to do with each other?

Ron described in his latest post, Welcome to the OCGM Generation! Part 2 that OCGM has two categories (items and actions) and two subcategories (base and complex.)

I agree with the items and actions categories but want to propose a different alternate category. I would also prefer to visualize it in this way.


OCGM Category Matrix

As you can see, Objects are literal items, and Containers are symbolic items. Manipulations are literal actions, and Gestures are symbolic actions. Items are elements of the interface, and Actions are ways to ways to interact with the interface. Literal means it is real, direct, actually what you think it is. Symbolic means it is representing something else, such as the relationships between objects or a short-cut for several manipulations.

There is a beautiful symmetry around these concepts.

Fine, but what is OCGM a metaphor for?

An Object is the basic unit of content or data. If your application has a database, what are all the basic entities in the database? Most of those entities are good candidates for being an object. The metaphor is that each piece of unique content is represented in the interface by an individual, self-contained object. An Object has a one-to-one relationship with the content or data. Don't represent multiple copies of content with a single object. That is what Containers are for.

Containers are all about the relationship between objects (not necessarily just hierarchical.) Think about how the application's data is naturally organized. There could be several different ways to organize it. Perhaps certain content is naturally alphabetical (like contacts) or other content is only grouped by categories. The metaphor is that containers should present the Objects with the relationship type in mind. Note that an Object could have a Container within it, thus allowing the nesting of Objects.

Gestures represent an indirect, discrete, symbolic action. With Gestures, the important part is the actual movement of the Gesture. Here is one way to think about this: Gestures are the keyboard shortcuts of NUIs. A keyboard shortcut, such as CTRL-C, represents an action that occurs only after the shortcut is complete. The buttons CTRL and C are not directly related to the copy function, except through the shortcut. All of this is the same idea with Gestures.

Manipulations are direct, continuous actions with immediate feedback while the Manipulation is occurring. With Manipulations, the movement itself is not important (unlike Gestures) since you could take many paths to get to the same end result. To contrast with CTRL-C in the Gesture example, think about how in GUIs you could also find the menu selection or toolbar button for copy and click that. It may involve several individual clicks, mouse movements, or drags to perform. All the while you can explore to find what you want. In NUI, the Manipulation is similar. Manipulations should let you explore without unexpected consequences.

To make the metaphors a little more apparent, I made this table that shows what the complex ideas and the familiar ideas are.

OCGM Metaphors

Keep in mind that OCGM is just the foundation. Application designers will need to build upon OCGM and create specific metaphors based upon the nature of the application's content, or content relationships.

What about [XYZ] concept?

Several people have suggested other concepts that might be added to OCGM. Richard Monson-Haefel is concerned that Affordance and Feedback are not getting enough attention. One of my colleagues at work, Josh Wall, suggested Transformations and Interactions are important. Ben Reierson has suggested Exist Sense Intend Act Result (ESIAR, like easier) as an alternative.

I agree with them that these are all good things that need to be considered during interface design; however, none of these are metaphors! It would not make sense to include them in a foundational metaphor acronym that is meant to replace the phrase WIMP. Concepts like Affordance and Feedback and Transformation are important, but are important a different part of the design process than the metaphor selection. For example, Affordance would come in to play when picking a specific manipulation for a specific type of content. The visual would need to be designed to help the user realize they can manipulate the content in the intended way.

WIMP is to GUI as OCGM (Occam) is to NUI

The Graphical User Interface has WIMP -- Windows, Icons, Menus, Pointing Devices -- as it's foundational metaphor. These four things permeate the GUI experience and serve as the base of all GUI interactions.

What is the equivalent for Natural User Interfaces? History will decide for sure, but based upon common NUI designs and applications so far and what is going on in deep-research at Microsoft and elsewhere, we might be able to figure out what it will be.

So far, the only common term is post-WIMP, used when referring to interfaces beyond GUI. That term doesn't reveal anything about what it is, only what it is not. We know that NUIs will not be based upon WIMP-y ideas.

There has been an on-going discussion of the actual definition of post-WIMP and there are several suggestions so far. There are really two things that are being discussed:

1. What are the metaphors that replace Windows, Icons, Menus, and Pointing Devices?
2. What is the best acronym?

The acronym seems like a minor thing but for mass acceptance it needs to be pronounceable and memorable. WIMP is both of those, even though it has a negative connotation.

I had a discussion on this with Jonathan Brill late last year. His opinion was that the term should be PATA - Places, Animations, Things, Auras. This was a good starting point, although terms were debatable and the acronym was not memorable.

I had my own ideas and modifications but wanted to check to see what people smarter than myself said.

August de los Reyes, Principal Director of User Experience for Surface Computing at Microsoft, has been giving a talk titled Predicting the Past for at least 18 months. It evolved and recently he briefly suggested TOCA - Touch, Objects, Containers, Actions, as the WIMP replacement. This can be seen in his ReMix 2009 presentation around the 19:15 mark:

[1/7/2010: Sorry, link and embedded video seems to be broken, so I removed the embedded video. Perhaps site maintenance. I will update if I find a working link.]

He only briefly mentions it and them moves on. Objects and Containers make sense, but specifying Touch seems to exclude any in-air free-form gestures like would be used in Natal or WiiMote Multi-touch style interfaces. In addition, Action could be interpreted in many ways, although I assume August intended that to refer to both gestures and manipulations.

I called up my new friend Ron George, who just left the Advanced Design Team at Microsoft to form his own NUI Design firm. It turns out that he was one of the people doing deep-research in Microsoft and has been laboring over this idea for a few months. Internally he had been playing with Manipulations, Gestures, Objects, and Containers as the post-WIMP concepts, and was actually in the process of writing a blog post on the topic when I called him. We spoke for a few hours and he suggested that Gestures and Manipulations are both important enough to be included as proper terms. The definitions of gestures and manipulations include the possibility of both touch and free-form input. All the other ideas like auras and animations become part of the behavior of an object, container, gesture, or manipulation. Ron had some genius insights based upon his experience within various Microsoft teams and he got me on board with those concepts.

Now the only remaining problem was the finding out the right order for the acronym that is pronounceable, memorable, and has a nice double meaning. I wrote out all the permutations of G, M, O, and C and found one that stood out as a beautiful, elegant acronym. Therefore; based upon several months of discussion and taking into account input from various experts in the field, we submit to the NUI community the following term as the replacement for WIMP:

Objects, Containers, Gestures, and Manipulations, or OCGM, pronounced Occam, as in Occam's Razor.

Occam's Razor is a logical principle that can be summarized as "All else being equal, the simplest answer is usually correct." This is the beautiful part, because this concept can and should be applied to interface design. Given two design choices, the simplest one would be preferred. In this case, measuring simplicity would depend upon the context and could mean the cognitive load, visual design, metaphoric layers, or other ideas.

Ron wrote a post about OCGM on his blog. He also started a thread at nuigroup.com on this topic. In a future post I will go into more detail about what OCGM means to developers.

What are your thoughts?

[Updated phrasing and wording at 12/28/2009 3:10pm]