Thursday, December 31, 2009

Metaphors and OCGM

It really seems like OCGM has sparked a lot of discussion! Ron first described it two days ago, and already there have been several different bloggers posting their own ideas and responses to those blogs. Lynn has a good summary of the conversation so far, and the thread on also has a summary of links. I think overall the reception has been positive. There is a lot of enthusiasm for NUIs, and OCGM is just one way for people to understand NUIs better.

There have been some comments about Objects, Containers, Gestures, and Manipulations that seem to be the result of not fully understanding OCGM yet. It is a new concept, and not much has been publicly discussed yet. I want to help people understand better by explaining a bit about metaphors and OCGM.

What is a Metaphor?

A metaphor is the understanding of a complex idea in terms of another, well-understood idea. Here is an example:

Most people (myself included) do not understand the complexities of the financial markets. When the economic meltdown really got going in 2008, the media struggled with understanding what was going on and how to explain it to the layman. One term that was heard in almost every story about banks was the Toxic Asset.

The phrase "Toxic Asset" refers to financial assets with some fault that means they cannot be sold for a reasonable price. The word toxic is used to describe specific attributes of these complex financial tools. There is nothing literally toxic about them -- they're contracts and legal documents, and touching them won't poison you. Toxic waste and these financial assets have certain similar properties, and the metaphor of a "toxic" helped people understand the complex idea (financial assets) in terms of a well understood idea (toxic waste.) If you had it, you want to get rid of it, but no one else wants to take it from you.

Metaphors are used literally all the time in every day language and activities and are core to the human experience. The word "core" in the previous sentence is a metaphor to help you understand the relationship between metaphors and the human experience. Metaphors are not actually inside of the human experience, in the way the core of a planet is inside the planet. But both the planet core and metaphors are important parts of the things they are metaphorically the core of. For more on this, I really recommend reading about Conceptual Metaphors and Conceptual Blending on Wikipedia.

What about metaphors and interfaces?

Computers are extremely complex things. The software running on the computer is also very complex. In order for non-computer geniuses to be productive at all, the human-computer interface has to help the user understand the complex parts of the software in terms of something the user understands.

Even though some people claim that NUIs will eliminate metaphors, that is not completely accurate. NUIs will still have metaphors (unless you want to interact directly with binary electrical pulses), but the NUI metaphors are based around a different core set of values. The NUI metaphors will be less complex than GUI metaphors. (There we go with Occam's Razor again!) This will result in easier to use applications and more enjoyable user experiences.

Even the command line interface had a few metaphors. It had the keyboard input, which to early users was similar to the typewriter. It had folders and files, which are familiar to office workers. It had the command prompt, commands, and responses, which are similar to the interaction with service industry workers: "What can I do for you today?"

A command line interface waiting at the prompt for your input.

We all know and have discussed ad nauseum that GUI's basic metaphors are Windows, Icons, Menus, and Pointers, so I will skip discussion of those.

Is OCGM a metaphor?

You can probably guess by now where I'm going with this. Objects, Containers, Gestures, and Manipulations are each metaphors, and they will help us understand and interact with software in a different way than before.

Aside from the interface, an application that transitions from GUI/WIMP to NUI/OCGM could be exactly the same. It can still have the same databases and same business logic and same web services and everything else. The new NUI interface will be designed using OCGM as the foundational metaphors. There will be other specific metaphors based upon the application content and interactions, but those metaphors build upon Objects, Containers, Gestures, and Manipulations.

What do Objects, Containers, Gestures, and Manipulations have to do with each other?

Ron described in his latest post, Welcome to the OCGM Generation! Part 2 that OCGM has two categories (items and actions) and two subcategories (base and complex.)

I agree with the items and actions categories but want to propose a different alternate category. I would also prefer to visualize it in this way.

OCGM Category Matrix

As you can see, Objects are literal items, and Containers are symbolic items. Manipulations are literal actions, and Gestures are symbolic actions. Items are elements of the interface, and Actions are ways to ways to interact with the interface. Literal means it is real, direct, actually what you think it is. Symbolic means it is representing something else, such as the relationships between objects or a short-cut for several manipulations.

There is a beautiful symmetry around these concepts.

Fine, but what is OCGM a metaphor for?

An Object is the basic unit of content or data. If your application has a database, what are all the basic entities in the database? Most of those entities are good candidates for being an object. The metaphor is that each piece of unique content is represented in the interface by an individual, self-contained object. An Object has a one-to-one relationship with the content or data. Don't represent multiple copies of content with a single object. That is what Containers are for.

Containers are all about the relationship between objects (not necessarily just hierarchical.) Think about how the application's data is naturally organized. There could be several different ways to organize it. Perhaps certain content is naturally alphabetical (like contacts) or other content is only grouped by categories. The metaphor is that containers should present the Objects with the relationship type in mind. Note that an Object could have a Container within it, thus allowing the nesting of Objects.

Gestures represent an indirect, discrete, symbolic action. With Gestures, the important part is the actual movement of the Gesture. Here is one way to think about this: Gestures are the keyboard shortcuts of NUIs. A keyboard shortcut, such as CTRL-C, represents an action that occurs only after the shortcut is complete. The buttons CTRL and C are not directly related to the copy function, except through the shortcut. All of this is the same idea with Gestures.

Manipulations are direct, continuous actions with immediate feedback while the Manipulation is occurring. With Manipulations, the movement itself is not important (unlike Gestures) since you could take many paths to get to the same end result. To contrast with CTRL-C in the Gesture example, think about how in GUIs you could also find the menu selection or toolbar button for copy and click that. It may involve several individual clicks, mouse movements, or drags to perform. All the while you can explore to find what you want. In NUI, the Manipulation is similar. Manipulations should let you explore without unexpected consequences.

To make the metaphors a little more apparent, I made this table that shows what the complex ideas and the familiar ideas are.

OCGM Metaphors

Keep in mind that OCGM is just the foundation. Application designers will need to build upon OCGM and create specific metaphors based upon the nature of the application's content, or content relationships.

What about [XYZ] concept?

Several people have suggested other concepts that might be added to OCGM. Richard Monson-Haefel is concerned that Affordance and Feedback are not getting enough attention. One of my colleagues at work, Josh Wall, suggested Transformations and Interactions are important. Ben Reierson has suggested Exist Sense Intend Act Result (ESIAR, like easier) as an alternative.

I agree with them that these are all good things that need to be considered during interface design; however, none of these are metaphors! It would not make sense to include them in a foundational metaphor acronym that is meant to replace the phrase WIMP. Concepts like Affordance and Feedback and Transformation are important, but are important a different part of the design process than the metaphor selection. For example, Affordance would come in to play when picking a specific manipulation for a specific type of content. The visual would need to be designed to help the user realize they can manipulate the content in the intended way.


  1. Excellent article! You've helped me understand that Affordance and Feedback , although critical to HCI design, should not be part of the OCGM. After all, Affordance and Feedback are not included in WIMP, but are still critical the GUI design process. OCGM is a higher level abstraction. Affordances, Feedback, and Transitions are addressed at a lower, more concrete, levels of abstraction. Nice work!

  2. Great post. I'm definitely liking OCGM more and more as the conversation continues. My initial concerns about the concept of defining a WIMP for NUI as a potentially useless exercise were likely unfounded. It definitely could be useful as a way to help explain NUI interfaces. Technically you could use OCGM to describe interaction with ANY designed interface, digital or not, but that doesn't prevent it from being useful.

    And to be clear, I'm totally on board with Ron's definitions for Gesture vs. Manipulation, and if his experience shows that specifying them at this level is going to be helpful, then he's probably right. :)

  3. Thanks for the comments!

    Ben, that's an interesting observation. OCGM is more abstract than WIMP, and that lets it be more flexible. WIMP interfaces are generally built with the specific metaphors in mind, and that restricts the designer's thinking.

    OCGM has enough definition to guide the design, but does not prescribe a specific visualization or method for solving a problem. The GUI design patterns are thrown out and the designers and developers are free to implement the best interface for the specific situation, while keeping in mind that the users need to quickly learn the interface. I think there are some NUI values or principles needed here to guide some of those decisions, and that is where the discussion of affordance and feedback and other things will go.

    And as you said, that does mean that you could potentially use OCGM to describe any interface, including some of the more advanced WIMP interfaces.

  4. Hi Joshua,

    Even if it's still a work in progress, I really like this OCGM concept. Well, to be honest, I would replace "Containers" by "Relations" since I understand "Containing" as "Embedding", which seems to me a bit restrictive. But I'm not a native english speaker so don't care about this :D

    I like the way you define OCGM as an abstract and flexible "pattern". May be we can imagine OCGM as a high-level metaphor, which will be specialized by some others metaphors according to the needs.
    I really think we now need more than one metaphor (the desktop) for our interfaces since computers are everywhere (not only in offices) and are used by more and more people not working in offices.

    Best regards.

    PS : +1 for your description of gestures as keyboard shortcuts.

  5. Laurent,

    I'm glad you like it.

    You're right that there are multiple layers of metaphors. The desktop or other metaphors (note: the desktop is a container!) are more concrete. The power of NUI and OCGM will be the freedom to develop and use metaphors that fit the need of the specific content or interactions. OCGM provides the pattern, as you say, so that users can understand and use different applications even if they use different content-specific metaphors.

    On Containers/Relations, you're right that the containing implies embedding. That is the metaphor which will explain the relationships between objects. The visualization doesn't necessarily need to be literal containment, it just need to be obvious that the Objects that have a relationship are part of a larger whole, and the visual and behavior of that larger whole (the Container) helps the user understand the specific type of relationship between the objects.

    Please keep the questions coming, and point out anything that doesn't make sense. It really helps me figure out how to better explain the ideas.


  6. Hi,

    I feel ok with OCGM concept since it's very closed to my own vision of how we should try to design new kind of applications.

    May be one more thought/question : I really agree with your description of gestures as shortcuts or symbolic actions, but can we really say gestures are natural since gestures imply a learning curve which may be long ?
    I know that following is not really "mainstream", but according to me, it would be better to consider gestures as being part of another "paradigm" than NUI.

    A few months ago, Richard wrote a post on his blog about Supernatural User Interfaces. Well, I would say that if manipulations are natural, gestures are supernatural. When I use manipulations, I'm myself. When I use gestures, I'm Harry Potter ;D

    I don't mean (at all) you should break OCGM. These 4 concepts are coherent and I think they're usefull.

    Even if I like the term NUI, sometimes I think it's too generic. Actually, I try to organize my thoughts about something I call "Fun User Interfaces" (don't know a better translation for the french word "Ludique") and OCGM is very related to this.

    Sorry for my bad english. I hope you'll understand what I tried to explain.


  7. @Laurent
    I'm sure Josh will respond as well, but I definitely think gestures can be considered natural. They may be less intuitive than direct manipulation, but we use gestures all the time in the real-world. One of the top definitions of the word gesture is "something done as an indication of intention". Think about waving to someone. By waving, you are communicating a friendly greeting. This is a learned set of movements that communicate a particular meaning after they are completed. It's not a direct manipulation, but it's natural once it's learned. It's just a slightly higher-level form of communication.

  8. @Laurent:

    Your English is fine! No problem understanding you.

    You're right that an application with a lot of advanced gestures may seem magical. Gestures are natural things. People fluent in sign language seem magical to me.

    Just think about how much people gesticulate (make gestures while speaking), or how people use their hands to communicate emotions. It is very natural to communicate using both words and gestures. We just aren't used to doing that with computers yet.

    On intuitive -- keep in mind that gestures aren't necessarily meant to be intuitive. Just like keyboard shortcuts, they're meant to be an advanced feature that power users will use. (Tom Cruise in Minority Report was a power user.) Most commands you can do with gestures should probably be also accessible through manipulations as well. Gestures are quicker than manipulations, but manipulations are easier to learn.

    Fun is also another important concept. August de los Reyes says "Play is not the opposite of Work" in his talk "Predicting the Past." NUIs, with the focus on fluidity and high frequency interaction, can be very fun. It's all about those pattern matchers and pleasing the brain.

  9. @Ben, @Josh
    I agree with 99% of your answers but I still have doubts about gestures and NUI. Here are the reasons :
    - 2 main values used to define NUI are natural and intuitive.
    - we agree to say gestures are not intuitive since they require to be learned.
    - I agree to say gestures, used to communicate with people, are "natural" (common sense of this word : the things we used to do, ...). But if we consider the main sense of this word (something related to nature, to all beings or to all things) I don't think we can say gestures are natural. For example : a specific gesture doesn't mean anything in France but will be considered as a strong insult in Italy ;D
    - Right now (but may be I'm wrong) I'm unable to find a good example of gesture to interact with an object (not a person) in our everyday life. I'm only able to think about examples in books or movies and related to magic (supernatural).

    So (with great help of Ben with definition of natural as things we have learned) I would say that :
    - gestures should be a good thing for UI, even if they're not natural (main sense / marketing term)
    - if we're able to define good gestures, they'll become natural (common sense).
    - the best indicator is the learning curve (one of the conditions I want to use to define a Fun User Interface).

    I totally agree with the statement : "Play is not the opposite of Work". In fact, A. de los Reyes's talk convinced me that idea of FUI could be a good and serious guidance for my works. I also see FUI as complementary to NUI and not opposite at all. I'll try to explain in a blog post what I mean with FUI. I'll be glad to get all your comments and advices about it.

    Thanks for this interesting discussion !

  10. Edit : please read "...(common sense of this word : the things we're used to do, ...)"

  11. Hello,

    Why no one actually mentions the context as an important factor of NUI based interaction? While present in WIMP (in a way) it's still influencing e.g. the meaning of gestures etc.

    Besides, WIMP definition, while very focused, represents computer-side point of view. In my opinion OCGM is mixed with user-side a bit. Clearing this up might reduce the value of a gesture and bring the affordances once again to the table.


  12. @Laurent:

    I agree, this is a very interesting discussion. I think this thread is clearly a record for number of comments on a post for my blog. :)

    I'm planning out a few more posts that break down the individual components of OCGM and explain some of the background, like I did with metaphors in this post. I'd like to see your post on FUI ideas.

    In the meantime, consider this:

    According to Mehrabian, non-verbal communication accounts for 93% of how humans interpret the speaker's attitude. In other words, humans place much more value on non-verbal information than verbal.

    Do you consider face to face communication between humans natural?

  13. @Stan:

    Thanks for your comment.

    Context is important in interface design ingeneral and especially in NUIs. Just like natural, intuitive, directness, affordance, feedback, and other concepts, those fall under NUI values. This particular post was just about OCGM and Metaphors, and no one is claiming that OCGM is the only important NUI concept.

    I don't quite understand what you mean about computer-side and user-side. In WIMP, the Windows, Icons, and Menus are interface element metaphors and the Pointing Devices is how users interact with those elements (besides keyboard.) Similarly, in OCGM, OC are interface element metaphors and the GM are how you interact with Objects and Containers.

  14. @Josh

    You're right ! I'm too talkative :D

    I totally agree with you that non-verbal communication can be considered as natural.

    But i really think the key point is that Natural means 2 different things :
    MAIN SENSE : something related to nature, to all beings or to all things.
    COMMON SENSE : something we've learned and we're used to do.

    Keeping in mind this, I would say that :
    P1- using gestures to communicate with a person is natural (MAIN SENSE), since it's an "activity" shared by all human beings.
    P2- using a specific gesture to communicate with a person is natural (COMMON SENSE), since I had to learn this gesture and its meaning before using it.
    P3- using gestures to interact with object is not natural. I can't find an example.

    Then, I would say that :
    P4- using gestures to interact with objects in Touch UI will be a good thing (my own opinion taking into account P1).
    P5- if we're able to define good gestures, they'll become natural (COMMON SENSE).

    May be you wonder why so many questions about this term "Natural" ? :D
    Well, I was thinking about it before, but after this discussion I'm convinced that remembering we work (as software designers, HCI experts, ...) on natural things (COMMON SENSE) will be the key to define good natural UI.
    If we remember this, we'll have to ask ourselves :
    - why do I think this gesture (manipulation, ...) is natural ?
    - when do I learned to do this ?
    - how do I learned to do this ?
    - will it be natural for end users ? (Who are my end users ? Did they already learned this or will they have to learn it ?)

    Here's an example :
    When I started to think about multitouch development, one of my first question was "How do I close something ?".
    Common solutions used in some others apps were :
    - a "X" icon at the top right corner of the object.
    - a "X" gesture.

    Asking myself "What does it mean to close a window in GUI ?", my best answer was "I want to make disapear this thing".
    We could note it like this :
    "to close" (GUI metaphor) == "to make disapear" (REAL LIFE)
    "to close" (GUI metaphor) != "to close" (REAL LIFE)
    So my conclusion was that the most natural way to "close" an object in NUI is throwing this object on a side of the screen to make it disapear.
    And I was very happy when sometimes later I watched this video ( at 0'35) showing a NUI app developed by another french developer.

    But after more reflexions I think it's not so easy. This manipulation is not very productive, so it needs a gesture as shortcut. More important, if this manipulation seem natural to beginers (users without any prior experience of computers) I'm quite convinced it won't be natural for GUI users who are used to click on "X" icon.
    And this lead us to another question : should we take into account GUI experience to define NUI or should we state that GUI users will have to learn new things ?

    Definitly a long discussion ! :D

  15. EDIT : please read in previous post :
    "If we remember this, we'll have to ask ourselves :
    - why do I think this gesture (manipulation, ...) is natural ?
    - when did I learn to do this ?
    - how did I learn to do this ?
    - will it be natural for end users ? (Who are my end users ? Did they already learn this or will they have to learn it ?)"

    Sorry for all mistakes :(

  16. @Laurent:

    I was in the process of responding back to you when I saw Bill Buxton has some thoughts that are infinitely better. See my latest post for my notes and the video. I think he will address some of your thoughts -- almost personally it seems!

    Also forgive me for asking, I know two Laurents involved in multi-touch, and I don't know which one you are! What is your last name (or blog or twitter) so I can differentiate you? Thanks! :)

  17. @Josh :
    Thanks for notes and videos ! I''m sure they'll be precious.
    My twitter account is "LaurentMT".
    My email address is


Note: Only a member of this blog may post a comment.