Metaphors and OCGM

It really seems like OCGM has sparked a lot of discussion! Ron first described it two days ago, and already there have been several different bloggers posting their own ideas and responses to those blogs. Lynn has a good summary of the conversation so far, and the thread on nuigroup.com also has a summary of links. I think overall the reception has been positive. There is a lot of enthusiasm for NUIs, and OCGM is just one way for people to understand NUIs better.

There have been some comments about Objects, Containers, Gestures, and Manipulations that seem to be the result of not fully understanding OCGM yet. It is a new concept, and not much has been publicly discussed yet. I want to help people understand better by explaining a bit about metaphors and OCGM.

What is a Metaphor?

A metaphor is the understanding of a complex idea in terms of another, well-understood idea. Here is an example:

Most people (myself included) do not understand the complexities of the financial markets. When the economic meltdown really got going in 2008, the media struggled with understanding what was going on and how to explain it to the layman. One term that was heard in almost every story about banks was the Toxic Asset.

The phrase "Toxic Asset" refers to financial assets with some fault that means they cannot be sold for a reasonable price. The word toxic is used to describe specific attributes of these complex financial tools. There is nothing literally toxic about them -- they're contracts and legal documents, and touching them won't poison you. Toxic waste and these financial assets have certain similar properties, and the metaphor of a "toxic" helped people understand the complex idea (financial assets) in terms of a well understood idea (toxic waste.) If you had it, you want to get rid of it, but no one else wants to take it from you.

Metaphors are used literally all the time in every day language and activities and are core to the human experience. The word "core" in the previous sentence is a metaphor to help you understand the relationship between metaphors and the human experience. Metaphors are not actually inside of the human experience, in the way the core of a planet is inside the planet. But both the planet core and metaphors are important parts of the things they are metaphorically the core of. For more on this, I really recommend reading about Conceptual Metaphors and Conceptual Blending on Wikipedia.

What about metaphors and interfaces?

Computers are extremely complex things. The software running on the computer is also very complex. In order for non-computer geniuses to be productive at all, the human-computer interface has to help the user understand the complex parts of the software in terms of something the user understands.

Even though some people claim that NUIs will eliminate metaphors, that is not completely accurate. NUIs will still have metaphors (unless you want to interact directly with binary electrical pulses), but the NUI metaphors are based around a different core set of values. The NUI metaphors will be less complex than GUI metaphors. (There we go with Occam's Razor again!) This will result in easier to use applications and more enjoyable user experiences.

Even the command line interface had a few metaphors. It had the keyboard input, which to early users was similar to the typewriter. It had folders and files, which are familiar to office workers. It had the command prompt, commands, and responses, which are similar to the interaction with service industry workers: "What can I do for you today?"


A command line interface waiting at the prompt for your input.

We all know and have discussed ad nauseum that GUI's basic metaphors are Windows, Icons, Menus, and Pointers, so I will skip discussion of those.

Is OCGM a metaphor?

You can probably guess by now where I'm going with this. Objects, Containers, Gestures, and Manipulations are each metaphors, and they will help us understand and interact with software in a different way than before.

Aside from the interface, an application that transitions from GUI/WIMP to NUI/OCGM could be exactly the same. It can still have the same databases and same business logic and same web services and everything else. The new NUI interface will be designed using OCGM as the foundational metaphors. There will be other specific metaphors based upon the application content and interactions, but those metaphors build upon Objects, Containers, Gestures, and Manipulations.

What do Objects, Containers, Gestures, and Manipulations have to do with each other?

Ron described in his latest post, Welcome to the OCGM Generation! Part 2 that OCGM has two categories (items and actions) and two subcategories (base and complex.)

I agree with the items and actions categories but want to propose a different alternate category. I would also prefer to visualize it in this way.


OCGM Category Matrix

As you can see, Objects are literal items, and Containers are symbolic items. Manipulations are literal actions, and Gestures are symbolic actions. Items are elements of the interface, and Actions are ways to ways to interact with the interface. Literal means it is real, direct, actually what you think it is. Symbolic means it is representing something else, such as the relationships between objects or a short-cut for several manipulations.

There is a beautiful symmetry around these concepts.

Fine, but what is OCGM a metaphor for?

An Object is the basic unit of content or data. If your application has a database, what are all the basic entities in the database? Most of those entities are good candidates for being an object. The metaphor is that each piece of unique content is represented in the interface by an individual, self-contained object. An Object has a one-to-one relationship with the content or data. Don't represent multiple copies of content with a single object. That is what Containers are for.

Containers are all about the relationship between objects (not necessarily just hierarchical.) Think about how the application's data is naturally organized. There could be several different ways to organize it. Perhaps certain content is naturally alphabetical (like contacts) or other content is only grouped by categories. The metaphor is that containers should present the Objects with the relationship type in mind. Note that an Object could have a Container within it, thus allowing the nesting of Objects.

Gestures represent an indirect, discrete, symbolic action. With Gestures, the important part is the actual movement of the Gesture. Here is one way to think about this: Gestures are the keyboard shortcuts of NUIs. A keyboard shortcut, such as CTRL-C, represents an action that occurs only after the shortcut is complete. The buttons CTRL and C are not directly related to the copy function, except through the shortcut. All of this is the same idea with Gestures.

Manipulations are direct, continuous actions with immediate feedback while the Manipulation is occurring. With Manipulations, the movement itself is not important (unlike Gestures) since you could take many paths to get to the same end result. To contrast with CTRL-C in the Gesture example, think about how in GUIs you could also find the menu selection or toolbar button for copy and click that. It may involve several individual clicks, mouse movements, or drags to perform. All the while you can explore to find what you want. In NUI, the Manipulation is similar. Manipulations should let you explore without unexpected consequences.

To make the metaphors a little more apparent, I made this table that shows what the complex ideas and the familiar ideas are.

OCGM Metaphors

Keep in mind that OCGM is just the foundation. Application designers will need to build upon OCGM and create specific metaphors based upon the nature of the application's content, or content relationships.

What about [XYZ] concept?

Several people have suggested other concepts that might be added to OCGM. Richard Monson-Haefel is concerned that Affordance and Feedback are not getting enough attention. One of my colleagues at work, Josh Wall, suggested Transformations and Interactions are important. Ben Reierson has suggested Exist Sense Intend Act Result (ESIAR, like easier) as an alternative.

I agree with them that these are all good things that need to be considered during interface design; however, none of these are metaphors! It would not make sense to include them in a foundational metaphor acronym that is meant to replace the phrase WIMP. Concepts like Affordance and Feedback and Transformation are important, but are important a different part of the design process than the metaphor selection. For example, Affordance would come in to play when picking a specific manipulation for a specific type of content. The visual would need to be designed to help the user realize they can manipulate the content in the intended way.

WIMP is to GUI as OCGM (Occam) is to NUI

The Graphical User Interface has WIMP -- Windows, Icons, Menus, Pointing Devices -- as it's foundational metaphor. These four things permeate the GUI experience and serve as the base of all GUI interactions.

What is the equivalent for Natural User Interfaces? History will decide for sure, but based upon common NUI designs and applications so far and what is going on in deep-research at Microsoft and elsewhere, we might be able to figure out what it will be.

So far, the only common term is post-WIMP, used when referring to interfaces beyond GUI. That term doesn't reveal anything about what it is, only what it is not. We know that NUIs will not be based upon WIMP-y ideas.

There has been an on-going discussion of the actual definition of post-WIMP and there are several suggestions so far. There are really two things that are being discussed:

1. What are the metaphors that replace Windows, Icons, Menus, and Pointing Devices?
2. What is the best acronym?

The acronym seems like a minor thing but for mass acceptance it needs to be pronounceable and memorable. WIMP is both of those, even though it has a negative connotation.

I had a discussion on this with Jonathan Brill late last year. His opinion was that the term should be PATA - Places, Animations, Things, Auras. This was a good starting point, although terms were debatable and the acronym was not memorable.

I had my own ideas and modifications but wanted to check to see what people smarter than myself said.

August de los Reyes, Principal Director of User Experience for Surface Computing at Microsoft, has been giving a talk titled Predicting the Past for at least 18 months. It evolved and recently he briefly suggested TOCA - Touch, Objects, Containers, Actions, as the WIMP replacement. This can be seen in his ReMix 2009 presentation around the 19:15 mark:

[1/7/2010: Sorry, link and embedded video seems to be broken, so I removed the embedded video. Perhaps site maintenance. I will update if I find a working link.]

He only briefly mentions it and them moves on. Objects and Containers make sense, but specifying Touch seems to exclude any in-air free-form gestures like would be used in Natal or WiiMote Multi-touch style interfaces. In addition, Action could be interpreted in many ways, although I assume August intended that to refer to both gestures and manipulations.

I called up my new friend Ron George, who just left the Advanced Design Team at Microsoft to form his own NUI Design firm. It turns out that he was one of the people doing deep-research in Microsoft and has been laboring over this idea for a few months. Internally he had been playing with Manipulations, Gestures, Objects, and Containers as the post-WIMP concepts, and was actually in the process of writing a blog post on the topic when I called him. We spoke for a few hours and he suggested that Gestures and Manipulations are both important enough to be included as proper terms. The definitions of gestures and manipulations include the possibility of both touch and free-form input. All the other ideas like auras and animations become part of the behavior of an object, container, gesture, or manipulation. Ron had some genius insights based upon his experience within various Microsoft teams and he got me on board with those concepts.

Now the only remaining problem was the finding out the right order for the acronym that is pronounceable, memorable, and has a nice double meaning. I wrote out all the permutations of G, M, O, and C and found one that stood out as a beautiful, elegant acronym. Therefore; based upon several months of discussion and taking into account input from various experts in the field, we submit to the NUI community the following term as the replacement for WIMP:

Objects, Containers, Gestures, and Manipulations, or OCGM, pronounced Occam, as in Occam's Razor.

Occam's Razor is a logical principle that can be summarized as "All else being equal, the simplest answer is usually correct." This is the beautiful part, because this concept can and should be applied to interface design. Given two design choices, the simplest one would be preferred. In this case, measuring simplicity would depend upon the context and could mean the cognitive load, visual design, metaphoric layers, or other ideas.

Ron wrote a post about OCGM on his blog. He also started a thread at nuigroup.com on this topic. In a future post I will go into more detail about what OCGM means to developers.

What are your thoughts?

[Updated phrasing and wording at 12/28/2009 3:10pm]

Pleasing The Brain with Magical Interfaces

I just read an extremely interesting blog by David Sherwin called Data, Delight, and User Experience Design. David talks about the emotion and feeling necessary for magic tricks to actually evoke the feeling of magic. He discusses some websites that have this quality, such as Google's I'm Feeling Lucky, Wikipedia's Random Article, and Flickr's Interesting feature.

Richard Monson-Haefel has also written about Magic Interfaces on his blog, and even started a separate website promoting the idea of Magic as a metaphor for Natural User Interface design, although his approach is a little more literal with roots in magic from fantasy stories (incantations, runes, artifacts.)

I think Richard has a good idea at the high-level, but the Magic metaphor needs to be a little more concrete for developers and designers to implement it. David's idea of designing to evoke the emotion of Magic is a little easier to do than translating fantasy magical metaphors.

Here is my take on implementing a magical metaphor, building upon the ideas above.

Human brains are pattern matching machines. Once you get past the brainstem and low-level wetware regulating our heartbeat and breathing and reflexes, the majority of our brains are dedicated to pattern matching. We detect patterns and meta-patterns and correlate them in several layers just to have vision. That feeds into higher level pattern matching that helps us do things like walk and talk at the same time without tripping.

Even higher up in our consciousnesses we put together patterns and experience in order to predict what we think will happen in the future. That allows us to anticipate the future position of a spinning ball undergoing projectile motion in the influence of wind so that we can start swinging a bat at the precise time to make contact and hit it as far as we can. At least, the professional baseball players can do that, because they have the experience from practice.

The rest of us can use that brain circuitry to, among other things, anticipate the consequences of clicking a mouse and typing keys and touching screens in order to get computing devices to do what we want. We do this dozens of times a second or more. Just watch some eye tracking videos to see how quickly our eyes can move, each time evaluating what we see and deciding if we should take action.



Our brain cells get a little kick out of every pattern they match. That's their motivation. If you design an interface that is intuitive and the users can easily predict, then as they use it, their little pattern matchers will get lots of jolts of happy juice and the user will like using the application!

On the other hand, if the user predicts that something will happen and it doesn't, then those brains cells responsible will be held accountable. Actually, they'll just get rewired a little bit and learn, and the user will get annoyed by the lack of happy juice.

Let's bring back the magic here. How can we design an interface to maximize the user's enjoyment, specifically by matching as many patterns as possible? This is exactly what magicians do during their acts. David's article talks about this, but I want to approach it a little differently.

I want to borrow a phrase from the movie "The Prestige". Watch the trailer below from about 1:05 to 1:50.



Approximate quote from the actual movie:

Every great magic trick consists of three parts or acts. The first part is called "The Pledge." The magician shows you something ordinary: a deck of cards, a bird or a man. He shows you this object. Perhaps he asks you to inspect it to see if it is indeed real, unaltered, normal. But of course it probably isn't. The second act is called "The Turn." The magician takes the ordinary something and makes it do something extraordinary. Now you're looking for the secret, but you won't find it, because of course you're not really looking. You don't really want to know. You want to be fooled. But you wouldn't clap yet. Because making something disappear isn't enough; you have to bring it back. That's why every magic trick has a third act, the hardest part, the part we call "The Prestige."

Now the analogy isn't perfect, but here is how this fits in:

1. The Pledge - This is where the context is set up. The brain establishes the expectation or prediction here. If the trick is making a coin disappear, then first you have to show the coin exists. If the pressing a button is supposed to make the interface do something cool, then there has to be a button with appropriate icon or text to set that expectation. "That button is labeled 'Push for Awesome' so it will probably do something awesome."


2. The Turn - This is where the conflict or tension is established. For the magician, it is the moment after he made the coin disappear and the audience wonders if and how it will come back. For the computer user, it is the moment between deciding to perform an action and seeing the result. "I wonder if it will be awesome if I push the button?"


3. The Prestige - This is where the tension is resolved. The coin returns and the prediction of the audience ("coins cannot vanish") is confirmed. The interface does something awesome and the user's brain is flooded with happy. "Awe-sooome."

The greater the tension is, the more the reward will be. This doesn't mean you should make the users wait before the interface responds. Rather, it means that the larger the conflict between the Pledge that a button will do as it says and any disbelief that it will work as advertised, the more pleased the user will be when he or she finds that it does work.

Over time, the disbelief and thus the magic will go away, but our brains will still get little rewards for predicting and being correct. Peek-a-boo follows this pattern and is a highly-pleasurable activity for babies and toddlers, but eventually the fun decreases. This is also true with the Magic trick. If your job was to watch a coin trick several times a day, five days a week (as often as you check your email perhaps), would you still be as amazed the 100th time as you were the 1st time? No, but there is still a little bit of tension inside the brain. "Will it still work?" After all, we still get pleasure out of re-watching good movies, even though we know the ending, and thus the resolution of the core plot conflicts.

We can try to keep things fresh in certain ways though, by keeping the pattern but varying the inputs. (For those who have seen The Prestige, the New Transported Man was the same general idea as the Original Transported Man, but used a different technique and flourish.) Google's "I'm Feeling Lucky" button does the same thing every time -- direct you to the first result -- but the replay value comes in the game of "Will Google find the best result for this query, too?"

So as you use your computer, think about what actions you enjoy doing the most and what the Pledge, Turn, and Prestige is for them. Figure out how you can apply that to your own interface designs.

Neuron image courtesy brainmaps.org

Deconstructing: SilverPAC's Multi-Touch Website

[Update 11/9/2009 to reflect updates to the SilverPac site.]

SilverPAC just launched a multi-touch website using Silverlight 3.

http://www.silverpac.us.com/multitouch/

There also a bit of write-up about the development of the site by Ciplex in Los Angelas, CA in the press release. They claim it is "The World's First Multi-Touch Interactive Website."

Below is a video for those who don't have multi-touch hardware:



Although when I first read about this, I had a healthy dose of skepticism, after watching the video and in particular trying out the website myself (on my new HP tx2z) I think this was pretty well implemented. I was particularly impressed because I know that the multi-touch APIs available in Silverlight 3 are pretty limited (particularly compared to WPF 4 Touch.) Here are a few notes on their implementation:

• Introduction animation - When first loading the site, it shows an 8 or 10 second animation that demonstrates the gestures you can use to navigate the website. Interestingly, this intro animation is in Flash whereas the rest of the site is in Silverlight. This introduction is critical to orient the user and show how the multi-touch gestures will work. They are illustrated by using auras identical to those used in the application.
• Limited gesture vocabulary - The site only uses four gestures: Tap (for selecting a "page"), Pan Up/Down (for text scrolling), Two-finger swipe (for changing "pages"), and Pinch in (for zooming out). I'm glad they only have a few gestures, and they all make sense. I have discussed limiting the multi-touch vocabulary, and Ciplex seems to have done that. The only thing I would have changed is adding the Pinch out gesture to allow zooming in, duplicating the behavior of tapping on a page. This provides symmetry and would be more intuitive. Update: Pinch out was added.
• Auras - They implemented their own auras within Silverlight. This is important for users to see whether their touches are being recognized by the application. Not a full implementation of Microsoft Surface SP1's auras described in the Ripples research paper, but good anyway.
• Page layout - The site consists of nine pages with simple free form navigation from one to another. Users can zoom to the large view, either through gestures or the mini-map icon in the bottom left, then select a new page, or use two-finger swipe to change pages. With a more complicated site and more pages, a more complicated navigation scheme, or multi-layer zoom, would be needed.
• Pages - The pages on this website are more like cards of data than hyperlinked pages. There are no links internally, only to external sites.
• Mini-map - When zoomed in, there is an subtle icon in the bottom left corner. It serves two purposes - remind the user where they are in the page layout, and act as a link back to the zoomed out view. This is important for users who navigate the page with a mouse.

There are a few things that could be improved. They are mostly implementation issues rather than design.

• Add pinch out, as discussed above.
• The auras often get stuck, as if one touch is still down. This prevents some of the gestures from working unless you reload the page. Seems like if a touch or gesture is done too quickly this occurs.
• The page content and headers could be sized so the user could read them better from the large view. The interface zooms anyway, so it might as well make use of the space.
• Some pages, such as the top left, have too much text and no indication of the subject of that text. Scroll down far enough and you'll find out the names of the CEO and others, but how would the the user know to look there? Also, the bottom center page is blank. Either put some content there or don't let the user navigate there. Update: The pages were updated to rearrange the text, and allow switching columns within a page.
• Aura performance is laggy and has a slow frame rate. The animation of the page transitions and product has good performance, though. Update: Aura performance was improved and I haven't seen any more stuck auras.
• Although the user can navigate the pages using just the mouse, they cannot do the equivalent of the two-finger swipe to change between zoomed-in pages. This may illustrate an advantage of touch, but for users with no touch hardware, it seems like an artificial constraint and switching back to the zoomed out view to switch every page is slow.

Overall, even though there could be a few improvements made , I think the SilverPAC Multi-Touch website is a fine example of a multi-touch web implementation. (Update: Many of my suggested improvements were implemented since my original review.) The bar of excellence will be higher in the future, of course, but this is a good start at this point in the Natural User Interface Revolution. [Thanks to Gene on NUI Group Community Forums for posting a link to this.]
[Thanks x2 to Gene for letting me know about the site updates.]

Limiting the multi-touch vocabulary

On the classic Nintendo Entertainment System, and many other game systems, there was an extremely limited controller vocabulary.

• Four way direction-pad
• A
• B
• Select
• Start

The entire game library of NES, Gameboy, and several other game systems existed in a world with only these five controls. In fact, many games didn't even use all of them! I remember being surprised when a game uses the select button.

At first, it might seem that having so few controls would be a limitation and result in obscure and non-intuitive control schemes. On the contrary, game designers pay a huge amount of attention to usability and design their games so people can pick them up and use them without much thought. Otherwise, gamers would quickly lose patience and their games wouldn't sell.

With multi-touch systems, the possibilities of many touches and complicated gestures seems to blind designers and developers from the fact that you don't really need a rich vocabulary to navigate an application. There are already a few common gestures that are almost universally supported and used across all of the different multi-touch platforms:

• Tap
• Pan
• Pinch in/out
• Rotate

Using only these few gestures we could build hundreds of applications. In fact, most applications would only need Tap and Pan. (I'm assuming the keyboard would still be used, as necessary.)

Sure, in scenarios that need it, more complicated or custom gestures and manipulations could be used, just like how later game consoles added a few more buttons. But even the latest games use a few primary controls or buttons and the others are only used in special cases.

Before we get crazy, though, we should make an effort to really get the basic gestures right. Figure out what scenarios they should be used, and what metaphors work. Should Pinch In and Pinch Out be used for menu navigation? Does one or two finger rotation make more sense in which scenarios? The early game consoles had hardware and computational limitations that forced them to be creative and make intuitive interfaces. We shouldn't get caught up in all the possible gestures before getting the core set right.

The call to action here is when designing applications, think about the smallest set of multi-touch gestures that would accomplish the job. Can you design a more creative interface that only requires tap and pan? If so, then you might want to opt for that, since it requires the users to learn fewer gestures.

Hand Gesture diagrams adapted from Dan Saffer's graphics. Thanks for the contribution, Dan!

WPF 4 beta 2 Touch documentation

With the public availability of VS2010 beta 2, the MSDN documentation has also been updated.

Two pages that any touch developers should read are:

What's New in WPF 4: Touch and Manipulation

Walkthrough: Creating Your First Touch Application (Basic scatterview-like pan/rotate/stretch on a rectangle.)

It's pretty easy to get started. I actually wrote a program similar to the walkthrough (before finding the documentation) in about 30 minutes.

Multi-touch mice from Microsoft Research

There is article and video in the Seattle Times about some research into multi-touch mice done by Microsoft Research. There is a video that include four concept mice, each using a different concept and technology. This video is worth watching to see what kind of concepts they came up with.



Before watching the video, I was skeptical about the utility of a multi-touch mouse. Mostly I think that point of view was colored by the fact that I think multi-touch trackpads as being inferior to multi-touch displays. I assumed that a MT mouse would be pretty much the same, but after watching the video, I can see there is some advantage.

My problem with the MT trackpad is that it does not allow the NUI concept of direct manipulation. Sure, it is multi-touch and can recognize gestures, but those gestures are only a small incremental improvement to the scroll-sensitive side area of trackpads. With the trackpad is used to move the cursor, the finger movement is relative to the cursor. When the trackpad is used with a multi-touch gesture, it is also relative, but multi-touch gestures are much better using absolute positioning and direct manipulation. True Natural User Interfaces are not possible with MT trackpads. (Would the iPhone be as cool with a non-touch display and a separate MT trackpad and cursor?)

In the video, they showed that each concept multi-touch mouse has one (or more) regular mouse position sensor on the bottom but also a method to detect (many!) touches somewhere on the top. This is different from MT trackpads in that the cursor positioning is done with relative movement of the entire mouse, but multi-touch gestures are done using the fingers on the MT-sensitive surface. Some of the concept mice even responded to whole palm gestures and positioning.

The separation of the relative cursor positioning from the multi-touch gestures may be a solution to the issues I had with MT trackpads. In addition, it allows additional degrees of freedom so that you can rotate and stretch a picture using gestures while and panning by moving the mouse all at once, for example, whereas with the trackpad you would likely be limited to one type of manipulation at a time.

I can envision an ecosystem where multi-touch mice being accepted as a low-cost enabling technology that introduces people to Natural User Interfaces without a large investment. Those would be used in addition to or to complement multi-touch displays and larger Surface devices. It would benefit NUI adoption by allowing existing computers to use NUI software. While it would add to the complexity of planning interfaces across a larger variety of hardware capabilities, the potential for mass adoption may be more important.

Update: Picture of the mice courtesy of CrunchGear:

Microsoft Courier - leaked Tablet photos and video!

For a while now there have been mostly unsubstantiated rumors of an Apple tablet. It has generated enormous excitement and imagination, but Microsoft may steal all that.

Gizmodo has details of the leaked photos and concept video.

This is much more than just a photoshop or concept art. It is CG photos and a matching use case video of how the Courier tablet (more like paper notebook, hinge and all) would be used. It features multi-touch gestures, stylus input, handwriting, dual screens, and a camera.

This would be the true consumer version of Microsoft Surface. Can't wait for more information. At this point I wouldn't be surprised if there was an announcement at PDC in November.

The MOADtouch Project

Open your mind. Relax your brain muscles. This will take a little creativity.

Rewind to the 1968. Douglas Engelbart is about to put on The Mother of All Demos (MOAD). This time, though, instead of inventing and demoing the first computer mouse, he invents a touch (or even multi-touch) screen and shows off an interface that has more in common with Surface or iPhone than Windows or OS X.

Pretend that mice were never invented. What would today's computer interfaces look like with 41 years of research and evolution and industry focus on multi-touch interfaces?

Recently, there have been several articles that show touch in a negative light, with conclusions like these: "Touch is Dead On Arrival" and "[multi-touch] adds little of value."

Those types of comments come about because when some people think of touch on a PC, they visualize a GUI (mouse + keyboard) application and then add some touch to it, and see that touch is redundant or unnecessary. Well of course it is! I agree! Who would want to use an application or OS designed for a mouse pointer with +- 5 pixel precision with their fat fingers with +- 30 pixel precision? In our imagined alternate MOAD touch demo world, though, every application would be designed to work well with gestures, multi-touch, and fluid, high-frequency touch interaction.

I'm proposing the MOADtouch project: Let's help people imagine what a NUI world would look like. Take your favorite GUI application, throw out the GUI, and totally reinvent it as a NUI. Pretend touch was only acceptable input device to the mass market. Forget the Windows, Icons, Menus, Pointers. Forget the rectangles. Start with your finger tips and design out from there.

I want everyone to participate, regardless of individual skills. Write blogs, create wireframes, mock up screenshots, mock up videos or even code up prototype interactions. Describe or show the experience. Do not throw out any possible application just because someone else doesn't think it would work with touch. Recreate the interface so that it does work.

I want this to be viral. Send this to all your friends and colleagues. Get them all to create and post something simple, even a snippet of an idea. "It'd be cool if XYZ was like this: ..." Twitter it. Blog it. Flickr it.

Tag everything: MOADtouch.

Remember: Open your mind. Relax your brain muscles. This will take a little creativity.

Touch feedback holodisplay

One of the main drawbacks of multi-touch displays compared to regular buttons and keyboards is the lack of tangible feedback. There have been various proposals on how to make flat displays with bumps that raise or other methods. Touch screen cell phones sometimes vibrate or simulate a click when the user activates an interface element.

Here is a video of a concept being presented at SIGGRAPH 2009 of how to achieve touch feedback in mid-air for a holographic display:



More information on the research team's website.

Stop counting clicks, start feeling interactions

Wanted to point out an interesting and relevant portion of this article by David Hamill about why the 3-click rule is bunk.

Stop Counting Clicks

It’s time for a cycling analogy

I’m a keen cyclist and there are quite a few hills near where I live. On a gradual hill, I shift into a fairly easy gear that still allows me to maintain a bit of speed. In order to get up the hill I might turn my pedals 300 times. Each pedal turn takes a bit of effort, but the hill isn’t very steep so I have no real problems getting up it.

The next hill I come to might be a lot steeper. In order to get up it I’m going to need to shift into my easiest gear. Each pedal turn is very easy, but I need to turn the pedals a lot more often in order to get up the hill. The steep hill is a more difficult task than the gradual one. By breaking the task up into lots of easy pedal turns I can still get to the top without problem.

So what has any of that got to do with websites? Well, in order to keep some tasks simple for your users, they may have to click more often. In the same way that a cyclist needs to perform more pedal strokes when climbing a steep hill.

Just as the cyclist is not counting her pedal strokes, your user is not counting his clicks. They both just want to get to where they’re going.

This kind of thinking applies perfectly to designing Natural User Interfaces. The gestures and manipulations we're designing are so much richer than point and click, though, so it's not just a matter of breaking down tasks into a certain number of clicks and screens.

With NUIs we can also decide to use a manipulation on content, apply rich animations for transitions, and blur the line between one user action and the next. The focus of NUI user experience design is much more about the quality of the interactions and playfulness than in GUI.

In a complex application, it may be impossible to count the number of taps or interactions if they are continuous. That's all the better though, since instead of quantifying the number of interactions, we should be qualifying the feeling that the interactions invoke. Is it fun or laborious? Is it engaging or boring? Is it social or anti-social?

How does your interface feel?

Channel9 interview about InfoStrat.VE

Josh Wall and I were interviewed on Channel 9 about InfoStrat.VE, the free control for Virtual Earth on WPF and Surface that we released.

The video is fairly long at 46 minutes, but it has a lot of interesting background about how the control works, how it came to be, how to use it and how to get started.

N-Trig DuoSense Development System

Not many details out yet, but an article on Business Wire claims that N-Trig has a DuoSense Development System that developers can use to target current and future multitouch system. (Full press release below.)

It sounds like the development system operates like a 12.1 inch monitor you can plug in like a second display, except it supports all the DuoSense features. I sent N-Trig a message asking about the availability and price of the development system.

This is pretty critical to getting good applications developed for systems that don't even exist yet. In addition, many of the multitouch systems out there are not that powerful, CPU-wise. The Dell Latitude XT2 maxes out with a 1.4 ghz processor, not something I'd want to be stuck on for a development system.

The N-Trig DuoSense supports true multitouch (more than just two-touch) plus stylus as well, so it is pretty popular. It is getting to be a standard choice for multitouch digitizers. It is used in the Dell Latitude XT and XT2 plus the HP TouchSmart tx2. The developer community around each of these tablets is quick to provide feedback to N-Trig and waits for each new release of the Windows 7 beta drivers.

Here is the full press release:

N-trig, providers of DuoSenseTM technology, combining pen and capacitive touch in a single device, are further advancing the multi-touch adoption process with the new DuoSense Development System. This device enables software developers to create touch-enabled applications on their own computer.

The DuoSense Development System is an extension touch-enabled display that provides a powerful and simple development environment for programmers, shortening the development life-cycle and reducing costs. The system connects to any PC via a USB and enables ISVs to develop multi-touch applications while simultaneously viewing the results of their work on their computer screen.

“Our development system is designed to give ISVs an easy way to build new and creative programs quickly and cost-efficiently from their existing development environments ready for the launch of Windows 7 and to fuel rapid consumer uptake,” said Lenny Engelhardt, VP Business Development of N-trig. “This new system underlines our strong commitment to providing software developers with the best tools to speed along the adoption of multi-touch capabilities in the market, particularly with multi-touch being such a strong component of Windows 7.”

As more OEMs are looking to incorporate multi-touch into their R&D roadmaps and the availability of Windows 7 becomes a reality, the evolution must start with the software developers. N-trig has designed this unique system to bridge the gaps that currently exist between ISVs and multi-touch enabled programs.

Going forward, no longer does multi-touch development require a pen and multi-touch enabled computer. The DuoSense development system boasts a 12.1 inch screen and can be connected to any computer. After loading the latest software versions, which can be found at http://www.n-trig.com/Content.aspx?Page=Multi_Touch, developers are all set to go multi-touch.

Surface tweet-storm PR nightmare

Richard Wand posted a summary of the PR nightmare for the Microsoft Surface team last week.

This whole thing really was unfortunate because I know the Surface team has gone to great lengths to make sure the user experience with Surface is top-notch. The experience this one company had was basically the result of a few people not thinking things through and then posting on their blog a comical version of their experience. Unfortunately, the comedy part of it is lost when fanboys of various flavors looking to repeat and magnify any criticism of Microsoft.

Fortunately, the Surface Team (and other people with brains) responded to the criticism immediately and made things right. Well, there wasn't much they could do since the problem was mostly those people not knowing what they were doing initially, but they at least made contact with them to discuss their feedback.

This whole event highlights the importance of user experience (particularly when you are positioning yourself as a premium UX) and also about the responsiveness of marketing teams. Bad news can spread in a matter of hours beyond your control, so if you do not participate in the online conversation, you won't know what people are saying about you and won't have a chance to set things right. (Of course, you have no hope of controlling what people say, but at least you can inject your version of things.) The easiest way to prevent this, though, is to have a top-notch UX. I think Surface does have a great experience, not only in software but the device installation teams and developer training. This one company just tripped over itself and fell through the crack.

I wonder that little marketing firm would have written about trying to setup Epson X-Desk?

Epson X-Desk: not impressed

Here is an X-Desk video. I'm completely underwhelmed. Comparing it to Microsoft Surface, the only possible advantage is that it is larger, but that may not be an advantage depending upon the specific application. In every other aspect the X-Desk (and/or its demo application) is sub-par:

• Watch the response time as they draw the lines at the beginning. It seems quite slow, slower than Surface.


• I guess the finger traces is meant to be the equivalent of Surface's attract application. It wouldn't attract anyone unless someone else was already using the table.


• "Draw a circle to call up the menu." This is not discoverable, and doesn't even make sense since the menu is already on screen. Maybe there is a way to hide the menu but it isn't obvious.


• The menu only has room for four buttons. Do they plan to allow more than four apps in the application launcher?


• Once inside an application, there are two ugly bars in the center of the long sides with a "Back" button. That takes up more space than necessary and uses prime real estate that would be better used by the actual application.


• The applications demonstrated are bad rip-offs of Microsoft's early Surface demos from two years ago:
  
  • Photo viewer - just a scatterview, no pods or groups or video even
  
  
  • Phone comparison - Even copies the two phone comparison circle from the AT&T store application. When the lady moved the single phone around, the circle with text moved around in weird ways to stay out of the way. At first I thought this was a little bit neat but now that I think about it more, it's a bad idea since it doesn't consider how multiple users might be using the device at once. "Hey, I was reading that!"
  
  
  • Drinks - Very basic app, displays some frankly unimpressive eye candy and the name of your drink when you put a tagged cup on the table. Would the form factor of this table even work well in a bar or night club?
  
  


• The IR tags have 16k possible values. This is between Microsoft's byte tag and high-res identity tag, and doesn't provide enough for many enterprise applications.


• The whole visual experience is very basic. It is obvious that they did not have a designer work on this or have any kind of UX expert.

Perhaps what we saw was just a single demo and the menu isn't some kind of common application launcher. That's even worse, since it means Espon doesn't have any effort towards providing a uniform experience.

The Surface team should not be afraid of X-Desk at all. It makes them look like an even better value with the better user experience.

NUI Principals by Dennis Wixon

I saw this video of Dennis Wixon talking about what NUI is, the core principals, and what might come after NUI and I had to repost it here. Video via Jonathan Brill.


Dennis Wixon | UX Week 2008 | Adaptive Path from Adaptive Path on Vimeo.

I've seen this basic material presented several times by different people, notably Micrsoft's August De Los Reyes, who has a very exciting presentation style. Dennis in this video also did a very good job presenting the material in an engaging way as well as tying the material into his personal credentials and back story.

When he started, he began doing the recent Microsoft presentation fad of introducing presentations by stating several off-topic useless facts about himself. I thought "oh no, here we go again" but was pleasantly surprised how he actually integrated the personal facts in with the main presentation content.

Dennis made an interesting statement that stuck with me (besides the eight principals which I've heard before) starting around 7:15 in the video:

"From that historical analysis we can extract a set principals and guidelines. The principals are what drives the design. The guidelines are simply derivations of the principals for an individual context. Principals are whats important. Principals and data drive successful design."

This is an interesting way to introduce the principals. He leaves it unsaid, but basically the principals are timeless patterns independent of any technology. Guidelines are how you apply them to, e.g. Surface, iPhone, etc.

Deconstructing: Considerations for Designing Touch UI

Here is a cool video by Punchcut summarizing some considerations when designing touch interfaces. Video via Richard Monson-Haefel.


Punchcut Perspective: Considerations for Designing Touch UI from Punchcut on Vimeo.

The content itself makes sense, but I was more interested in the unique way the content was presented. The same content could have fit easily into a handful of PowerPoint bullets, but instead it is presented as if using a multitouch interface to control the content. You can see the shadows of hands in some parts and the auras of the fingers leave traces to make it clear what gestures are being used to manipulate the content.

It is a little bit like the HP commercials where the famous person moves their hands around mid-air and photos or skateboard designs or documents move and animate as well. Of course the HP commercials are not meant to suggest that you could actually manipulate content like this with their current products. Well, maybe with the sixth sense device some of those gestures and visuals could be reality, but that's not what HP was thinking about.



Unlike the HP commercials, the Punchcut video and the finger auras are actually close to a feasible design. It made me think what would have to change in order for that to be a real application. Let's deconstruct this NUI!

Each of the sections in the video illustrated a different way to transition or manipulate content. This could be thought of similar to how PowerPoint allows different slide transitions or animations, but the manipulation mode is much more fundamental here and integrated with the content. Let's enumerate the sections, my name for them, and the gestures used within.

1. (0:00) "Hello." - Flip card. Swipe to display text, then swipe to flip text around. Two-touch zoom out gesture to grow text.


2. (0:05) "Five considerations" - Pivoting text. First finger establishes pivot point. Second finger swipes to control the pivoting transition from one content item to another. Finally swipe through text ("touch interfaces") to erase it.


3. (0:12) "1" - Direct drawing. The finger draws any shape, in this case numbers, and the aura trail lingers about five times longer than in the other modes.


4. (0:13) "Design for" - Pick a word. Swipe to fade in foreground text and background words. Tap each background word in order to add it to the foreground text. Kind of like typing except with entire words instead of letters. Note that the words are preformatted and punctuated. "immediate access" shows up in the title font size, while the rest of the words show up in the subtitle font size. Swipe completed text to push it off-screen.


5. (0:24) "2" - Direct drawing.


6. (0:26) "Tap is the new click" - Scatter text Swipe to drag text in from off-screen, then more swipes to push it around. Double tap to fade in additional text. Finally swipe the text off-screen.


7. (0:32) "3" - Direct drawing.


8. (0:35) "Clear mental models" - Rolodex Swipe up to rotate text in to view in 3-D. Previous and next text phrases are shown out of focus in the appropriate place in 3-D. Swipe up again to rotate the last phrase away.


9. (0:41) "4" - Direct drawing.


10. (0:43) "Design for hand sizes" - Zoom text & Keyboard Zoom in gesture to grow initial text from zero size. Touch and drag part of the phrase open to make room. Minimal keyboard appears, type to insert text. Press the larger button (enter?) to show additional lines of text. Note the keyboard doesn't display any labels to us, just outlines of buttons. Presumably the presenter either knows what the buttons do, or can see button labels on a different screen, or the specific buttons don't matter and will insert the correct text regardless. Finally, drag the text and keyboard off-screen with separate fingers.


11. (0:53) "5" - Direct drawing.


12. (0:56) "Touch feedback is key" - Tap feedback Drag text in from off-screen. Tap each line of text for a different highlight effect. Drag text off-screen.


13. (1:03) "Touch is merely one dimension" - Directional transition by line Touch two fingers to establish an unseen line. A third finger swipes toward the line and text is animated on or off in a direction perpendicular to the line. (Sounds complicated but it makes sense when you see it.) One final swipe fades in some text.

Now that I have actually written all this down, there isn't anything on this list that would not be a reasonable way to manipulate a presentation. There are some considerations necessary to make an application like this usable, though:

• If replicating this presentation exactly, all the text and transitions would have to be prebuilt in an authoring mode. It would basically be glorified PowerPoint, and after the shininess factor wears off, it would not be that compelling. Some of the ideas below show how this application could be used in a much more dynamic and collaborative way.


• In addition, there are sometimes no visual cues to the presenter which gesture he or she needs to use to continue the presentation. Worse, the same gesture (swipe left) could do totally different things in different modes. In a presentation of any length with a variety of these transitions it would be very easy to forget the order and embarrass yourself by making gestures aimlessly trying to make content appear.


• Instead of having an authoring and presentation mode, merge the two and allow the presenter to choose the content and which manipulations to use during the presentation. This would require some type of manipulation menu and a content library. The content could still be authored in advance, but it could also be sucked up from external sources, for example a network drive or Flickr account.


• Allow the presenter to organize the content library however they want, but provide some tools to help. They might do it completely free form with piles of content, or might use a tool to draw lines between content or groups of content representing different relationships. The relationship could be specific to a presentation order or relate to the content itself. Let the user decide what works best for them. This by itself would be an enormous improvement over the enforced PowerPoint outline.


• Provide a resizing presentation area that overlays the rest of the interface. When the user is ready to present, he or she can drag content in and out of the presentation area as needed. Only the presentation area is shown on an external monitor or projector. If the presentation is really just a few people around one computer, the presentation area isn't necessary and the meeting should become more fluid and collaborative.


• Keep all the non-presented controls like the manipulation menu and unused content out of the presentation area. The presenter will be able to select from the manipulation menu or navigate the content library relationships off-screen, then bring items into the presentation area when ready. In this way the presenter can react to audience feedback and questions to bring up appropriate content, or a small group working together could just explore content and discuss it in a much more social way than linear PowerPoint presentations allow.

I better stop here, otherwise I'll give away too many ideas for free! There is a lot of potential with NUI and touch interfaces for rethinking how we did things in the GUI world. The Punchcut video demonstrates this kind of NUI thinking perfectly.

Breyn: multi-user multi-touch brainstorming

Here is what appears to be a design course prototype for a multi-user brainstorming tool.


breyn - multiuser-multitouch-brainstorming from Joschka Wolf on Vimeo.

Key features to watch for:

• Triangular toolkit object - drag or touch one of three corners to set the nexus for an action such as typing text


• Multiple toolkit object, one for each user (maybe color-coded) - allows different users to select different tools and perform different tasks at the same time


• Virtual keyboards - appears to be bound to specific triangle objects


• One tool creates a box, then drag the text tool into the box to enter text


• Draw lines from one box to another to connect them hierarchically


• Box orientation indicates ownership


• Extra long circular text (group ownership?) is spiraled


• There's some mechanism (unseen) to rotate all the boxes to the next side

This is a great example of a collaborative approach to multi-touch. Many users can do individual tasks towards a common goal.

via http://www.conpark.de/trends/multiuser-multitouch-brainstorming/

Brogge releases ByteTag printing tool

Brogge over at Bo's blog has created and released a Microsoft Surface ByteTag printing tool. Looks like it would be pretty useful for anyone deploying lots of ByteTags. (Unfortunately, there is no spec for IdentityTags yet so it doesn't include those, only the 8-bit ByteTag.

Features:

• Select the tags that you want to print
• Indicate how many times a certain tag needs to be included
• Limited layout option for printing your tags (page size, page margins, tag margins)
• Printing the tags. I have tested them on our surface unit and they seem to be recognized correctly.

http://www.metanous.be/boblog/post/ByteTag-printing-tool-for-Microsoft-Surface.aspx

Deconstructing: Razorfish DaVinci Surface video

Razorfish released a new Microsoft Surface video called DaVinci, which is basically a physics illustrator with some cool Surface enhancements.


DaVinci (Microsoft® Surface™ Physics Illustrator) from Razorfish - Emerging Experiences on Vimeo.

The basic physics functionality is similar to the several physics illustrators already out there for PC, but this integrates the vision system of Surface for some pretty cool effects. There looks like some extras like the gravity control which makes it work well in the multi-orientation UX. It also uses some physical objects as dials, which allows great affordance. (Pretty similar to the ReacTable.)

I realize it's more of a concept demo than full application, so isn't expected to be fully featured. Given that, here are my thoughts regarding the user experience:

Good UX:

• Very good integration with vision system, shows off unique features of Surface


• Good use of the ink system. The finger, paintbrush, and toy car all create different size strokes


• Multi-orientation through controlling gravity and gravity vortexes (maybe they call it something different)

Bad UX:

• The control panel is pretty "WIMP"-y. There has to be a better NUI way to integrate those controls, like the physical (gravity?) dial


• Dragging components around seems laggy, although this may have been on purpose due to the physics simulation. This is particularly obvious during the pong simulation, and just makes Surface itself look laggy. If it is meant to be springy, then a visualization of the temporary spring connecting the component and the fingers would help a lot.


• The three finger gesture to connect a component to the background is not discoverable and doesn't have any feedback! At least a color change would help. On the plus side, three finger gestures are not likely to done accidentally. Perhaps this one is filed under advanced gestures, but how will the advanced user learn it unless they are shown by else in the know? This needs progressive disclosure.

Don't get me wrong, this is a very cool application. The core functionality looks awesome, but I know there is more potential for refining the control mechanisms and fully implementing the Surface UX guidelines.

Announcing: InfoStrat.VE - Virtual Earth 3D for WPF and Microsoft Surface

Huge news!

Today I got final approval to publish our WPF Virtual Earth 3D control on CodePlex. It is open source under the LGPL, and it is now live!


http://virtualearthwpf.codeplex.com/

This control comes in two versions: one for regular WPF applications and one with Microsoft Surface enhancements (multitouch pan and zoom gestures).

I am the primary developer for this project at InfoStrat - take a look at some of the projects we have used this in.

Previous implementation of Virtual Earth on Surface were done using transparent windows and the map itself had limitations (no rotation or transforms, etc.) due to Win32 interop.

InfoStrat.VE uses some magic to eliminate these restrictions. It also takes advantage of WPF data binding for controlling various properties such as the camera location. You can add pushpins by simply adding a WPF-based VEPushPin as a child element. VEPushPins can also be provided by a data-bound collection.

It's as easy as this:

Go check it out!

Update 5:07 pm: CodePlex itself is having some timeout issues, so if the link doesn't work try again later.

Microsoft Surface comic

I tried my hand at some humor using Stripgenerator.com. Here is the result:


(Click for full comic.)

Funny? Lame? Let me know.

World Builder (short video)

This video is not meant to be a UI design prototype (like 2019) but it has some neat UI eye candy anyway. A man creates a 3-D world using augmented reality interface, which we find out later is probably transmitted into his brain similar to the Matrix.

Watch for the different types of spacial gestures he uses to create his world.

For your enjoyment:


World Builder from Bruce Branit on Vimeo.

What do you want in a NUI prototyping tool?

Richard Wand finds existing prototyping tools lacking when creating prototypes and mockups of NUI interfaces. We had a short conversation about what features would be nice and then he poses the question on his blog:

In my quest to find the most efficient tool for defining the behaviour of an application I posted a tweet for the #Surface community to come forth with their digital sketch tool of choice. I received responses from the usual suspects - @joshblake @stevedawson @lhamilton – who are active Microsoft Surface twitterers (and well worth following). My question was timely as they were individually considering developing a UI prototyping tool for NUI’s.

Joshua (@joshblake) posed the question back to me - what features would I want from such a UI prototyping tool? UI prototyping tools such as Denim and Balsamiq were inspired by the application authors looking for a better solution for digital sketching. Before I test these out I thought I’d open Joshua’s question up to a wider audience...

What would you want from a digital sketch tool for the early stages of UI design?

Head on over there and join the conversation.

Surface SDK Version.Next Video

Came across this video in my twitter feeds. This video shows some unreleased features of the next version of the Surface SDK.

EDIT: Comments report this video is from a prototype implementation at Disney.

Notice:

• Applications running across multiple tables
• Automatic application switching when a tag is recognized
• Loading bar when switching application
• The application launcher was never shown

I bet the tags are identity tags, which were always planned for the next version. It wouldn't make much sense to allocate the few 8-bit tags to switching applications.



Via lhamilton

Joe Fletcher on Touch and Gesture Computing

Joe Fletcher from the Microsoft Surface team talks to Robert Hess about Touch and Gesture Computing.

Channel 9 - The Knowledge Chamber

He has some insightful comments about designing gestures, the advantages and drawbacks of different touch technologies, and Natural User Interface development for both Surface and Windows 7. Overall Joe has a very pragmatic approach and attitude towards touch and gesture computing.

One interesting comments (to me at least) was when Joe talked about how he works on Surface projects but also Windows 7 touch projects (5:45 mark.) He also suggests that pretty soon there will be a huge variety of touch and multi-touch hardware. This relates to my posts last weeks about the Future of Microsoft Surface (Part 1, Part 2, Part 3).

One very insightful comment (6:30 mark) is about how touch features with Windows 7 needs consider that not everyone will have touch hardware. Touch applications need to add to the experience, not be required for the experience. Robert points out that this is similar to the early days of the mouse when Windows provided keyboard shortcuts for everything the mouse could do as not everyone had a mouse.

He also suggests later (10:00 mark) on that the current multitouch computers (Dell XT, HP Touchsmart) have some quirks that still need to be worked out, and the hardware platform is a moving target. The advantage of Surface (and why we should be investing in Surface now!) is that it is a stable platform with many features that these other platforms will not have for a while. Surface's vision system and tag recognition is used in production application today, which Windows 7 hardware may not have that for a while.

Joe also discusses (12:45 mark) the novelty factor of touch computing. If you jump in now you can learn as you go and create relatively simple apps and people will be satisfied. Later on when people expect more advanced applications you won't be able to wow them with basic demos anymore. Pong used to be the state of the art and everyone was amazed, but today even young children expect more out of their computer games.

Microsoft Office Labs Vision of 2019

I just came across this concept video created by Microsoft Office Labs. There are so many interesting concepts in here. I'm still going through them but take a look.

Trailer:

<a href="http://video.msn.com/?mkt=en-GB&playlist=videoByUuids:uuids:a517b260-bb6b-48b9-87ac-8e2743a28ec5&showPlaylist=true&from=shared" target="_new" title="Future Vision Montage">Video: Future Vision Montage</a>

Full video:



Via: istartedsomething via twitter SteveDawson

InfoStrat interviewed by on10.net

Josh Wall was interviewed at the Army Symposium a few weeks ago about our Microsoft Surface applications. That interview is now live. (Josh Wall and I created each of the applications highlighted.)


Sneak Peek at Surface Apps

The Future of Microsoft Surface, Part 3

In part 1 I discussed how Microsoft Surface's business model seems extremely unprofitable when compared to Windows and Xbox business models.
In part 2 I compared Microsoft Surface to a SmartPhone prototype.

Now let's wrap this all up.

Consider the plans for the Surface SDK -- all the good multi-touch stuff is being baked into .NET 4 and will be available on any computer, not just special Microsoft Surfaces. Only the very unique vision system remains outside as an extra component. Windows 7 will have support for everything that Surface does, depending upon the hardware it runs on.

Surface is perfectly positioned for a transition to the highly profitable Windows business model upon the release of Windows 7. In fact, the Surface SDK and Windows 7 will be the same thing, so there won't be so much of a transition as a merger. Microsoft Surface will no longer be just a single product. Surface will become known as a brand of related products, just like Windows Mobile and the SmartPhone. The current Surface provides a model user experience which third-party manufacturers will replicate in their own Windows 7 Surface designs.

Microsoft created an entire business unit to gather data about and prepare the public for the Natural User Interface Revolution. The current Surface device is just the first step. The real potential of Surface will unleashed when hardware manufactures create their own version that run Windows 7.

So now after considering all of this, the big question is whether you should bother with the current Microsoft Surface. The answer is absolutely! Getting involved with Surface now gives you a huge head start in the upcoming NUI revolution. Usually prototype concepts (like a prototype SmartPhone) have limited access to hardware partners until it is released for mass distribution. In this case, the Surface is much more available and there is a rare opportunity to get in on the ground floor.

Very soon, Microsoft Surface is not just going to be a "big-ass coffee table" as some believe. It will be a whole range of products in many different forms, and those who prepare themselves will be able to take advantage. Are you ready?

(Thanks to Jonathan Brill for inspirational conversation that led to these ideas. We shared the basic thesis in this series and his post on Why the Failure of Surface is a Success for Microsoft. Subscribe to his blog if you haven't already, since he claims it is slightly more interesting than mine! :-O)

The Future of Microsoft Surface, Part 2

In part 1 I discussed how Microsoft Surface seems to be an extremely unprofitable venture. It combines the least profitable aspects of the Windows and Xbox business models. But there are good reasons for this.

My explanation is don't think of Microsoft Surface like Windows or Xbox. Think of it as a research and development investment on steroids. Think of it like a prototype SmartPhone.

The Windows Mobile business model is almost exactly the same as the regular Windows business model, with two exceptions:

1) Windows Mobile runs on specific hardware that is manufactured by third-party hardware manufacturers
2) Microsoft (just like NVidia, Intel, and others) creates reference hardware prototypes to show hardware manufacturers the basic features. Manufacturers take the reference design, create their own, embellish and elaborate, then sell sell sell, all with Windows Mobile on it.

There is also a chicken-and-egg problem with getting hardware manufacturers to buy into new products like SmartPhones running Windows Mobile. Manufacturers must invest a great deal into creating and marketing new hardware, and they won't bother unless there is a reasonable guarantee that they will be able to sell it in large quantities. Microsoft will provide market studies and the reference prototype to manufacturers for this reason.

So should anyone bother investing in a prototype? The answer in this case is absolutely, and the reasons why are in part 3.

The Future of Microsoft Surface, Part 1

The Microsoft Surface business model is a unique. The Surface business unit doesn't operate like any other within Microsoft and it is unclear how profitable Surface will become. Lets compare Surface to other business units in Microsoft. Consider how the Windows business model (where makes most of it's money) works:

1) Create new version Windows
2) Release free or cheap development tools to create a huge eco-system of developers writing applications for Windows
3) Secure agreements with hardware manufacturers to put Windows on almost all PCs
4) ???
5) Profit!

On the surface (ha!) it would seem that Microsoft Surface is more like the Xbox business model because it involves selling hardware. The Xbox business model looks like this:

1) Create and sell Xbox at a loss
2) Extract massive licensing fees from Xbox game developers at a profit
3) Pay for a huge maintenance and support department for broken Xboxes
4) ???
5) Hope for profit?

Compare the two models above to Microsoft Surface:

1) Create and sell Microsoft Surface (at a loss?)
2) Release free or cheap development tools to create an eco-system of developers writing applications for Surface
3) Pay for a huge maintenance and support department for broken Surfaces
4) ???
5) Lose money? (for now...)

So it looks like the Surface business model combines the least profitable aspects of the Windows and Xbox business models. Why in the world would Microsoft operate like this? How long is this even sustainable?

There are genius reason behind all this, though. Continue reading why in Part 2.

Edit 3/10/09: modified phrasing and word choice

GUI + Touch != NUI

Since the Windows 7 pre-beta and beta has been available, I've been reading some articles and seeing videos where people review the touch/multi-touch aspects of the new OS. Opinions have been mixed:

Engadget: Windows 7 Multitouch: It's a gimmick (for now)

OSNews: Windows 7 Beta's Touch Capabilities Underwhelm

Lifehacker: Windows 7 Multitouch: So Far, So Good

Gizmodo: Windows 7 Touch Control Makes Media Center More Awesome

GottaBeMobile: Will the multi-touch ecosystem be ready for Windows 7 launch?

A common theme can be seen across all of these articles -- applications designed for touch (such as Media Center) work well, but regular Windows and applications do not benefit from touch or multitouch. At best, the multitouch features in the Windows 7 shell are forced.

Media Center (originally designed for use on living room tvs) is probably the best built-in experience. There are only a few MT applications out there -- the most common is the air hockey game. Depending upon the hardware and beta drivers, that game works mediocre to okay, and some of the above articles discuss.

Note this video shows pre-beta drivers on a pre-beta OS build, but watch how the user accidentally closes the app while trying to maximize it.

What this boils down to is the fact that you can't just throw a touch or multitouch screen on top of a GUI application (written with WIMP in mind) and expect it to be a new or compelling experience. NUI is not just a GUI with touch. You really have to design a Natural User Interface application with touch (among other things) in mind for the best experience.

To Microsoft's credit, they have made modifications to e.g. the task bar to make it more touch friendly, and have updated the UX guidelines to account for touch. They defined three categories for touch support:

• Good - Free touch support provided by Windows (i.e. regular GUI apps)


• Better - Gesture support, touch-friendly UX (i.e. GUI + basic touch aware apps)


• Best - Touch-optimized experience (i.e. GUI + touch integrated apps and NUI apps)

(My comments in parenthesis.)

Windows 7 shell (GUI), a "Better" experience but not very fulfilling. (Note at 1:39 it switches to a Mac with multitouch touchpad, which is even worse.)

Media Center, a "Best" experience which looks and feels natural

I think that touch devices are going to have a hard time gaining market share until the are plenty of "Best" applications. This seems very similar to the rollout of the Tablet PC. Stylus input was touted as the next big thing, but there were not many stylus aware applications. Most applications were just GUI + Stylus.

If we want to build demand for NUI applications and touch and multitouch devices, we are going to have to think beyond GUI + Touch and make developers, decision makers, and the general public aware of the Natural User Interface Revolution.

What do you think we should do to make NUI applications a success in the market?