Interview: Tamir Berliner of PrimeSense

The natural user interface space has been very busy recently. Last month after Microsoft launched Kinect for Xbox, I organized people interested in using Kinect on PC and founded the OpenKinect community. Since then the community has grown significantly and we’ve seen some amazing tech demos and applications using the libfreenect, the open source Kinect for PC drivers developed by OpenKinect community members.

A few weeks ago, I was contacted by PrimeSense co-founder Tamir Berliner, who wanted to discuss their plans and ways PrimeSense could engage with the community. (Microsoft licensed PrimeSense’s technology and based Kinect on the PrimeSense reference design.) After seeing the dozens of OpenKinect videos and the community enthusiasm, PrimeSense decided to move up their plans and yesterday announced the OpenNI initiative and have open sourced their middleware for natural interaction and drivers for their depth cameras.

Tamir graciously agreed to let me interview him for my blog. 

Joshua Blake: Thanks for agreeing to answer my questions. First, could you please give us a brief history of PrimeSense and its role in the natural interaction space?

Tamir Berliner: Thanks for having me :)

PrimeSense started from the founders’ understanding that we are bored with standard gameplay. We felt that computer games didn’t innovate any more but rather mainly used a different story to differentiate. We felt it was time for a change, and understanding where natural interaction is headed to we decided to take the first step towards that vision in the games domain, with a 3D camera that was built for that purpose and at the right price point

Josh: What is your role within PrimeSense?

Tamir: It’s been changing all along in the past 5 years, the main idea is to do whatever it takes to make sure the vision is being fulfilled and that we keep on striving towards Natural Interaction. In the past 5 years I’ve been the head of an R&D team, the first software product manager, I created our partners program and now I’m running this amazing effort of contributing our best and latest to the developers community so you guys can do what we didn’t even start to imagine

Josh: I know that any technology takes a long time to mature. For example, Microsoft Research’s Bill Buxton has doing multi-touch research since before the Apple Macintosh came out in 1984, and there were others before him. Multi-touch has only become mainstream in the last few years. How long has research been going on in the motion tracking and pose estimation space?

Tamir: Over 20 years if you count the first time image processing was done in order to extract a human skeleton from an image, or the usage of two cameras to reconstruct a 3D image.

Josh: How long has PrimeSense been involved in this field? What have you guys been up to that has prepared for today when are seeing mass adoption of motion tracking and natural user interfaces in the home?

Tamir: We started ‘running’ in Sept 2005. It took us 5 years to get the first product that uses our technology to hit the shelves. From day one, our goal was computer games. We actually had first trials of skeleton running in early 2006 which helped us both sell the technology and explain what’s possible, and set up all the different parameters so that they will fit best to the usage in the living room domain and even took some tradeoffs to tailor make it to computer gaming.

Josh: I have heard several people make the comment that what can be done with depth sensors has been done before with single RGB cameras and computer vision techniques. Many of those approaches require controlled environments and are not robust enough to use in any random living room, for example. How do depth sensors and PrimeSense technology change the game as far as the interfaces and interactions that are possible now? How does hardware cost play in to the equation?

Tamir: The main issue that PrimeSense has solved is a robust, real time, low-cost depth-map generating device that can be configured to create a skeleton on a low end PC. Our middleware takes less than 1 ATOM CPU and provides a great skeleton model for two people in real time. It also allows you to control a fully fledged media center from 15ft away with mere hand gestures that we keep on making sure passes user testing so everyone can use those at home

Josh: We touched on pose estimation before. When thinking about human-computer interaction, what can we do with a skeleton model of humans in a real space that we cannot do with just raw depth data or point clouds?

Tamir: The raw data is great for testing and rapid prototyping of basic interaction. The main issue is that it usually requires the user to be too aware of the algorithms – take for example the virtual touch screens that we’ve seen - you have to know exactly where to stand and how far your hand needs to reach. These cannot work in the living room or in standard home environments as you will just walk through the virtual screen, interacting with the TV without wanting to. Also imagine that you need to get up every time you want to flip channels or change the volume?. Another example are all the different ‘trainers’. Whether it’s titles like Dance Central or Your Shape, those cannot be made with just raw data, you have to know where the user is and what exactly he/she is doing

Josh: You have described PrimeSense technology as enabling Natural Interaction. What does the phrase mean and where is it heading?

Tamir: The goal of Natural Interaction from my point of view would be to be able to take any person (even those who have never used a technological system) and have them be able to use it without any training. I think that Kinect and iPhone are both a great step towards that direction, but we’ve got a lot to do both on the hardware side and on the software side. Merging the interfaces between all the different devices that you have and making them “know you” is a key step and require tons of work. I believe this revolution will take at least 10 to 15 years till we can look back and say ‘what were we thinking with those mice and keyboards?!’

Josh: Is the future of natural interaction and 3D cameras limited to games and entertainment or will it see adoption in other fields as well?

Tamir: Natural Interaction is already coming from all different directions at us, you can see it with Wii, iPhone, and even in public restrooms where the water tap understands you want to wash your hands and starts the water flow automatically. The more we see Natural Interaction being introduced into the market, the more you will see everyone using technology. The keyboard and mouse seem ‘normal’ today, but if you come to think of it, they are as natural to a human as leash is to a dog.

3D cameras are of course one of the key enablers of Natural Interaction, just as speech and touch are, and as such I foresee this technology being incorporated in many more technological devices in different forms and as different input paradigms. We can see great examples from the community with regards to what’s possible and I believe this is merely the beginning. Another very important area where 3D cameras are coming into play is providing technology with “eyes”. Just like providing phones with accelerometers and GPS was a big step, so will this be and some of the main targets will be technologies that need to move and navigate in the real world

Josh: We’re already seeing a lot of early videos of these types of applications come out of the OpenKinect community. What was the reaction inside PrimeSense like to the really creative and diverse ways that people are using the Kinect depth sensor? I heard it caught you by surprise.

Tamir: The reaction was ‘Wow’ after 5 years of meeting with people and trying to explain what is possible, you finally see people who not only do things you’ve thought of, but do much more and create amazing applications that we didn’t even consider! It’s mind blowing to see that creativity is far greater when it’s in the hands of a community, and it’s one of the main reasons for why PrimeSense has decided to provide the community with the work that it has invested a lot into. I don’t know if the people in the community will like what we’ve got to offer, but what led us to make the decision was that we’ve seen people starting to tackle issues that we’ve been working on for years now so it would almost be a ‘sin’ not to contribute from our experience.

Josh: Clearly there is a quickly growing community that is extremely enthusiastic about the possibilities unlocked by this type of technology. (The OpenKinect mailing list has grown to over 1250 members join in just four weeks.) Even so, Natural Interaction is still new and we are still at the beginning of the Natural User Interface revolution. What are the biggest challenges that we will see to adoption of Natural Interaction across the industry?

Tamir: I believe that the value of Natural Interaction is great, but only when it’s implemented properly. Only a small set of companies have been able to really create devices that both are worth the money and have provided a real Natural Interaction leap. As I mentioned I see companies like Microsoft with the Kinect, Apple with the iPhone and Nintendo with the Wii as being the first and I can only hope (as a consumer) to see everyone in the world understanding the this is merely the beginning. The number one issue for all of the devices is to simulate the interfaces that we already know from the ‘real world’. Wii did it with motion, iPhone did it with touch, and Kinect has taken it to a whole new level of full body integration into games and media control with the great adder of speech detection.

Josh: This leads us to the OpenNI initiative. Tell us about Open Natural Interaction, who is involved, and the role you see it playing.

Tamir: I believe OpenNI to be a major key player in Natural Interaction. As I’ve mentioned, Natural Interaction consists of hardware devices that we’ll see coming from a variety of CE [Consumer Electronics] vendors, and software modules that will be developed by many different software providers. Having all of those comply with one standard ensure that any application that is developed can deploy on a large number of systems with an install base that is not fragmented. As the first step we’ve worked with some of our partners to create OpenNI and at this point we’ve decided to launch it a bit earlier than expected due to the large amount of developers already working. Our initial goal was to launch the OpenNI standard a few months from now. Many industry leaders are interested and when they are ready they will join us in the management of the OpenNI consortium.

Josh: Along with OpenNI, PrimeSense released a whole suite of software, including some open source. What can the public do with this software, and how would you like to see the community engage with OpenNI? Are you open to people contributing to the software projects or extending it with new capabilities?

Tamir: The software is divided into two main parts, the first being the OpenNI standard which is mainly the APIs to devices and image processing algorithms. Extending OpenNI and submitting changes is welcomed. In the next versions, the OpenNI consortium will publish what will be included prior to closing up the version so everyone can suggest introducing their changes to the official release.

The second part is NITE which is PrimeSense’s middleware for its SoC (System on Chip) and includes UI framework for controlling the TV from 15ft away without the need to start searching for the remote. This framework takes a bit time to learn to use but provides great power to the developers, as it provides them with exactly what they need – events that are generated according to the user’s motions but that are still tailor made for their applications.

The other part NITE includes is the skeleton image processing module that enables the application developer to receive the joints of the user that initiated interaction (the way to start interaction is by standing with your hand to the sides in a ‘freeze’ position). After that you have a skeleton that sticks to you. In order to contribute even further we’ve also provided the “scene analyzer” which segments the users from the background so you actually can enumerate the depth map and get the number of users. You can see both the Skeleton and Scene Analyzer in these videos on Youtube

Josh: Those videos are awesome. I’m just imagining what I can do with those capabilities. Is there anything else you’d like to share or tell us about?

Tamir: Sure – I believe that till today the devices we’ve been using, made us learn greatly lot about them before we could use them and gain their value. I’m pretty sure everyone who is reading this has got at least 3 remotes sitting on his living room table, and at least once a week needs to help someone use their computer/media center/phone/etc. It’s time for that to change and it’s up to us, the technologists to make this revolution happen, it’s time for the devices to take the step of understanding what we want and making sure we get that, even without asking if it’s a trivial task as opening a door when we approach, closing the lights when we leave the room, even making sure we have hot water to shower with when we return from work or wake up in the morning, depends on what we normally do.

Having said that, we need to remember that now localization is not only changing the written language, it’s becoming the gestures that people are used to, the accent in speech detection. Even the difference between when a user wants to control and when they don’t is not so trivial anymore as it’s not just “press the button”… it’s up to us to make sure that the users enjoy a new set of very compelling experiences that don’t break. It’s a new huge growing market, and I’d suggest anyone with technological skills to jump on and enjoy the creation of this new era.

Josh: I’m very excited about all of this. Thanks again for answering these questions and I’m looking forward to working with you to make some amazing natural user interfaces and natural interactions.

InfoStrat.VE and the Bing Maps 3D end-of-life

On Tuesday, the Bing team announced that the Bing Maps 3D control (the ActiveX plug-in, not Silverlight) is being end-of-lifed. The 3D control will be removed from the website in a few weeks, but the data feeds that support it will remain live until December 2011. I've gotten some questions on twitter and the InfoStrat.VE forum about what this announcement means for InfoStrat.VE so I wanted to post an update about this.

Based upon this announcement, the current version of InfoStrat.VE will still work until December 2011. After that date, any applications using the current version will essentially stop working. We are pursuing several options that will replace as much of the functionality of the Bing Maps 3D backend to InfoStrat.VE as possible.

I'm really happy to see the support from people on twitter and comments on the Bing blog for InfoStrat.VE. Unfortunately the decision to end-of-life Bing Maps 3D was made much earlier this year. We had been working closely with some of the team members so they did give us a heads up back then, but we couldn't discuss it publicly of course.

At that time we provided a lot of feedback and as much push-back as we could, including several of the ideas that people posted in the comments on the Bing blog. This is also why you haven't seen much new lately for InfoStrat.VE. Of course, Microsoft and this project has always said using this control is unsupported with no guarantees.

We are disappointed that Microsoft is bowing out of the 3D mapping market, which is important to commercial and government customers. They've chosen to focus on 2D mapping, which primarily targets consumers. The 2D solution does have Birds Eye and some other cool features, though.

That said, this is not the end of mapping on Surface. InfoStrat.VE should still work in your existing projects until December 2011. We're hoping to have news of a transition path well before then in order to give time to switch over current and future projects. This might end up using 2D only, or possibly extending the backend to support other 2D or 3D mapping providers. We don't have anything solid at this point but we will definitely post some information when we can.

If you have any preferences about where you'd like to see InfoStrat.VE go, please chime in on the comments or send me an email: joshblake at gmail.com.

Thanks for your support!

Speaking about NUI at CapArea.NET Silverlight SIG

Tonight (10/19/2010) I'll be speaking at the CapArea.NET Silverlight SIG about NUI development with WPF and Silverlight. It is at 7pm at the Navy League Building at 2300 Wilson Blvd, Arlington VA 22201 and is easily accessible via the Court House Metro Station.

Abstract
The Natural User Interface (NUI) is the next revolution of human-computer interaction. Microsoft Surface has shown the potential of multi-touch NUIs to uniquely engage users, and multi-touch tablets and displays are becoming more and more common. In this presentation, Joshua will cover fundamental NUI and multi-touch concepts and discuss multi-touch development options with WPF 4 and Silverlight. You will learn about raw touch input and the manipulation API and see the differences between WPF and Silverlight multi-touch development. If you are interested in creating rich, engaging multi-touch interfaces for the web or client, then you won't want to miss this presentation!

If you are in the area, come by and say hi afterwards. See you there!

Recent NUI videos and links

In the last week or two there have been several interesting new links about NUI research and innovations. I thought I'd share them all together in hopes that they are interesting to you as well.

PhoneTouch
http://www.technologyreview.com/computing/26458/
Researchers at University of Lancaster are coordinating cell phone accelerator data timing with touch data on a touch table to uniquely identify users. When a user uses the phone to touch surface they can create user specific reactions, such as pulling data from that phone, as opposed to another person's phone. Of course, the phone needs to run some software and already be paired via bluetooth or Wifi but this is an interesting idea about sensor fusion.

Video:



LightSpace
http://news.cnet.com/8301-13860_3-20018451-56.html
Ina Fried has a good write up about a new Microsoft Research project by Andy Wilson, one of the founders of the Surface Computing group at Microsoft, and his team. They've taken Surface Computing a step further and using ceiling-mounted projectors and depth cameras (perhaps borrowing a bit from Kinect technology), have turned an entire room into a Surface. A standard wood table-top is now touch-sensitive Surface.

The interesting thing with this is that it can detect presence and coordinate interactions between the interactive zones. The video shows them picking up a video from the table, carrying a representative dot across the room and putting it onto a wall to play the video. They also show a menu system created in a column of air where you select an item by hovering your hand in the right place. The menu items are projected onto your hand.

The interface and interaction technique are still clunky and unrefined, but this is still a research project and they're exploring new capabilities. I think about these types of future interactions a lot. We're going to be seeing more and more of this integrated interactions and reacting to presence as well as touch in the future.

Video:



Manual Deskterity
http://www.zdnet.com/blog/microsoft/microsoft-courier-concepts-move-foward-on-the-surface/7570
Mary Jo Foley updates us about another Microsoft Research project, Manual Deskterity. A video of Manual Deskterity was released earlier this year and now the team has published a paper with more details of their findings and user research. The project focuses on multi-modal and bi-manual (two-handed) interactions.

They use an IR light pen and Microsoft Surface but the techniques are applicable to any type of pen and touch system. (Mary points out a connection to the leaked Courier videos last year and the InkSeine project.) Note that most commercial tablet PCs available today that support stylus and touch only allow one at a time. You'll need special drivers to do both at once.

Video:


Bill Buxton, "NUI - What's in a name?"
http://channel9.msdn.com/posts/TechTalk-NUI-Whats-in-a-Name
Bill Buxton recently gave another presentation about NUI, this time at the Microsoft Development Center in Copenhagen. The presentation is about 60 minutes with another 30 minute Q&A. It combines some of his previous speeches this year with a few new things. I'd encourage you to plan some time to watch this, even if you've already seen everything else he's ever presented. He has a great way of breaking down difficult-to-understand topics and making you see things in a new way.

Video:

I’m a Microsoft Surface MVP!

On Friday, October 1st, I got an email that I was hoping to get for a while. It started:

Dear Joshua Blake,

Congratulations! We are pleased to present you with the 2010 Microsoft® MVP Award! This award is given to exceptional technical community leaders who actively share their high quality, real world expertise with others. We appreciate your outstanding contributions in Surface technical communities during the past year.

The Microsoft MVP Award provides us the unique opportunity to celebrate and honor your significant contributions and say "Thank you for your technical leadership."

The email went on to tell me how to log in to the MVP site and that I should be expecting an award package.

The Microsoft MVP is a year-long award with new and returning MVPs announced every three months. The Surface MVP category was created just before the last cycle of MVP awards that went out July 1st. Dr. Neil from nsquared in Australia holds the honor of being the first Surface MVP since he made it in during the last cycle.

This cycle adds Dennis Vroegop in Netherlands, Rick Barraza from Cynergy in San Diego, CA, and me from InfoStrat in DC, bringing the number of Surface MVPs to four in the world. You can see our MVP profiles here. I’m very excited, proud, and humbled to be recognized among the best of the Surface community.

I have to say thanks for all of my readers and supporters. I plan to continue to contribute to the community and encourage others to do so as well. There are many new and exciting things coming up, both from me and InfoStrat as well as within the Surface community. It’s going to be a great year!

NUI session at Heartland Developers Conference

I'm here at the Heartland Developers Conference in Omaha, NE. If any of my readers are here, please look me up and we'll chat! You can email me or send me a message on twitter: @joshblake.

I'm presenting a session tomorrow (9/9/2010) at 1:15pm right after lunch:
Deconstructing the Natural User Interface: UX design in the NUI revolution 

I'll be using NaturalShow again, and there may just be an announcement. You'll have to come to the session to find out! (Or read my blog when I post a follow-up.)

Multitouch on Windows book update

Here is a quick update about the progress of my book, Multitouch on Windows. The MEAP (Manning Early Access Program -- pre-order access to the eBook) now includes chapters 1-2 and 4-7. You can get access the MEAP by pre-ordering either the eBook or print+eBook. (Disclosure: that link is affiliate and sends me a few extra dollars if you buy through it.) You can also download the source code samples from the book page at that link.

I'm currently working on chapter 8 about the WPF manipulations API. This is a key chapter so I'm giving it some extra love and care. I'm also balancing out work and family life. It is a challenge sometimes but this stuff motivates me so I push through. I'm also motivated by the comments, feedback, and successes of my readers. If you're doing something cool with multi-touch let me know!

Since I'm closing in on half-way through the book, I posted some survey questions for readers on the Manning forums. I basically need to know how important Silverlight and Surface content is to readers so I know how much effort to put into various areas. I'm also trying to build the case for the book to be printed in color (the eBook is already in color) so the more responses the better case I can present to Manning. Please take a few minutes and answer these four questions, even if you haven't read the book yet:

Multitouch on Windows reader survey

If you don't want to post the answers on that forum or don't want to make an account, you can always just email me directly: joshblake at gmail dotcom. Thank you to those of you who have already answered the survey.

Microsoft Surface and Magical Object Interaction

Work at InfoStrat has been really exciting recently. We are getting more and more multi-touch projects for both Surface and Windows 7. We've created some really interesting interfaces and applications and soon we'll be able to share more.

For now, I can share this video. We built a suite of Microsoft Surface applications for a well-known museum. These are designed to be used by young children. During user testing we found that it works great for them and also does pretty well at bringing out the inner child from people of all ages. I'll post more details on the findings of our user testing in a future post.

The theme of the applications was "light" and we designed them to really take advantage of the unique capabilities of Surface: bringing people together socially and physical object interaction.



Let me know what you think!

By the way, if you're interested in working for InfoStrat and you know WPF, we're hiring! Take a look at our job posting for WPF experts as well as Microsoft CRM and Microsoft SharePoint experts.

Sharing binaries between Surface and Windows 7

This post is part of a series about Blake.NUI, my open-source library for multi-touch natural user interface development with WPF 4 and Surface. In this post I discuss how to set up your projects to share 99% of your binaries between Surface and Windows 7. See my Blake.NUI announcement post for an overview of the project as well as a discount code for my book Multitouch on Windows, good until July 23.

In my last post, I showed how to use the SurfaceTouchDevice class to let WPF 4 Touch applications work on Microsoft Surface. In this post, I will discuss how to set up your project to allow you to reuse almost your entire WPF 4 code-base between a Surface SDK application and Win7 Surface Toolkit application. This approach will make your code binary compatible between the two platforms.

Note that as I discuss the technique, I am assuming that you are setting up your projects on a Windows 7 multi-touch machine with Visual Studio 2010 and both Surface SDK SP1 Workstation Edition and Surface Toolkit beta installed. Once the projects are setup, you can compile the solution on a Surface device as long as you create a solution configuration that does not build the projects that reference Surface Toolkit.

1. Project Structure

If you want to target both Win7 and Surface with the same code-base, you'll need to keep the Surface SDK-specific code separate from your reusable code. There are several possible approaches, but here is the one I have been using. Figure 1 shows the architecture of the solution.

Figure 1 Most of the controls and code is located in a common project that is consumed by a Surface project and a Win7 project.

I recommend creating a shell project for Surface and a shell project for Win7. Each of these projects has only a window. In the Surface project, add SurfaceTouchDevice.RegisterEvents(this) to the SurfaceWindow constructor. (I also add MouseTouchDevice.RegisterEvents(this) to both Surface and Win7 projects.) Then in a separate common library I put all of my MVVM code and added the same view user control to the window in each of the shell projects.

Note that the Surface project references the Surface SDK (v1.0) and the Win7 project references Surface Toolkit (v1.5). You will need to check the properties on the assembly references to ensure they are the correct version for each application. The Surface project probably is referencing v1.5 of Microsoft.Surface.Presentation.dll and Microsoft.Surface.Presentation.Generic.dll. Simply remove those reference and in the Add Reference dialog browse and add references to the 1.0 version of the assemblies.

Surface Toolkit contains a subset of controls from the full Surface SDK. This means that you can reference either version 1.0 or 1.5 of Microsoft.Surface.Presentation.dll in your common project and use any of the common controls that exist in both assemblies, such as SurfaceButton and ScatterView. As long as you stick to the subset of common controls, the same binary will work on both Surface and Win7. I would recommend using v1.0 in the common project so you can still compile the project on your Surface device without Surface Toolkit.

2. Loading the correct assemblies

Assuming that you do reference Microsoft.Surface.Presentation.dll v1.0 in the common project and Microsoft.Surface.Presentation.dll v1.5 in the Win7 project, you need to do one more thing to ensure that the application will load correctly. If you do not, then depending upon the order in which components load, you may get a runtime exception like the one shown in figure 2.

Figure 2. If you mix the assembly references, you may get an IOException: Cannot locate resource 'controls/touchvisualizations/touchvisualizationsresources/trailtop.png'. This exception occured on the first touch after the Win7 application loaded.

The solution to this is to add a versioning policy to redirect the v1.0 assembly to v1.5. This blog has a good overview of versioning policies. In the Win7 project, add this to the app.config within the configuration section:

<runtime>

  <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">

    <dependentAssembly>

      <assemblyIdentity name="Microsoft.Surface.Presentation"  

          publicKeyToken="31BF3856AD364E35" culture="neutral"/>

      <bindingRedirect oldVersion="0.0.0.0-1.5.0.0"  

          newVersion="1.5.0.0"/>

    </dependentAssembly>

  </assemblyBinding>

</runtime>

Just to be safe, you may want to also add the same thing to the Surface project's app.config, except change the newVersion to 1.0.0.0. This ensures that each project will always load the correct version of the assembly, provided it is installed.

I have committed a few updates to the Blake.NUI project, including TestApplication.Common, TestApplication.Surface, and TestApplication.Win7. These three projects illustrate the ideas discussed in this blog. Download the latest code from the Source Code tab on the CodePlex site.

3. Summary

We have learned how to structure a WPF 4 solution to target both Win7 and Surface while reusing almost all of our code in a common library. We also saw how to configure the projects to ensure they load the correct version of the Surface assemblies at runtime.

So far, we know that we can only reference the Surface controls that exist in both Surface SDK and Surface Toolkit. In the next post, I will discuss some techniques that can enable you to use the Surface SDK-specific controls such as TagVisualizer and ElementMenu while maximizing the code reuse.

How to write WPF 4 Surface applications with Blake.NUI

This post is part of a series about Blake.NUI, my open-source library for multi-touch natural user interface development with WPF 4 and Surface. In this post I discuss the origin of SurfaceTouchDevice and MouseTouchDevice and how to use them in your application. See my Blake.NUI announcement post for details as well as a discount code for my book Multitouch on Windows, good until July 23.

Between InfoStrat projects and writing my book, I've been neck-deep developing WPF 4 Touch and Surface applications for months. If you have used them both then you'll know that the WPF 4 Touch API is a little bit easier to develop for than the Surface SDK it is based upon. This is because WPF 4 touch and manipulation features are integrated directly into the framework. There are also a few other subtle differences, like not needing a separate ManipulationProcessor and InertiaProcessor.

On my current project, I needed to create a Surface and a Win7 version of the same application. I decided that I wanted to use .NET 4 on Surface. This would let me take advantage of the new animation easing functions in .NET 4 and it would let me reuse more code between both versions of the application. First problem: can I install Visual Studio 2010 and .NET 4 on Surface? I tweeted @Surface and Eric Havir replied. Figure 1 shows this twitter conversation.

Figure 1. A twitter conversation between me and Eric Havir from @Surface where he confirms .NET 4 would probably install but could not be used for Surface apps.

That is pretty much what I expected. It's been known since PDC09 that the roadmap for the future of Surface includes the WPF 4 Touch API on Windows 7, as shown in figure 2. This sounds like a great unified platform, but it doesn't help me now. The currently available Surface SDK was designed for .NET 3.5 SP1 and uses a custom Surface Input mechanism that doesn't feed into the WPF 4 touch system.

Figure 2. A slide from the PDC09 Multi-touch on Microsoft Surface and Windows 7 session.

I went ahead and installed Visual Studio 2010 on my Surface anyway because I wanted to use the easing functions. It turns out that Surface applications compile just fine under .NET 4 and even the SDK works, both in the emulator and on a Surface device. I figured I would just create some common controls and branch most of my project to target either the WPF 4 Touch with Surface Toolkit or the Surface SDK.

But wait! WPF 4 uses TouchDevices. TouchDevices are extensible. Anson Tsao, a Principal Program Manager who was responsible for the WPF 4 Touch API, wrote about creating custom touch devices in January.

So I'm thinking maybe if I create a custom touch device that listens to the Surface Contact events, I could process those events into WPF 4 TouchDevices and not even write against the Surface SDK.

Well once I had the idea in my head, I had to do it. I created a class, SurfaceTouchDevice, that derives from TouchDevice and promotes Surface SDK Contact events to WPF 4 TouchDevice events. This means that if you install .NET 4 on your Surface and write against the WPF 4 Touch API, It Just Works™.

Here's the best part. All you need to do to enable this dark magic is add this line of code to the constructor of your SurfaceWindow:

SurfaceTouchDevice.RegisterEvents(this);

That's it. Now the same code that works with WPF 4 Touch will also work on Surface. This also means you can develop with Visual Studio 2010. The SurfaceTouchDevice class will not work on a Win7 machine, though, since the Contacts class is not present in the Surface Toolkit.

In most cases you won't need to do anything else. Surface SDK controls, such as TagVisualizer, will react to and handle Contact events. Unhandled events will be promoted to WPF 4 Touch events by the SurfaceTouchDevice class, and then your code will react to the WPF 4 Touch events.

If you do want to check an actual Surface property, such as whether the TouchDevice represents a tag or a blob, you can get access to the original Contact by casting the TouchDevice to a SurfaceTouchDevice:

   1:  private void TouchDown(object sender, TouchEventArgs e)

   2:  {

   3:      bool isFinger = true;

   4:   

   5:      SurfaceTouchDevice device = e.TouchDevice as SurfaceTouchDevice;

   6:      if (device != null)

   7:      {

   8:          isFinger = device.Contact.IsFingerRecognized;

   9:      }

  10:      //Do something 

  11:  }

You can also filter out touches from your manipulations by iterating through e.Manipulations, casting each IManipulator to SurfaceTouchDevice, and calling Manipulation.RemoveManipulator for undesired IManipulators. I will probably add this feature to SurfaceTouchDevice in an update to Blake.NUI.

I also created a MouseTouchDevice class that operates similarly except promotes mouse input to touch input. This is helpful so I do not need to create MouseDown/Move/Up events when I'm creating a custom touch control with TouchDown/Move/Up.  It also means the mouse can participate in the manipulation processor.

Both SurfaceTouchDevice and MouseTouchDevice are available, free and open-source, in Blake.NUI, my library of helpful multi-touch NUI code for WPF 4 and Surface.

Now that SurfaceTouchDevice is available, I cannot think of a reason why you would not want to install Visual Studio 2010 on your Surface and write all applications with WPF 4. In my next post, I'm going to talk about how to set up your application architecture so you can share 99% of your code with no changes or branches for Microsoft Surface and Win7.

Announcing Blake.NUI - a multi-touch NUI library

Almost all my free time and creative efforts recently have been focused on my book, but I've been able to put together something really cool that I think will be useful for anyone doing multi-touch natural user interface work with WPF. Actually, it's a couple really cool things packaged into one.

I'm happy to announce Blake.NUI, an open-source library for multi-touch NUI development. Blake.NUI is available under the MS-PL license at CodePlex:

http://blakenui.codeplex.com

It is a collection of the most useful reusable code that I have created for my own projects and for my book's code samples. It currently includes code for WPF 4 and Surface, although in the future I may add Silverlight code as well.

The library is still in undergoing changes. I may add or rearrange things, but I will document changes from version to version. Here are some of the things currently included in Blake.NUI:

• Controls
• ZoomCanvas - an infinite canvas control that is the heart of NaturalShow
• CircleText - layout text along the circumference of a circle
• Gestures
• TapGestureTrigger - a configurable Blend Trigger that fires when the user completes a tap gesture
• DoubleTapGestureTrigger - a configurable Blend Trigger that fires when the user completes a double tap gesture
• TouchDevices
• MouseTouchDevice - promotes mouse input to WPF 4 touch events
• SurfaceTouchDevice - promotes Microsoft Surface vision system input to WPF 4 touch events
• Common classes and utilities
• DisplayMatrix
• AnimateUtility and SurfaceAnimateUtility
• ScatterMatrixHelper
• MathUtility
• VisualUtility
• NaturalShow
• Common models, views, factories, and other classes

The full NaturalShow application is not yet included, but many of the core classes are. I'm updating NaturalShow to use Blake.NUI rather than its older, internal versions of the same code.

I'm particularly excited about SurfaceTouchDevice, since with one line of code it enables you to write multi-touch applications against the WPF 4 Touch API, including the integrated touch events and manipulations, and run it on your existing Microsoft Surface devices without modification. This also means you can, with a simple project structure, target both Win7 Touch and Surface v1 with 99% of the same code base! You do have to install .NET 4 and/or Visual Studio 2010 on your Surface.

I will be celebrating the launch of Blake.NUI all this wee by posting a series of blog posts about how to use the library. In addition, you can buy the ebook edition of Multitouch on Windows until July 23rd for 35% off through this link using coupon code blake35bl. (Disclosure: if you buy after clicking this affiliate link it gives me a few extra dollars from the sale.)

I am asking as a courtesy that if you find Blake.NUI useful, particularly for commercial projects, that you buy a copy of my book, Multitouch on Windows. The coupon code provided should help. 

Please let me know what you think about Blake.NUI and whether you think you'll use it in your applications.

NaturalShow and new interviews

Coincidentally, two different shows featuring me talking about NUI were posted yesterday, even though the interviews were several weeks apart.


Brian Peek interviewed for for his Channel9 Coding4Fun show. We talked about NUI and NaturalShow and I showed some of the actual code and how it works.
http://channel9.msdn.com/shows/Coding4FunTV/The-Coding4Fun-Show-Natural-User-Interfaces-with-Josh-Blake/


I also spoke at Richmond Code Camp about NUI last mont and while I was there I was included in a segment of the The Community Megaphone Podcast hosted by Dane Morgridge and Andrew Duthie. I'm the second to last interview.
http://www.communitymegaphonepodcast.com/

I also found out recently that my session submission for the Heartland Development Conference was accepted! HDC is September 8-10 in Omaha, NE.
Deconstructing the Natural User Interface: UX design in the NUI revolution

Finally, I want to make the announcement on my blog that I will be making NaturalShow open source and it will be made available along with my book. I'm going to integrate it with some of the topics in some of the chapters. MEAP subscribers will get the first access to it, although those chapters are not complete yet. I'll be sure to announce when they're included.

[Update: fixed links, thanks Ben!]

Why won't WPF controls work with touch manipulations?

I recently tweeted asking for some scenarios using WPF Touch, and I got an email from Josh Santangelo (@endquote) with few interesting touch challenges. I created some sample code and sent him back some solutions to his challenges, but I figured they would be useful for the community.

In this blog, I'll answer the question: "Why won't WPF controls work with manipulations?" (This is my own phrasing, not Josh Sanangelo's.)

Problem: 
You have a container with some manipulations (perhaps like a ScatterView) and this container has some standard WPF controls like buttons or checkboxes. You can use touch to manipulate the container, and you can use the mouse to click the controls, but touch mysteriously doesn't affect the controls. (The mouse also can't manipulate the container, but that is another post.)

Reason:
When you touch the screen, WPF generates touch events, such as TouchDown, TouchMove, and TouchUp. These are all routed events, which means that first the PreviewTouchDown event is fired on the root of the visual tree, then the next element down the hierarchy, then the next, all the way down the source element that the touch event occured over, as long as the event is not handled along the way.

Once it reaches the source element, the TouchDown event is fired starting from the source element and proceeding up the visual tree to the root. At any point if any of the visual elements set e.IsHandled = true, then the event propagation stops.

On the other hand, if the event propagation reaches all the way up to the root, then the touch event is promoted to a mouse event. At this point, PreviewMouseDown and MouseDown is fired down and up the visual tree.

In our case, when you use touch and there is no manipulation, the touch events return unhandled, then the mouse events are fired and the button handled that and calls your Click event handler. The WPF controls such as button only listen for mouse events (exception: ScrollViewer when PanningMode is set.) This process is illustrated in figure 1.

Figure 1. (Click for larger size.) The touch event flow with no manipulations. The touch events are unhandled, so WPF promotes the event to the mouse equivalent. The button is listening for mouse events and handles that, then calls your Click event handler.

When you do have manipulations enabled in the visual tree, something different happens, as illustrated in figure 2.

Figure 2. (Click for larger size.) When the border has IsManipulationEnabled = true, the manipulation processor handles the TouchDown event and captures the touch device. All future events go directly to the Border, and the ManipulationProcessor handles the rest.

In the case where the Border has a manipulation, the touch event flow is never promoted to mouse events, so the button doesn't even have any idea what is going on. Button didn't get the memo.

Solution:
Of course, it doesn't have to be this way. Sometime soon, the Surface Toolkit for Windows Touch will be released and you'll be able to use the SurfaceButton and the other controls which are designed for touch and handle the touch events. Figure 3 shows a list of the WPF controls the Surface toolkit optimizes for touch.

Figure 3. (Click for larger size.) Surface Toolkit for Windows Touch offers touch-optimized controls that can replace most of the common WPF controls. This is only a partial list of what the Surface Toolkit offers.

But that doesn't help you now. Suppose you have this XAML inside of a container with manipulations:

   1:  <TextBox Name="txtCounter"

   2:           Text="0"

   3:           Margin="10"

   4:           HorizontalAlignment="Center" />

   5:  <Button Content="Native Button Won't work"

   6:          Margin="10"

   7:          Height="40"

   8:          Click="button_Click" />

and this code in the code behind:

   1:  private void button_Click(object sender, RoutedEventArgs e)

   2:  {

   3:      IncrementCounter();

   4:  }

   5:   

   6:  private void IncrementCounter()

   7:  {

   8:      int number = int.Parse(txtCounter.Text) + 1;

   9:      txtCounter.Text = number.ToString();

  10:  } 

This button will not work and the event flow will look like figure 2. Instead, update the button XAML to this:

   1:  <Button Content="Will work with TouchDown/Up"

   2:          Margin="10"

   3:          Height="40"

   4:          Click="button_Click"

   5:          TouchDown="button_TouchDown"

   6:          TouchUp="button_TouchUp" />

and add these methods in the code behind:

   1:  private void button_TouchDown(object sender, TouchEventArgs e)

   2:  {

   3:      FrameworkElement button = sender as FrameworkElement;

   4:      if (button == null)

   5:          return;

   6:              

   7:      button.CaptureTouch(e.TouchDevice);

   8:   

   9:      e.Handled = true;

  10:  }

  11:   

  12:  private void button_TouchUp(object sender, TouchEventArgs e)

  13:  {

  14:      FrameworkElement button = sender as FrameworkElement;

  15:      if (button == null)

  16:          return;

  17:   

  18:      TouchPoint tp = e.GetTouchPoint(button);

  19:      Rect bounds = new Rect(new Point(0, 0), button.RenderSize);

  20:      if (bounds.Contains(tp.Position))

  21:      {

  22:          IncrementCounter();

  23:      }

  24:              

  25:      button.ReleaseTouchCapture(e.TouchDevice);

  26:   

  27:      e.Handled = true;

  28:  }

Now your button will work inside of the container with both mouse and touch. A little explanation:

• Line 7: We capture the touch so that the TouchUp and other touch events will be sent to this button, even if it occurs somewhere else.
• Line 9: We must set this event to handled, otherwise the TouchDown event will continue to bubble up and the border's manipulation processor will steal the capture, depriving the button of a TouchUp event.
• Lines 18-20: We check to make sure the TouchPoint is still within the bounds of the button. The user could have touched the button, but changed his or her mind and moved outside of the button to release the touch. This is consistent with the mouse behavior of buttons.
• Line 22: We just call the same function that the button_Click() event handler called to get the same effect.
• Lines 25-27: We release the capture and handle this event as well. Technically we might be able to get away without these lines in this scenario but we should do it anyway. In a more complicated scenario there might be an unintended side-effect if we do not do this.

You can apply this technique to the other non-touch aware WPF controls if necessary, although it may get a little tedious. I almost wrote a Blend behavior (part of the Blend 3 SDK and usable in more than just Blend) for this, but figured that the Surface Toolkit will be out soon enough anyway.*

(* No I don't have a date for the release of Surface Toolkit. Sorry!)

You can download the source code for this project below:

ButtonInManipulation.zip

Thanks to Ryan Lee for Gesturecons, where I got the hand icon used in the figures.

NUIs reuse existing skills (updated NUI definition)

Last month I posted an excerpt from my book with a section that contained a definition of natural user interface.

A natural user interface is a user interface designed to use natural human behaviors for interacting directly with content.

It sparked some great conversation both in the comments of that post and some other blogs. Richard Monson-Haefel suggested on his blog that I consider changing the "use natural human behaviors" part to match how I talk about natural in terms of innate abilities and learned skills. Those discussions were derived from some of Bill Buxton's thoughts on NUI.

At the time, I was on the fence about this change. I wrote in a comment that I liked using the simpler vocabulary words since it will be understandable by more non-technical folks, and that I liked the term behavior for a rhetorical reasons since in the beginning of chapter two I had an interesting set of phrases that needed it to say "behavior" to make sense. (I know that isn't a very good reason and actually I since deleted those pages with that particular discussion.) So I was leaning towards my initial definition but I was keeping my mind open for more convincing arguments.

Well as it turned out, I was convinced by Bill Buxton at MIX10. He wasn't talking directly to me about this, but in his keynote he described how a saxaphone input device is a natural interface for someone who has years of saxaphone training, and he proceeded to use his electronic saxaphone (a Yamaha WX7 wind controller) to create flute and electric guitar sounds. Below is a screenshot, but if you want to watch his part of the keynote, skip to mark 96:45 of the MIX10 Day 2 Keynote video.

Bill Buxton uses a Yamaha WX-7 wind controller and his existing saxaphone skills to create music that sounds like a variety of non-saxaphone instruments.

When I heard him describe this, I immediately understood that my existing definition ("use natural human behaviors") excluded advanced NUIs designs for people who were already experts with advanced skills. I had already prepared my NUI presentation for day 3 including the original definition, so directly after the keynote I went and changed it. In my session video you can see I used a revised definition of NUI, and in the next revision of chapter 1 (now updated in the MEAP) I also updated the definition.

Here is the updated book excerpt:

There are several different ways to define the natural user interface. The easiest way to understand the natural user interface is to compare it to other type of interfaces such as the graphical user interface (GUI) and the command line interface (CLI). In order to do that, let's reveal the definition of NUI that I like to use.

A natural user interface is a user interface designed to reuse existing skills for interacting directly with content.

There are three important things that this definition tells us about natural user interfaces.

NUIs are designed

First, this definition tells us that natural user interfaces are designed, which means they require forethought and specific planning efforts in advance. Special care is required to make sure NUI interactions are appropriate for the user, the content, and the context. Nothing about NUIs should be thrown together or assembled haphazardly. We should acknowledge the role that designers have to play in creating NUI style interactions and make sure that the design process is given just as much priority as development.

NUIs reuse existing skills

Second, the phrase "reuse existing skills" helps us focus on how to create interfaces that are natural. Your users are experts in many skills that they have gained just because they are human. They have been practicing for years skills for human-human communication, both verbal and non-verbal, and human-environmental interaction. Computing power and input technology has progressed to a point where we can take advantage of these existing non-computing skills. NUIs do this by letting users interact with computers using intuitive actions such as touching, gesturing, and talking, and presenting interfaces that users can understand primarily through metaphors that draw from real-world experiences.

This is in contrast to GUI, which uses artificial interface elements such as windows, menus, and icons for output and pointing device such as a mouse for input, or the CLI, which is described as having text output and text input using a keyboard.

At first glance, the primary difference between these definitions is the input modality -- keyboard verses mouse verses touch. There is another subtle yet important difference: CLI and GUI are defined explicitly in terms of the input device, while NUI is defined in terms of the interaction style. Any type of interface technology can be used with NUI as long as the style of interaction focuses on reusing existing skills.

NUIs have direct interaction with content

Finally, think again about GUI, which by definition uses windows, menus, and icons as the primary interface elements. In contrast, the phrase "interacting directly with content" tells us that the focus of the interactions is on the content and directly interacting with it. This doesn't mean that the interface cannot have controls such as buttons or checkboxes when necessary. It only means that the controls should be secondary to the content, and direct manipulation of the content should be the primary interaction method.

I decided to say just "reuse existing skills" and not include the innate abilities part to keep it simpler and also because the thing that convinced me to change it was the fact that some NUIs may use advanced skills, so innate abilities may not always play into NUI. The core thing that did make it natural was reusing existing skills.

I do need to thank the commenters on my last blog, Ben and Laurent, and particularly Richard, who had also put himself out there trying to define NUI but still had great comments about my thoughts. Richard pointed me in the right direction and with some time and thought I realized that he was correct. It's you, my readers, who will help me make this book totally awesome.

Please keep the feedback coming and feel free to call me out if you think I could improve something.

MIX10 NUI session sample code

I promised that I would provide the source code for the TouchGrid and ScatterTouch samples for WPF, Silverlight for web, and Silverlight for Windows Phone 7 from my MIX10 session. I provided a zip file with the sources to the people running the MIX website after my session, but it still hasn't been posted yet. So I'll just post it here on my blog.

http://www.handsonnui.com/files/MIX10_NUI_EX18.zip

Important: I cannot re-distribute the Microsoft Surface Manipulations and Inertia Sample for Microsoft Silverlight files so you will have to download that yourself and add a reference to System.Windows.Input.Manipulations to these projects SL_ScatterTouch and WP7_ScatterTouch. See the readme.txt for more details.

Of course if you want to run the Windows Phone 7 samples you'll need the Windows Phone 7 SDK.

Have fun!

MIX10 NUI Recap

Well I'm pretty much recovered from MIX10 by now (March 15-17 in Las Vegas). The whole experience was a blast. Here are the highlights for me:

Windows Phone 7 Series - Finally we learn the full scoop of the WP7 development story. Very excited about the Silverlight and XNA focus. I'm also relieved to learn that WP7 flavor of Silverlight has manipulation processors built in. It isn't quite the same as the WPF manipulations or even the Silverlight manipulations sample (no rotation support) but it makes multi-touch development feasible.

NUI MIXup - I organized a get-together for NUI and multi-touch enthusiasts Monday night. Marc Schweigert (@devkeydet) was able to get sponsorship from his team for drinks and appetizers for the group. It was well attended, about 25 or 30 people. We even had Dr. Neil and Bill Buxton (yes, keynote favorite Bill Buxton!) hang out! After we chatted for about an hour in one of the conference spaces, we moved down to the Border Grill where Marc treated us to drinks and light dinner. See some pictures below, thanks to fellow presenter and NUI enthusiast Mario Meir-Huber (@mario_mh):

Bill Buxton, Ben Reierson (@seraph_ben), and Dr. Neil chatting about Windows Phone 7 design.

The whole group of us heading towards dinner. Marc leads the charge on the left, and I'm in the middle with the black shirt chatting with James Chittenden (@okayjames).

Just about the highlight of my whole week, I had the opportunity to chat with Bill for over an hour at dinner along with several others that attended. It was a humbling experience.

NUI Development session - I didn't go to many sessions in person (they're available for download) because I spent a lot of time Monday and Tuesday putting the final touches on my presentation for Wednesday. I prepared a custom multi-touch presentation tool called NaturalShow that takes the place of PowerPoint. Even Tuesday evening I was sweating some last minute changes in the application, trying to maximize the performance. I also have to thank Josh Wall for listening to at least two dry runs and giving me great advice and feedback.

You can watch or download my presentation at the MIX10 site.

The presentation itself went pretty smoothly. The session had great attendance (144+) and the audience seemed engaged. My demos worked as expected, and NaturalShow didn't crash or otherwise behave abnormally. I included a hashtag #MIX10NUI so that twitter inclined people could follow the conversation. One posted this photo mid-presentation:

Taken from about 1/4 of the way back. The smudges around the word "design" are auras of my fingers starting to zoom in.

I have to admit that part of the reason I threw out the #MIX10NUI hash tag was so I could easily go back and read real-time feedback. When I did I was blown away by the things people said. There were two fair complaints (I took a shortcut when I described Occam's Razor and I forgot to zoom in on code, even though I had practiced doing that) but overall everyone said great things. At the risk of seeming self-focused for a moment, here are two of the most superlative comments that really made my day:

My session was also mentioned in several other people's conference recap blog posts: Luke Wroblewski, Cory Plotts, Roland Weigelt, Ben Reierson.

I guess the point here is that MIX10 was very much about the networking and making new acquaintences as well as meeting online friends for the first time. I'm still pumped by getting to meet so many awesome people, including the ones mentioned above, Steve Dawson, Pete Brown, Laurent Bugnion, Sean Getery and his crew, and many others I'm know I'm missing. Many times it was just a challenge to associate someone's real name with their twitter handle.

After the conference itself died down, Josh Wall and I went out to dinner at Hard Rock Cafe, checked out how they were using Microsoft Surface and a custom touch wall, and chatted about how we were going to apply the loads of information and ideas we gathered at MIX10 to improve InfoStrat.

This touch wall at Hard Rock let you explore memorabilia and was very popular.

Josh Wall testing the limits of interaction design at the Hard Rock Surface deployment.

This girl played with a bar-embedded Surface for over an hour.

Thank you so much to all the people I met and those who attended my session. I hope that we will continue our conversations throughout the year, and I can't wait for MIX11!

NUI at MIX10

I'm headed to Las Vegas tomorrow for the MIX10 conference. I'm excited about a lot of the topics and news that will be discussed. It will be great to meet a lot of the people I've only talked to online and meet new people.

If you'll be at MIX10, I'd love to meet you! Here are two places I'll definitely be:

NUI MIXup, Monday the 15th, 7pm, The Commons
I'm organizing a tweetup at MIX10 (hence, MIXup) for NUI and multi-touch enthusiasts. Come hang out and chat with others about our shared interests. Depending upon how many people come, we'll chat for a bit then head to a bar or restaurant. Marc Schweigert will be buying a round or two.

Developing Natural User Interfaces session, Wednesday the 17th, 10:30am, Lagoon F
This is my session! I'm very very excited about it (and thank you again to those who voted for me in the Open Call). I've created a custom multi-touch presentation application that takes the place of PowerPoint so I can walk the walk as I discuss how to create NUIs and multi-touch applications.

I'm available on twitter @joshblake or by email joshblake at gmail dot com. Send me a note if you'll be coming.

See you there!

What is the natural user interface? (book excerpt)

[Update 4/6/2010: Based upon feedback and careful thought, I have slightly modified my preferred definition. See my new post.]

Recently, there has been some discussion on establishing a definition for the term "natural user interface". My friend Richard Monson-Haefel (who just signed with O'Reilly on an iPad SDK book, congrats!) went through several iterations of a definition on his blog and ended up with this:

"A Natural User Interface is a human-computer interface that models interactions between people and the natural environment."

Wikipedia also has a paragraph describing natural user interfaces as invisible interfaces and lacking keyboard and mouse, but did not have a real concise definition. Ron George was a major contributor to the NUI wikipedia article. The first sentence says, in part, that a natural user interface is:

"...a user interface that is effectively invisible, or becomes invisible with successive learned interactions, to its users."

NUIGroup.com also has a wiki page with a lot of description language on the natural user interface, but no concise definition.

As you may have heard, I'm writing a book Multitouch on Windows. A key part of my approach is teaching the readers not just the APIs but also the new ways of thinking required for creating natural user interfaces. My first chapter is titled "The natural user interface revolution" (appropriate since it was also the title of my first blog post) and so right up front, I had to tackle the problem of defining natural user interface for my readers in a concise and comprehensive way.

I took into account both Richard's and the Wikipedia article's approaches, but I was not satisfied with what they had. I think Richard is on the right track, but the way he phrases it seems limiting. Whether or not he intended it this way, modeling the interactions between people and between people and the natural environment implies rather literal interface metaphors with NUI interactions that simulate real-world interactions, but there is no reason why this should be so. The Wikipedia's description talks about invisible interfaces, but to a lay-person this does not make sense and requires additional explanation of what an invisible interface means.

Now, I don't necessarily disagree with how Richard and the Wikipedia article are describing NUI. NUI does have something to do with how people interact with the environment, and NUI interfaces do seem to be invisible, but why are these descriptions true? To help figure this out, I turned to Bill Buxton's presentation in January where he talked about natural user interfaces. I took detailed notes and one particular thing that he said really resonated with me:

An interface is natural if it "exploits skills that we have acquired through a lifetime of living in the world."

I used that definition to write a section in chapter 1 on what "natural" means, and then derived my own definition. Below is an excerpt from chapter 1 of my book where I present my definition for natural user interface.

There are several different ways to define the natural user interface. The easiest way to understand the natural user interface is to compare it to other type of interfaces such as the graphical user interface (GUI) and the command line interface (CLI). In order to do that, let's reveal the definition of NUI that I like to use.

A natural user interface is a user interface designed to use natural human behaviors for interacting directly with content.

There are three important things that this definition tells us about natural user interfaces.

NUIs are designed

First, this definition tells us that natural user interfaces are designed, which means they require forethought and specific planning efforts in advance. Special care is required to make sure NUI interactions are appropriate for the user, the content, and the context. Nothing about NUIs should be thrown together or assembled haphazardly. We should acknowledge the role that designers have to play in creating NUI style interactions and make sure that the design process is given just as much priority as development.

NUIs use natural human behaviors

Second, the phrase "designed to use natural human behaviors" tells us that the primary way humans interact with NUI is through our natural behaviors such as touching, gesturing, and talking, as well as other behaviors we have practiced for years and are innately skilled at. This is in contrast to GUI, which is described as using windows, menus, and icons for output and pointing device such as a mouse for input, or the CLI, which is described as having text output and text input using a keyboard.

At first glance, the primary difference between these definitions is the input modality -- keyboard verses mouse verses touch. There is another subtle yet important difference: CLI and GUI are defined explicitly in terms of the input device, while NUI is defined in terms of the interaction style. Any type of interface technology can be used with NUI as long as the style of interaction focuses on natural human behaviors.

NUIs have direct interaction with content

Finally, think again about GUI, which by definition uses windows, menus, and icons as the primary interface elements. In contrast, the phrase "interacting directly with content" tells us that the focus of the interactions is on the content and directly interacting with it. This doesn't mean that the interface cannot have controls such as buttons or checkboxes when necessary. It only means that the controls should be secondary to the content, and direct manipulation of the content should be the primary interaction method.

Excepted from Multitouch on Windows by Joshua Blake
Chapter 1, "The natural user interface revolution"

I think this definition is very powerful. It gets right to the core of what makes natural user interfaces so natural in a way that does not restrict the definition to particular input technology or interaction pattern. It also can support the points-of-view presented by Richard and on Wikipedia, but in a more general way. 

By talking about directly interacting with content, we establish that content interaction should be primary and artificial interface elements should be secondary and used only when necessary. This is an easier way to say the interface is invisible. 

By framing the definition around natural human behaviors, we can talk about reusable patterns of behavior derived from human-human and human-environment interaction without implying we should model the interface after specific interactions. We can apply natural behaviors by reusing existing skills, which is what Bill Buxton was talking about. In the chapter, I spend a lot of time discussing these skills and how to apply them.

If you would like to read more on this, the entire chapter 1 is available for free download from Manning, where you can also pre-order the MEAP and read chapters as I write them.

Multitouch on Windows MEAP now available!

I've been working on a Manning Book called Multitouch on Windows with a subtitle NUI Development with WPF and Silverlight. I'm writing this book to accomplish two things. One, of course, it will teach you about developing multitouch applications using WPF 4 and Silverlight. The other, and perhaps more important, thing is to teach you about the new ideas and new ways of thinking required for creating natural user interfaces. There is a lot of talk about NUI at a high level, and a lot of interest in multitouch development and APIs, but there are not many resources out there to bridge the gap. This book does bridge the gap, and I hope it will give you everything you need to create multitouch application that live up to the potential of the natural user interface.

Today Multitouch on Windows became available under the Manning Early Access Program (MEAP). That means that if you pre-order the book, you get access to the chapters as I write them, before they go through the full editing and review process. This gets you the information now, and also gives you the opportunity to provide feedback, if you choose, and help improve the overall book. If you get the MEAP, then whenever I send Manning a new chapter or update an existing one, they will send you an email with a download link. When the book is complete, you'll also get the final e-book, or the e-book and printed book, if you choose that option. (There is a MEAP FAQ for more details on this.)

Right now chapter 1 "The natural user interface revolution" and chapter 4 "Your first multitouch application" are available to MEAP subscribers. Chapter 1 is also available as the free chapter, and you can download it right now! As this is an early look at the book, there are still a few quirks in the formatting and the illustrations, but all the content is there.

Please take a look at chapter 1, available from Manning's book page. It has a lot of very interesting ideas about natural user interfaces that build upon what I talk about in this blog. I hope you'll decide to subscribe to the MEAP. I really want your feedback, too. Feel free to email me directly or participate in the Author Online forum set up for this purpose.

To go along with the book, I have also setup a website at www.HandsOnNUI.com. This site is about the book, what events I will be speaking at, and general natural user interface resources. Right now there isn't much there, but I have big plans for it. The only thing I will say now is that the site will soon be a complete multi-touch experience.

One last housekeeping note: Prior to yesterday, I had been referring to this book as Multi-touch development with WPF and Silverlight. After some conversations with the very talented people at Manning, we decided to update the title as it is now. I only mention this in case there is any confusion.