I wanted to jump in and let you guys know that the Kinect for Windows SDK v1.7 is about to come out. It will be available for download on Monday, March 18th from http://www.kinectforwindows.com.
I have been using Kinect since the beginning, having started the OpenKinect community which created the first hacked Kinect drivers for PC, but today prefer developing using the Kinect for Windows SDK. I use the Kinect for Windows SDK for both work and personal projects and it has come a long way since the first release. Version 1.7 will be their fourth release in 14 months, and each release has promised something new. It already had the best API design for developing Kinect applications, but this new release adds several new important features that puts it over the top.
There are two major features new in Kinect for Windows SDK 1.7:
Kinect Interactions - an API and set of controls with a really mature design for general purpose hand tracking Kinect interactions. No longer do we each have to re-invent the wheel for Kinect hand interactions! Kinect Interactions are more advanced than the hand tracking used with Xbox 360 - the Kinect for Windows SDK can now accurately track and respond to both grip and push interactions! This makes it super easy to make hand tracking interactions, and will help lead to some consistently among applications.
Kinect Fusion - Finally, we now have access to the 3D scanning technology known as Kinect Fusion. This is the technology first developed and demoed by Microsoft Research in 2011. Kinect Fusion in 1.7 lets you use a Kinect for Windows sensor to scan and export high resolution static 3D models. As a side effect, this also allows applications to track the movement of the sensor with 6 degree-of-freedom, which is particularly useful if the sensor is hand held.
I recently spent some time visiting with the Kinect for Windows team in Redmond during the Microsoft MVP summit. They gave us a preview of these features, and while I cannot talk about certain details, I know that they have spent a significant amount of time perfecting Kinect Interactions and Kinect Fusion. In particular, the hand cursor designs and tuning have taken thousands of hours of testing and refinement, and the hand grip recognizer is extremely robust only because they invested many months and many people into perfecting it.
I've been neglecting this blog a bit since I'm trying to focus on finishing my book, but I had to break my self-imposed radio silence to break this news:
I installed Windows 8 on Microsoft Surface 1.0 hardware, and it works! By "it works", I mean the Microsoft Surface 1.0 shell and applications run as expected on Windows 8, and immersive-mode applications and regular desktop applications can be controlled using the Surface vision system touch input.
Windows 8 Start screen working on Microsoft Surface 1.0
I tweeted the above photo while video taping proof of this technical hackery. The video is embedded below:
Yes, this is the Windows 8 that was released publicly only a little over a day ago. I did not have early access, so this is another 24 hour hack for me like Kinductor from the Kinect Code Camp, though it didn't take me a full 24 hours of straight effort.
How did I do it?
Since Surface 1.0 is intended to run on Vista, the fact that any of this works is a testament to the backwards compatibility story of Windows 8. In short, I shrunk the Surface's Vista partition then installed Win 8 from USB key to the new partition (make it 40 GB minimum if you want to develop on it). The installer automatically set up dual booting, so I can boot back to Vista for the "correct" Surface 1.0 environment if I like.
The tricky part was getting the Surface SDK (full version that comes on Surface, not the workstation edition) to install (edit the msi launch conditions using Orca) and then getting the inputs from the Surface vision inputs to drive the Windows 8 UIs. To do that I used MultiTouchVista, which has a component that can connect to SurfaceInput.exe as a client as well as push touch data to an HID touch input driver. Windows 8 doesn't know it's not a capacitive touchscreen at all. (This technique also works for combining Windows 7 with Surface.) Win8 UI also won't be aware of tags or blobs but if you want to write a Surface application that uses those, just use the Surface SDK! Some other time I'll post a deep technical walkthrough, but this thread should be sufficient to get you started.
All of the Surface hardware drivers and Surface applications run great. The immersive mode touch responsiveness is a bit flaky as you can see in the video, but this could have been any number of things from the MultiTouchVista plug-ins to the Win8 touch stack to the Start screen. The Win8 touch visualizations (trails and auras) were always working fine, so I'm leaning towards the problem being in the Win8 UI layer. This is a pre-beta build though, so entirely understandable.
I do need to say that this configuration is completely unsupported and would probably violate the warranty if it wasn't already expired for this particular machine. It also has nothing to do with Surface 2.0. You won't ever see Microsoft create a configuration like this for general use since running three shells (Surface, Immersive, and Desktop) side-by-side would be entirely confusing for real users. In the video I was basically in the equivalent of Surface administrator mode, manually running SurfaceShell.exe, and then switching ad-hoc into immersive mode, which is like a full screen start menu. There are too many competing metaphors and modes for this configuration to be used by the general public.
Does Immersive mode make the Surface shell obsolete?
No way. It makes the Surface shell even more important. My initial impressions with actually using the Win8 immersive mode on Surface is that it really doesn't work for a large, horizontal, multi-user form factor. For one, everything is oriented in one direction, and two, I need to reach too far to get to certain interface elements across the table. That's the opposite of the ergonomic intent of the immersive mode, which was design for smaller screens and slates. It might do better on a vertical wall-mounted form factor. There is definitely a place here for both Immersive mode as well as the Surface shell since they are each designed to do different things in different scenarios. This is especially true if we account for the improved Surface shell that is coming soon with Surface 2.0 hardware.
I think we'll see that there will is a large difference between the design of an immersive mode application to run on a Slate or PC and used by one person, and the design of Microsoft Surface applications to be used by multiple people in a 360 degree interface.
What Surface and Win8 can learn from each other
Even though the designs for each of these modalities need to be different, there is a lot that they can learn from each other. I really like swiping in from the edge to open the charms and app bars, and could see that being incorporated into the version of Surface after 2.0. Windows 8 isn't perfect though, since as soon as there is an extra finger somewhere in immersive mode, the whole interaction breaks down.
It may just be the developer bits, but many or most of the Win8 sample games were almost unusable as a Surface application. My 3 year old tried to play a few with me but we kept getting in each others way since the games were designed for a single orientation. Worse than that, my touches would block her from interacting with the game, and vice versa. Some of the interns that developed those samples apparently did not account for multiple types of input (mouse and touch) or even multi-touch at all, indicating they were still thinking about coding interactions from a single-point mouse point-of-view.
I know that on slates and PCs with a single user they may not need to plan for a full 360 degree interface, but Windows 8 developers need to learn the lessons that Microsoft Surface developers have learned years ago about allowing multiple inputs, multiple touches, and letting the user recover from mental mistakes and accidental touches.
Watching the Windows 8 unveiling during the Build Day 1 keynote was somewhat amazing for me because all of a sudden NUI design and development skills (my focus since 2008) has gone from a small niche to a core requirement for the Windows platform. Anyone interested in building truly great Windows 8 applications needs take an active interest in learning from Surface developers. I'm expecting that some of my colleagues in the Surface community will end up coming up with the most amazing Windows 8 applications in no time at all. (Feel free to send those apps to me for testing on my FrankenSurface Window 8 machine!)
Natural User Interfaces in .NET While at first I was nervous that the Windows 8 announcements would make my almost-complete book a waste, now that I think about it things are still pretty good. If you want to use the premier technology for creating Natural User Intefaces, then .NET and WPF is still the way to go. You can't target Surface or Kinect in immersive mode, and the immersive mode interface isn't well suited for multiple people. You also have to consider that Windows 8 is still six months to a year away, but you can develop NUIs for Windows 7 multitouch, Surface, and Kinect today with WPF. You can even give them a metro look-and-feel.
If this interests you and you haven't already, take a look at the free sample chapter of my book, and if you like it pre-order it and get the early access PDF today with all of the completed chapters.
Microsoft had previously announced its plans to make the Kinect for Windows SDK available this Spring, initially for non-commercial use and later for commercial use. At the MIX11 conference this past week during the day 2 keynote (starting at the 1:40:10 mark of the video) they reiterated this plan and gave a few more details, including support for C++, C#, and VB development. They did not mention it explicitly but they used WPF during the live coding demonstration.
On day 3, several Kinect and Microsoft Research (MSR) presented a panel Q&A session. At 26:05 of the Kinect panel video, I introduced myself as the founder of the OpenKinect community and asked whether they had considered embracing the open source Kinect movement started by this community, which has lead to hundreds of Kinect Hacks around the world, by open sourcing their code.
To my surprise, Dr. Kristin Tolle, Director of MSR’s Natural User Interface team, responded that they will be publishing the Kinect SDK as open source! She referenced earlier discussion about embracing and enabling the community to take what Microsoft has created and using it in new ways and mentioned that MSR has a history of releasing open source SDKs.
When I returned to my seat, another MSR researcher approached me and suggested that Kristin may have misunderstood my question or had misspoken; however, I followed up with Kristin afterwards and reconfirmed that she meant open source as in the public can modify and recompile the sources as well as share the modifications. She also emphasized that they want to enable people to share modifications and enhancements.
This is great news!
It is not clear yet what type of license they will use. If the license allows modification and redistribution but is, at least initially, limited to non-commercial use, then it would not be a standard open source license according to the Open Source Definition. Looking at other projects published by MSR, I could see them choosing the MSR-LA as used by the Singularity experimental OS.
I’ll restate here that Microsoft has on multiple occasions confirmed that a commercial license will be available at a date after the initial non-commercial release. The ideal scenario would be if they switched from MSR-LA to a true open source license and set up a collaborative project between MSR, Kinect team, and the open community styled after the successful Nuget project, which uses the Apache 2 license and includes Microsoft and non-Microsoft contributors.
Considering the reaction by the other MSR researcher to this open source statement, there may be some confusion internally about whether this will happen like this. I think the best thing that we as a community can do is to express support for Microsoft open sourcing the Kinect for Windows SDK (including the skeleton tracking and training components), even if they use a non-commercial license initially, and keep the pressure on them to create a better relationship with the open community and transition to a Nuget style collaboration with a liberal license such as Apache 2.
Update 4/18 7:34pm: Mary Jo Foley picks up this story, but the Microsoft spokesperson she talked to denied that the Kinect SDK will be open source. As she notes, Microsoft has pulled 180’s before regarding Kinect. After spokespeople initially were hostile to the idea of Kinect hacking, Xbox executives later embraced the idea that people are using Kinect for non-gaming purposes on the PC. Let’s hope Microsoft stays open to this idea.
It's only a week away from MIX11 and I'm very excited. NUI will be represented even more than last year. Here’s a run down of all the NUI events I’m aware of:
Monday, April 11 6-9pm Ballroom D
Open Source Fest A showcase of 52 open source projects, including InfoStrat.MotionFx, our project for integrating Kinect with WPF 4 Touch.
(This is the open call session I submitted and voted in by the community. Thanks!)
Tuesday, April 12 7-9pm Lobby in front of The Commons
NUI MIXup at MIX11 NUI MIXup is back! RSVP at the link above. Last year’s NUI MIXup was a great success. Frank La Vigne, a Microsoft Public Sector Developer Evangelist, will be sponsoring drinks and appetizers for the group. I hope to see you there!
UX Lightning Sessions Three UX Lightning sessions, each with four speakers. Not purely NUI, but probably still of interest to most NUI enthusiasts.
April 12-14 Shorelines A
The Connect Lounge Microsoft Research and Coding4Fun will be showing off some cool NUI and Kinect projects.
I can’t say where yet, but InfoStrat will also have a presence with some of our demos. I’m also hearing rumors of some things that might be announced during the keynotes, but we’ll just have to wait and see.
If you’re looking for me at MIX11, the best way is to check my twitter stream @joshblake or send me a mention. You can also email me joshblake atgmail com.
MIX is a great conference that focuses on development, design, and upcoming technology trends. Last year, MIX started an “Open Call for Content” where the general public was invited to submit sessions and the general public voted to decide which sessions would be given actual spots during the conference. The Open Call sessions can be about any topic the community desires to hear and they augment the Microsoft sessions that make up the meat of the conference.
This year Open Call is back and there are 228% increase in proposals! Out of 384 entries, they posted 207 sessions. I read through every single one yesterday and expected to see some great sessions and some mediocre sessions, but to my surprise almost every single session sounds fantastic. MIX11 is going to have some great content.
Last year, I was fortunate enough to be one of 12 MIX10 Open Call sessions. I presented “Developing Natural User Interfaces with Microsoft Silverlight and WPF 4 Touch [and Windows Phone 7]”. (Read my MIX10 recap.) My session was one of three that focused on NUI, multi-touch, or Surface topics.
This year there are 8 9 entries about Surface or NUI and I’m hoping several will be picked. I’m organizing a panel session featuring the Microsoft Surface MVPs, some interesting discussion, and some cool demos. You can vote for up to 10 sessions, so please consider including the Surface MVP panel and any of the other NUI sessions that sound interesting to you in your voting.
The Natural User Interface (NUI) is a hot topic that generates a lot of excitement, but there are only a handful of companies doing real innovation with NUIs and most of the practical experience in the NUI style of design and development is limited to a small number of experts. The Microsoft Surface MVPs are a subset of these experts that have extensive real-world experience with Microsoft Surface and other NUI devices.This session is a panel featuring the Microsoft Surface MVPs and an unfiltered discussion with each other and the audience about the state of the art in NUI design and development. We will share our experiences and ideas, discuss what we think NUI will look like in the near future, and back up our statements with cutting-edge demonstrations prepared by the panelists involving combinations of Microsoft Surface 2.0, Kinect, and Windows Phone 7.
How do you design a Microsoft Surface application that creates a magical experience in less than thirty seconds, uses real-world physical objects, and is both fun and educational for kids? That is quite a task, but oh yeah – you need to build seven of these magical applications that are distinct yet connected by a common theme. This was the challenge presented to InfoStrat, a Surface Strategic Partner in Washington DC, by the Smithsonian Institution. This session will explore the design and development process that went into building the innovative physical object interactions for a suite of Microsoft Surface applications used in the “Wonder of Light: Touch and Learn!” exhibit at the Smithsonian Institution in Washington, DC. Josh Wall, director of the InfoStrat Advanced Technology Group, will demonstrate and analyze five different types of physical object interactions that take advantage of the unique vision system in Microsoft Surface. He will also discuss lessons learned from the experience, including designing Surface applications for kids, when to use a linear or non-linear task path, and how to apply Natural User Interface (NUI) concepts to interactive museum exhibits.
There are also many other interesting topics, but I hope you’ll help vote in the NUI and Surface sessions. The MIX content team takes into account the popularity of various topics. (For example, last year they accepted 12 Open Call sessions when they were originally planning only 10.)
Voting is open until midnight PST on February 4th, so lock in your votes now!
P.S. Here is a really nifty pivot viewer for visualizing and filtering the Open Call entries: http://www.xpert360.net/MIX11/
I make epic multitouch NUIs on Surface and Win7 with WPF. OpenKinect Community founder. Author of Natural User Interfaces in .NET. Microsoft Surface MVP. Technical Director of the InfoStrat Advanced Technology Group.
I'm writing Natural User Interfaces in .NET: WPF 4, Surface 2, and Kinect, due Spring 2011. Pre-order the MEAP+eBook or MEAP+printed book now to read chapters as I write them. Order through this link to throw a few more dollars my way. Thanks!
To my surprise, 
Posts
