Wednesday, July 14, 2010

How to write WPF 4 Surface applications with Blake.NUI

This post is part of a series about Blake.NUI, my open-source library for multi-touch natural user interface development with WPF 4 and Surface. In this post I discuss the origin of SurfaceTouchDevice and MouseTouchDevice and how to use them in your application. See my Blake.NUI announcement post for details as well as a discount code for my book Multitouch on Windows, good until July 23.

Between InfoStrat projects and writing my book, I've been neck-deep developing WPF 4 Touch and Surface applications for months. If you have used them both then you'll know that the WPF 4 Touch API is a little bit easier to develop for than the Surface SDK it is based upon. This is because WPF 4 touch and manipulation features are integrated directly into the framework. There are also a few other subtle differences, like not needing a separate ManipulationProcessor and InertiaProcessor.

On my current project, I needed to create a Surface and a Win7 version of the same application. I decided that I wanted to use .NET 4 on Surface. This would let me take advantage of the new animation easing functions in .NET 4 and it would let me reuse more code between both versions of the application. First problem: can I install Visual Studio 2010 and .NET 4 on Surface? I tweeted @Surface and Eric Havir replied. Figure 1 shows this twitter conversation.

joshblake: @MSSurface Can .NET 4 be installed on a #Surface device? Just curious. / surface: @joshblake I can check. My guess is it can, but the SDK only works with 3.5, you wouldn't be using it directly with Surface apps.
Figure 1. A twitter conversation between me and Eric Havir from @Surface where he confirms .NET 4 would probably install but could not be used for Surface apps.

That is pretty much what I expected. It's been known since PDC09 that the roadmap for the future of Surface includes the WPF 4 Touch API on Windows 7, as shown in figure 2. This sounds like a great unified platform, but it doesn't help me now. The currently available Surface SDK was designed for .NET 3.5 SP1 and uses a custom Surface Input mechanism that doesn't feed into the WPF 4 touch system.



I went ahead and installed Visual Studio 2010 on my Surface anyway because I wanted to use the easing functions. It turns out that Surface applications compile just fine under .NET 4 and even the SDK works, both in the emulator and on a Surface device. I figured I would just create some common controls and branch most of my project to target either the WPF 4 Touch with Surface Toolkit or the Surface SDK.

But wait! WPF 4 uses TouchDevices. TouchDevices are extensible. Anson Tsao, a Principal Program Manager who was responsible for the WPF 4 Touch API, wrote about creating custom touch devices in January.

So I'm thinking maybe if I create a custom touch device that listens to the Surface Contact events, I could process those events into WPF 4 TouchDevices and not even write against the Surface SDK.

Well once I had the idea in my head, I had to do it. I created a class, SurfaceTouchDevice, that derives from TouchDevice and promotes Surface SDK Contact events to WPF 4 TouchDevice events. This means that if you install .NET 4 on your Surface and write against the WPF 4 Touch API, It Just Works™.

Here's the best part. All you need to do to enable this dark magic is add this line of code to the constructor of your SurfaceWindow:

SurfaceTouchDevice.RegisterEvents(this);

That's it. Now the same code that works with WPF 4 Touch will also work on Surface. This also means you can develop with Visual Studio 2010. The SurfaceTouchDevice class will not work on a Win7 machine, though, since the Contacts class is not present in the Surface Toolkit.

In most cases you won't need to do anything else. Surface SDK controls, such as TagVisualizer, will react to and handle Contact events. Unhandled events will be promoted to WPF 4 Touch events by the SurfaceTouchDevice class, and then your code will react to the WPF 4 Touch events.

If you do want to check an actual Surface property, such as whether the TouchDevice represents a tag or a blob, you can get access to the original Contact by casting the TouchDevice to a SurfaceTouchDevice:

   1:  private void TouchDown(object sender, TouchEventArgs e)
   2:  {
   3:      bool isFinger = true;
   4:   
   5:      SurfaceTouchDevice device = e.TouchDevice as SurfaceTouchDevice;
   6:      if (device != null)
   7:      {
   8:          isFinger = device.Contact.IsFingerRecognized;
   9:      }
  10:      //Do something 
  11:  }

You can also filter out touches from your manipulations by iterating through e.Manipulations, casting each IManipulator to SurfaceTouchDevice, and calling Manipulation.RemoveManipulator for undesired IManipulators. I will probably add this feature to SurfaceTouchDevice in an update to Blake.NUI.

I also created a MouseTouchDevice class that operates similarly except promotes mouse input to touch input. This is helpful so I do not need to create MouseDown/Move/Up events when I'm creating a custom touch control with TouchDown/Move/Up.  It also means the mouse can participate in the manipulation processor.

Both SurfaceTouchDevice and MouseTouchDevice are available, free and open-source, in Blake.NUI, my library of helpful multi-touch NUI code for WPF 4 and Surface.

Now that SurfaceTouchDevice is available, I cannot think of a reason why you would not want to install Visual Studio 2010 on your Surface and write all applications with WPF 4. In my next post, I'm going to talk about how to set up your application architecture so you can share 99% of your code with no changes or branches for Microsoft Surface and Win7.

5 comments:

  1. Hi Josh,

    I just wanted to get your opinion on this situation. A client would like to me to add a new piece of distinct functionality to an already sizeable surface application. Modifications to the existing parts of the application are out of scope.

    I would like to target .NET4 with for the new functionality (there is a plan to port it to a standalone Windows 7 Touch application at a later date).

    Would you recommend using your SurfaceTouchDevice for the new functionality in this situation?

    Just bought the book, cheers!

    Alex.

    ReplyDelete
  2. Alex,

    Thanks for buying!

    As to your question:
    Whether you should do a hybrid depends upon the nature of the interaction between the old and new pieces. SurfaceTouchDevice has not been testing in all situations so there is a possibility that it doesn't react well with a particular combination of controls and events.

    Since you have the source, you can change it if needed.

    If you are in a time crunch, you'd probably be better off just using the regular Surface SDK since you'll know it works. If you have enough time, you could convert and test the project under .NET 4, but maintain the capability to revert if necessary.

    ReplyDelete
  3. I have experience using .NET 4 as well so I am not too afraid of moving away from the Surface SDK.

    In terms of the interaction between the old and new parts, there will be basically none. The new part will effectively be a new 'screen'. I guess my main question is, if I use SurfaceTouchDevice on this standalone screen will it have any effect on the rest of the app that uses the Surface SDK?

    (I guess at a minimum the Application project itself will need to target .NET 4).

    Alex.

    ReplyDelete
  4. SurfaceTouchDevice should not affect the rest of the app. The stand-alone screen is probably the least risky situation for trying that out.

    Let me know how it works out!

    ReplyDelete
  5. Hello from where i can buy the surface 2.0 hardware ,
    what are the supporting touch computers.
    i need it on urgent basis as my client needs demo on the device

    ReplyDelete