For this article, I wanted to demonstrate how to access gestures and flicks using C# code. I'll talk about accessing flicks using the tablet API and accessing gestures using the N-Trig native libraries. Surprisingly I couldn't find much out there that goes into any kind of depth in coding for either of these topics. So it seemed like a good opportunity to present some examples. I've included code for a C# game that uses the gestures for control. I hope you find it fun and useful.

Even though they are called "flicks", a flick is really a single touch gesture and an N-Trig gesture is a double touch gesture.  All the examples run on Windows Vista Ultimate 32bit. Windows Vista will only detect up to two fingers on the touch surface. However, Windows 7 promises to allow more than two finger detection. I'll have to try that next. HP is already shipping hardware that is multi-touch enabled on their TouchSmart PCs.  The Dell Latitude XT and XT2 tablet PCs are also using the N-Trig hardware for multi-touch.

Code to Detect If Running on a Tablet PC

I found some simple code to determine if your app is running on a tablet PC. It is a native call to GetSystemMetrics which is in User32.dll and you pass in SM_TABLETPC. I've included this code in the Flicks class.  You may want your app to auto-detect if it's running on a tablet to determine which features to enable.

Accessing Flicks (Single Touch Gestures) or UniGesture

If you've never heard of flicks before, they are single touch gestures that are triggered based on the speed of the touch across the tablet surface.  You can find more detail here.  It currently gives you eight directions that can be programmed at the OS level for use by applications.

I used the tabflicks.h file that I found in the folder - "C:\Program Files\Microsoft SDKs\Windows\v6.0A\Include" This header file is included with the Windows SDK. It contains all the flick structures and macro definitions needed to access flicks. I created a  class library project called FlickLib that contains the C# structures, DllImports and code to store and process the data. I also wrote a test WinForm application that shows the usage of the library.

I override the WndProc method of the WinForm to gain access to the WM_TABLET_FLICK messages sent to the window. The flick data is encoded in the WParam and the LParam of the windows message. I discovered that even though the params are IntPtr type, they are not actually pointers but are the real integer values to be used. I kept getting an Access Violation exception when I was trying to marshal the IntPtr to a structure. The solution was non-obvious. I couldn't find that documented anywhere. But it works correctly now so this is my contribution to the web for anyone trying to solve the same problem.

It is important to note that on line # 117, I set msg.Result = IntPtr.Zero.  This tells the base.WndProc method that the message has been handled so ignore it.  This is important because the underlying pen flicks can be assigned to actions such as copy and paste.  If the message is not flagged as handled, the app will attempt to perform the action.

Accessing Double Touch Gestures or DuoGestures

The N-Trig hardware on the HP TouchSmart tablet provides the ability to detect two (or more) touches on the surface. This allows the tablet to respond on two finger gestures. You can see the current DuoGestures in use from N-Trig here.

I've created a class library project called GestureLib which wraps the native libraries from N-Trig so that they can be called using C#. Obviously this will only run on tablets that have the N-Trig hardware. Currently there is the HP TouchSmart tx2 and the Dell Latitude XT & XT2 laptops. I'll be testing and updating code for the HP TouchSmart IQ505 panel that uses the NextWindow hardware. If someone has any other tablets, touch panels and/or UMPC devices, I'd be happy to work together to create a version for that as well.

Once you look at the code you will note that you have to connect to the N-Trig hardware at the start of the app.  I put that call in the OnLoad method.  Once you have a handle to the hardware, then you will use that throughout the lifetime of the app.  It is stored in the Gestures class.  With the handle, you will register which gesture messages you want the window to receive.  Once registered, you will start to see NTR_WM_GESTURE message showing up in the WndProc method.  I created a ProcessMessage method in the Gestures class that will determine which gesutre type it is and extract the data for the particular gesture.

One thing to note is if you look at the DllImports for the NtrigISV.dll native library you will see that there is name mangling with the method calls. I had to use Dependency Walker to determine the actual names of the methods. So be warned that when N-Trig releases a new version of their DLL that this code will probably break.

Problems with Marshalling Data from C++ to C#

I had an issue where I was only getting the Zoom messages.  My friend, Ben Gavin, helped me solve the problem.  It had to do with marshalling the data into the struct.  The original C++ struct looks like this:

For my original port, I just did a LayoutKind.Sequential and set everything as a bool like below.

Trouble is that bool in C++ is one byte where as a bool in C# is an Int32 or four bytes. Now if the C++ struct was labeled with a BOOL, then it would have been four bytes. So the new declaration of the struct in C# looks like this:

Once I used the FieldOffsetAttribute to set each bool to one byte each, the Rotate messages started to work correctly. Thanks Ben...

WPF Gestures Example Code

Using the same GestureLib class library, I also created an app that shows how to capture the gestures using a WPF application.  That was an interesting exercise because I had never gotten a Windows handle from a WPF app.  I discovered the WinInteropHelper class that gets the underlying Windows handle. I also had to hook the MessageProc handler to get the messages.  The WndProc method is not available to override in WPF.

Gesture Blocks - C# Game using Gestures and Flicks

Just creating test apps is boring and I wanted to create a game example that uses the flicks and gestures for control. I chose to build a simple game that shows off some of the possibilities for touch control. I call the game Gesture Blocks. It will give you a good starting point on using gestures in games. Watch the video to see the gestures in action.

Conclusion

I hope the code I provided was useful and helpful. I will try to update the examples as issues are resolved. I'm also going to work on some other game examples that continue to show what can be done with gestures. The next project will be to load Windows 7 onto the tablet and write some code for more than DuoGestures.

NextWindow's touch Application Program Interface (API) provides programmers with access to touch data generated by a NextWindow touch screen. It also provides derived touch information. For information about the capabilities of NextWindow products and which ones you can interface to using the API, see NextWindow Latest Technical Information.

The touch events, data and derived information can be used in any way the application wants.

Communications are via HID-compliant USB.

The API is in the form of a DLL that provides useful functions for application developers.

Multi-Touch

Multi-touch simply refers to a touch-sensitive device that can independently detect and optionally resolve the position of two or more touches on screen at the same time. In contrast, a traditional touch screen senses the position of a single touch and hence is not a multi-touch device.

	“Detect” refers to the ability to sense that a touch has occurred somewhere on screen.
	“Resolve” refers to the ability to report the coordinates of the touch position.

NextWindow’s standard optical touch hardware can detect two touches on screen. The firmware can resolve the position of two simultaneous touches (with some limitations).

NextWindow Multi-Touch API Downloads

NWMultiTouchAPI_v1260.zip contains the following files:

Note

NextWindow's DLL is 32-bit. If you are building an application on a 64-bit machine, you need to specify a 32-bit build. To change this setting in Visual C#, change the platform target to x86 under the Build page of Project Settings. See the following Microsoft article for more details.

http://msdn.microsoft.com/en-us/library/kb4wyys2.aspx

Example Code

NWMultiTouchCSharpSample_v1260.zip

NWMultiTouchCPlusPlusSample_v1260.zip

Documentation

NextWindow USB API User Guide v1.7 (91 KB pdf file)

NextWindow Multi-Touch Whitepaper (2722 KB pdf)

An introduction to building HP TouchSmart-hosted applications and a step-by-step tutorial for a “Hello World”

Building Your First TouchSmart Application
A developers guide to the TouchSmart Platform

Introduction

The Hewlett-Packard TouchSmart PC is an innovative new PC form-factor which affords users the unique ability to control interactions using only their hands. Through the use of a very stylish “10 foot” style user interface, the user can simply touch the integrated screen and perform most essential functions – viewing photos, listening to music, and surfing the web – all without using a keyboard or mouse.

The TouchSmart interface can be considered an entirely new user-interface platform – and now HP has opened up this platform to the entire developer community. With the release of the TouchSmart Application Developer Guidelines you can now build applications which run on this unique platform – and cater to its expanding user-base.

This article walks you through building your first TouchSmart application, with the goal of creating a basic application template which you can use for all your TouchSmart projects.

Getting Started

In order to follow the steps in this document, you will need to be familiar with Windows Presentation Foundation (WPF) and .NET development with C#. This article assumes a basic knowledge of these technologies. Please visit the Microsoft Developer Network (MSDN) at http://msdn.microsoft.com for more information.

In addition, the following is required:

• TouchSmart PC (model IQ504 or IQ506, current at the time of this writing)
• Latest TouchSmart software (available from http://support.hp.com)
• Microsoft Visual Studio 2008, Express (free) or Professional Edition
• HP TouchSmart Application Development Guidelines (available from here).
• .NET Framework 3.5 (although the same techniques can be used to build an application using Visual Studio 2005 and .NET 3.0)

It is recommended you install Visual Studio 2008 on the TouchSmart PC for ease of development, however it is not required. Since the TouchSmart application we are building is simply a Windows Presentation Foundation application it will run on any Vista or Windows XP PC with .NET 3.5.

Important Background Information

Before getting started you should know some important background information about a typical TouchSmart application:

• To best integrate with the TouchSmart look-and-feel you should develop your application using the Windows Presentation Foundation (WPF)
• The application runs as a separate process with a window that has the frame removed, it is not running within the TouchSmart shell process.
• The TouchSmart shell is responsible for launching your application. Additionally, the TouchSmart shell will control the position and sizing of your window.
• You must create three separate UI layouts for your application. This is not as difficult as it sounds; only one layout is interactive with controls – the other two layouts are for display only.
• Design your application for a “10 foot” viewing experience. Plan to increase font and control sizes if you are converting an existing application.
• Do not use pop-up windows in your application. This is perhaps the most unique requirement in building a TouchSmart application. The Windows “MessageBox” and other pop-up windows are not supported and should not be used. The best model to follow is to build your application like it was going to run in a web browser.
• Your application must support 64-bit Vista, as this is the operating system used on the TouchSmart PC.

<AppManagementSetting>
<UserEditable>true</UserEditable>
<RunAtStartup>true</RunAtStartup>
<RemoveFromTaskbar>false</RemoveFromTaskbar>   <AppPath>C:\TouchSmart\HelloTouchSmart\HelloTouchSmart\bin\Release\HelloTouchSmart.exe
</AppPath>
<AppParameters>chromeless</AppParameters>
<AllowAttach>true</AllowAttach>
<Section>Top</Section>
<Type>InfoView</Type>
<DisplayName>HelloTouchSmart</DisplayName>
<IsVisible>true</IsVisible>
<UserDeleted>false</UserDeleted>
<InDefaultSet>false</InDefaultSet>
<IconName />
</AppManagementSetting>

Matrix Result

Using the quaternion to matrix conversion here we get:

so substituting the quaternion results above into the matrix we get:

Substituting the following expansions:

(v1 x v2).x = v1.y * v2.z - v2.y * v1.z
(v1 x v2).y = v1.z * v2.x - v2.z * v1.x
(v1 x v2).z = v1.x * v2.y - v2.x * v1.y
(v1 x v2).x² = v1.y * v2.z * v1.y * v2.z + v2.y * v1.z * v2.y * v1.z - 2 * v2.y * v1.z * v1.y * v2.z
(v1 x v2).y² = v1.z * v2.x * v1.z * v2.x + v2.z * v1.x * v2.z * v1.x - 2* v2.z * v1.x * v1.z * v2.x
(v1 x v2).z² = v1.x * v2.y * v1.x * v2.y +v2.x * v1.y * v2.x * v1.y - 2 * v2.x * v1.y * v1.x * v2.y
v1•v2 = v1.x * v2.x + v1.y * v2.y + v1.z * v2.z

This is getting far too complicated ! can anyone help me simplify this?

Code

sfrotation angleBetween(sfvec3f v1,sfvec3f v2) {
float angle;
// turn vectors into unit vectors
n1 = v1.norm();
n2 = v2.norm();
angle = Math.acos( sfvec3f.dot(n1,n2) );
// if no noticable rotation is available return zero rotation
// this way we avoid Cross product artifacts
if( Math.abs(angle) < 0.0001 ) return new sfrotation( 0, 0, 1, 0 );
// in this case there are 2 lines on the same axis
if(Math.abs(angle)-Math.pi) < 0.001){
n1 = n1.Rotx( 0.5f );
// there are an infinite number of normals
// in this case. Anyone of these normals will be
// a valid rotation (180 degrees). so I rotate the curr axis by 0.5 radians this way we get one of these normals
}
sfvec3f axis = n1;
axis.cross(n2);
return new sfrotation(axis.x,axis.y,axis.z,angle);
}

/** note v1 and v2 dont have to be nomalised, thanks to minorlogic for telling me about this:
* http://www.euclideanspace.com/maths/algebra/vectors/angleBetween/minorlogic.htm
*/
sfquat angleBetween(sfvec3f v1,sfvec3f v2) {
float d = sfvec3f.dot(v1,v2);
sfvec3f axis = v1;
axis.cross(v2);
float qw = (float)Math.sqrt(v1.len_squared()*v2.len_squared()) + d;
if (qw < 0.0001) { // vectors are 180 degrees apart
return (new sfquat(0,-v1.z,v1.y,v1.x)).norm;
}
sfquat q= new sfquat(qw,axis.x,axis.y,axis.z);
return q.norm();
}

matrix version

sfmatrix angleBetween(sfvec3f v1,sfvec3f v2) {
// turn vectors into unit vectors
n1 = v1.norm();
n2 = v2.norm(); 	sfvec3f vs = new sfvec3f(n1);
vs.cross(n2); // axis multiplied by sin	sfvec3f v = new sfvec3f(vs);
v = v.norm(); // axis of rotation
float ca = dot(n1, n2) ; // cos angle	sfvec3f vt = new sfvec3f(v);	vt.scale((1.0f - ca);	sfmatrix rotM = new sfmatrix();
rotM.m11 = vt.x * v.x + ca;
rotM.m22 = vt.y * v.y + ca;
rotM.m33 = vt.z * v.z + ca;	vt.x *= v.y;
vt.z *= v.x;
vt.y *= v.z;	rotM.m12 = vt.x - vs.z;
rotM.m13 = vt.z + vs.y;
rotM.m21 = vt.x + vs.z;
rotM.m23 = vt.y - vs.x;
rotM.m31 = vt.z - vs.y;
rotM.m32 = vt.y + vs.x;
return rotM;
}

see also code from minorlogic

LINK:http://www.mpi-inf.mpg.de/~kettner/courses/lib_design_03/proj/polygon_triang.html

Design the interface between data structures that represent simple polygons and generic algorithms that triangulate simple polygons.

The Standard Template Library (STL) [Austern98, SGI-STL] contains several examples of a similar interface design: Iterators are the interface between sequences of items in container classes and generic algorithms on sequences. However, they (usually) do not modify the container. Among the container classes some modifying functions, e.g., insert or remove, describe a standardized interface for modifying the container classes and can be used for generic algorithms on container classes. This project is expected to design a generic interface in the same spirit for triangulating polygons.

We start with a set of data structures that can represent a simple polygon, and with a set of algorithms that can triangulate it. We suggest to use the Computational Geometry Algorithms Library (CGAL) <www.cgal.org> [Fabri99] as a source of data structures and algorithms in geometry and also as a platform for realizing this project. Possible data structures could be:

• A std::list of 2D points.
• The class CGAL::Polygon_2.
• An individual facet of a 3D polyhedral surface, e.g., the class CGAL::Polyhedron_3.
• While working with the triangulation data structure underlying the class CGAL::Triangulation_2 one can end up in the situation to triangulate a polygonal hole in the triangulation structure.

All these representations would be for simple polygons without holes. An optional extension would be to extend this project to polygons with holes as they can be represented with:

• A std::list of std::list's of 2D points, where the first list is the outer boundary of the polygon and all succeeding lists are the inner boundaries of holes in the polygon, one list per hole.
• The class CGAL::Planar_map_2 can contain faces with holes.
• The class CGAL::Nef_polyhedron_2 can contain faces with holes.

Examples of algorithms that triangulate polygons are:

• Ear-cutting algorithm: An ear is a triangle formed by two consecutive edges of the polygon with a convex angle that contains no other point of a polygon. There exist always at least two ears in a polygon. An ear can be cut from the polygon, reducing its size and thus triangulating it.
• A sweep-line algorithm, see [Chapter 3, deBerg00].
• The constrained triangulation in CGAL. It creates a triangulation of the convex hull of a set of points respecting a set of constrained edges that have to be present in the triangulation, which would be the polygon boundary edges here. Afterwards the triangles in the polygon interior have to be selected, i.e., distinguished from those outside of the polygon.

Clearly iterators can be used in examining the input polygon. The new part will be the modifying part of the algorithms; where do we create and store the result triangles:

• For a single polygon we might just write triangles to an output iterator, for example, for storage in a container or for rendering.
• For polygons embedded in a data structure, such as a triangulation or a polyhedral surface, one wants the result triangles to replace the original polygon and to have the proper neighborhood pointers.

The goal of the project is to design the interface and to realize some of the data structures and algorithms, possibly based on the already existing CGAL implementations. The task includes:

• Analyse the set of operations of each agorithm and how it could be implemented for each data structure considered.
• Find a common set (or several sets) of operations that support the different combinations.
• Implement the interface for the data structures, possibly with adaptors for the existing data structures in CGAL.
• Implement the algorithm based on this interface, maybe also with adaptors to the already existing algorithms in CGAL.
• Test your generic algorithms on the different representations.
• Document the design, for example, in the style of CGAL or using Doxygen.

Prerequisites

This project requires interest in geometry or graphics and some knowledge of geometric algorithms. Since CGAL will be covered later in the course, it might be necessary to learn about CGAL prior to that.

References

[Austern98]

Mathew H. Austern. Generic Programming and the STL: Using and Extending the C++ Standard Template Library. Addison-Wesley, 1998.

 

[SGI-STL]

Silicon Graphics Computer Systems, Inc. Standard Template Library Programmer's Guide. http://www.sgi.com/tech/stl/.

 

[Fabri99]

Andreas Fabri, Geert-Jan Giezeman, Lutz Kettner, Stefan Schirra, and Sven Schönherr. On the Design of CGAL, the Computational Geometry Algorithms Library. Software -- Practice and Experience, submitted 1999, to appear. (also available as technical report)

 

[deBerg00]

Mark de Berg, Marc van Kreveld, Mark Overmars, and Otfried Schwarzkopf. Computational Geometry: Algorithms and Applications. Springer, 2nd edition, 2000.