This is still a draft so any feedback is most welcome…
Project Proposal:
A High-Level Gestural Classification Framework for Developing Context-Aware MultiTouch Applications
Introduction:
With the release of the iPhone, Microsoft Surface, Jeff Han’s Multitouch Table and the inception of a vibrant community of Multitouch Developers like the NuiGroup, Multitouch is the “in-thing” among many human interface designers and developers. There are currently many individuals or groups who have taken their own initiatives to build their own hardware, basic software frameworks and multitouch applications. There has also been a lot of talk about developing a gesture framework which is very much needed in this new arena where standardization is very much needed at this point in time. Although there is currently a basic framework available for recognizing basic gestures such as the Touchlib Gesture Library and other open-source projects like the sparsh-ui, there is a lack in moving forward from the usual gestures of zooming, rotation, panning, etc. I believe that this is due not only to the lack of current viable applications available for multitouch but also due to the fact that we’re still only looking at minimal parameters to recognize gestures, for example, blob locations, sizes,etc. which has been very well documented so far, in the ever so popular TUIO protocols. But with the proposal from Martin for the TUIO 2.0 protocol, research into SecondLight and ShapeTouch from Microsoft, I believe there is an urge from the current community of multitouch developers to bring this technology to the next phase, beyond the usual gestures that we’re so used to by now.
Through our lab’s recent usability studies conducted on Multitouch Screens, we’ve discovered that gestures should be classified into 2 main categories, Intuitive and Progressive gestures. Through our observations, when users are presented with a Multitouch Screen, they have an expected “Mind Model” of how the device should work based on either what they know or their previous experiences with touch screen devices. For example, without any form of instructions, users tend to interact with the touch screen initially with only 1 finger or 5 or 10 fingers. This is classified as Intuitive. However, after being presented with a minimal set of instructions on simple multitouch based applications, users learn to use multiple fingers and bi-manual gestures. This is classified as Progressive. When the device fails to operate according to their expectations, they intuitively form a new “mind model”. In other observations, we also noticed that most users cannot remember complicated gestures that are hard-coded into many current applications. And the applications themselves are not context-aware either.
Therefore, I move to propose the development of “A High-Level Gestural Classification Framework for Developing Context-Aware MultiTouch Applications”.
Objectives:
The project will have 3 specific objectives that constitute a progressive workflow:
1) To develop a framework to collect context-related parameters for Multitouch such as finger /chord (pressure maps, force vectors, etc.) , hand (shape recognition, pose estimation, etc.) and user detection (in-direct detection using foreams, RFID or BlueTooth) that is hardware independent specific to optical-based multitouch solutions, for example, FTIR, DI, DSI, LLP and on hardware platforms such as those provided by NextWindows and PQLabs. This is because parameter availability will be dependent on the respective hardware platform’s limitations.
2) Develop a High-Level Gesture Framework based on the 2 main Classifications, Intuitive and Progressive gestures with the ability to extend learning to account for archaic and cultural anomalies, in other words, customization.
3) Develop a Context-Aware Framework for Multitouch Applications so that applications will be aware of the changes to the parameters and react accordingly.
