X Gesture Extension protocol - draft proposal v1

Chase Douglas chase.douglas at canonical.com
Mon Aug 16 08:13:20 PDT 2010


                            The X Gesture Extension
                                  Version 1.0

                                 Chase Douglas
                          chase.douglas at canonical.com
                                Canonical, Ltd.

********************************************************************************
********************************************************************************
**************************                           ***************************
**************************    DRAFT PROPOSAL (v1)    ***************************
**************************                           ***************************
********************************************************************************
********************************************************************************


1. Introduction

The X Gesture Extension is a mechanism to provide the following:
- Interface for X clients to register and receive gesture primitive events
- Interface for an X client to act as a gesture engine

Gestures may be seen as logical groupings of multitouch input events. Thus,
this extension is dependent on the X Input Extension version 2.1, which
implements multitouch input support.

                              ❧❧❧❧❧❧❧❧❧❧❧

2. Notations used in this document

Notation for requests:
┌───
    Name of request
        name of request field:       type of request field
        name of request field:       type of request field
        ▶
        name of reply field:         type of reply field
└───

Notation for events:
┌───
    Name of event
        name of field:               type of field
        name of field:               type of field
└───

Complex fields are specified in the following notation:
          name of field:                  COMPLEXFIELDTYPE
or, if multiple of these fields exist:
          name of field:                  LISTofCOMPLEXFIELDTYPE

COMPLEXFIELDTYPE:  { name of subfield:   type of subfield,
                     name of subfield:   type of subfield }

                              ❧❧❧❧❧❧❧❧❧❧❧
3. Data types

DEVICE { DEVICEID, AllDevices }
        A DEVICE specifies either an X Input DEVICEID or AllDevices.

DEVICEID { CARD16 }
        A DEVICEID is a numerical ID for an X input device currently available
        in the server. The server may re-use a device ID after a device's
        removal. The device IDs 0 and 1 are reserved.
        AllDevices ........ 0

GESTUREID { CARD16 }
        A GESTUREID is a numerical ID for an X Gesture type currently available
        in the server. The server may re-use a gesture ID if available gesture
        types change.

GESTUREFLAG { MUTEX }
        A flag telling the server to not propagate gestures to child clients
        when a gesture type in the associated mask is set. The gesture will only
        be sent to the registering client.
        When registering for gestures, a client using this flag may not listen
        for gesture IDs that any other client has registered for with the MUTEX
        flag.

GESTUREMASK
        A GESTUREMASK is a binary mask defined as (1 << gesture ID). A
        SETofGESTUREMASK is a binary OR of zero or more GESTUREMASK.

GESTUREPROP { property:             ATOM
              property_type:        ATOM }
        A GESTUREPROP is the definition of a single property of a gesture. The
        property field specifies a label for the gesture. The property_type
        field specifies the data type of the property. For example, the property
        type may be the atom representing "FLOAT" for an IEEE 754 32-bit
        representation of a floating point number. Where applicable, both the
        property and the type should conform to the standard gesture
        definitions.

EVENTRANGE { start:                 CARD16
             end:                   CARD16 }
        An EVENTRANGE specifies a range of event sequence numbers, inclusively.

                              ❧❧❧❧❧❧❧❧❧❧❧
3. Gesture Primitives and Events

The XInput protocol through version 2.0 supports single touch input devices.
Beginning with version 2.1, XInput will support multitouch input devices. The
protocol extension provides a mechansim for selection and propagation of
multitouch events to windows they occur within. The selection and propagation
is performed per touch; if a user touches a parent window and a child window
with separate touches simultaneously, and both windows have input events
selected for, the touch events will be sent to their respective windows. This
functionality is useful for general multitouch events, but is ill-suited for
user interface paradigms that require groupings of touches to be interpreted as
a single unit.

The X Gesture extension aims to provide for selection and propagation of gesture
primitives. Gesture primitives can be best thought of as fundamental multitouch
input groupings that convey specific meaning. Gesture primitives comprise events
such as pans, pinches, and rotatation. Primitives are also defined by the number
of touches that comprise them: two finger pans and three finger pans are separate
primitives.

3.1 Gesture Event Selection and Propagation

The mechanism for gesture selection and propagation follows the typical X window
event model. An X client requests for gesture events for primitives it selects
for on a given window. When a gesture engine recognizes a primitive that occurs
within a window with a client that selected to receive events for the primitive,
a gesture event is sent to the client.

Proper selection and propagation of new gesture events depends on a few
variables: where each touch event occurs, which clients requested for a
recognized primitive, and if any clients hold primitive mutual exclusion for the
window. There are two gesture primitive selections a client can make: initiation
and continuation selections.

3.1.1 Gesture Life

The lifetime of a gesture is comprised of its beginning, continuation, and
ending. During this time, a gesture may change primitive forms. For example, a
user may initiate a gesture with a three finger pan, but then continue the same
gesture by lifting one of the fingers. When a client makes a gesture event
selection request, the initiation selection is used to determine which clients
may begin receiving a newly recognized gesture. The continuation selection is
used to determine whether a gesture continues or ends when the primitive form
changes. Thus, a gesture may continue even if the number of fingers or the type
of gesture primitve changes.

3.1.2 Gesture Mutual Exclusion

Clients can request to receive gesture events on any windows, including windows
it does not own. This level of control provides for many use cases, but it
limits the ability of gestures to be controlled in an environmental manner. For
example, a window manager may wish to define gestures that should always be
available to a user, regardless of what other clients select for.

The X Gesture extension provides a mechanism for mutual exclusion of gestures.
When a client registers for a selection of gestures and uses the MUTEX flag in
the request, it may be granted mutual exclusion for the selected gestures on the
window and all its children. At any given moment, only one client may select for
mutual exclusion on each window. Any gesture occurring within the window and its
children that matches a mutually exclusive client gesture initiation selection
will only be sent to the mutually exclusive client. Further, mutually exclusive
gesture initiation selections of parent windows are given priority over their
child windows. Note that mutually exclusive gesture propagation is in opposite
priority ordering from non-mutually exclusive gesture propagation.

3.1.3 Gesture Event Initiation

Each touch event location occurs within a hierarchy of windows from the child
window, the top-most window the touch occurred in, and the root window of the
screen in which the touch event occurred. The common ancestry of all touch
events is used for propagation.

The common ancestry is traversed from child windows to parent windows to find
the first window with a client selecting for initiation of the gesture primitive
comprising the touches. The first window meeting this criteria is the normal
event window.

Propagation continues through the ancestry tree to find the eldest window with a
client holding a mutual exclusion flag for the gesture primitive. If such a
window and client are found, the event is sent to the client exclusively.
Otherwise, the event is sent to all clients of the normal event window that have
requested for the gesture primitive.

3.1.4 Gesture Event Continuation

After clients are selected to receive events, any future gesture events for the
same gesture primitive are sent to them exclusively, until the gesture primitive
changes or the all touches end.

When a gesture primitive during the lifetime of a gesture changes, the set of
all clients selected for the gesture is checked for their continuation
selection. If the new gesture primitive type is within the continuation
selection for a client, it will continue to receive gesture events with the new
primitive. Otherwise, the gesture will end for the client, and the client will
be removed from the set of clients selected for receiving gesture events. Once
all clients have ended their gesture events, new gesture events can occur
through the gesture event initiation process.

                              ❧❧❧❧❧❧❧❧❧❧❧
4. Gesture Primitive Events

Gesture primitive events provide a complete picture of the gesture and the
touches that comprise it. Each gesture provides typical X data such as the
window ID of the window the gesture occurred within, and the time when the event
occurred. The gesture specific data includes information such as the focus point
of the event, the location and IDs of each touch comprising the gesture, the
properties of the gesture, and the status of the gesture.

The focus point is a point computed by the gesture recognition engine to provide
context about where an event occurred. For example, the focus point of a
rotation primitive is the pivot point of the rotation. The location and IDs of
each touch point can be used to map each touch to a point on screen. The
properties of a gesture event are dependent on the gesture primitive type. For
example, a pan gesture has properties such as the change in the X and Y
coordinates. The status of the gesture defines the state of the gesture through
its lifetime.

                              ❧❧❧❧❧❧❧❧❧❧❧
5. Gesture Engine

A gesture engine is a client of X that recognizes gesture primitives. Only one
engine may be registered at any moment. When the gesture engine registers, it
provides the server with the set of gesture primitives it recognizes. A
Primitive is defined by its name and properties. The name and properties are
defined by strings, represented as X atoms in the protocol, and a property has a
type that is also defined by a string in the same manner. For example, the type
of a property may be "FLOAT" to designate an IEEE 754 floating point binary
format of the property data.

When a gesture engine registers or unregisters, the primitives it can recognize
become available for other clients to select. Clients will be sent gesture
availability update events if they have registered for them or if they are
currently selecting for any gesture primitives. In the latter case, gesture
primitive selections will be unregistered. An exception is made for mutual
exclusion clients: their event selections are reset to an empty set, but not
unregistered.

It is necessary for a gesture engine to recognize mutually exclusive gesture
primitives for each set of primitives defined for a given number of touches.
This is a limitation required for proper selection and propagation of gestures.
For example, two-finger pan, pinch, and rotate are mutually exclusive. A pan
cannot also be a pinch nor a rotate. However, a gesture engine should not
provide for pan, pinch, rotate, and a separate gesture primitive that is a
combination of some or all of the other primitives.

5.1 Gesture Engine Operation

Once a gesture engine is registered, it will begin receiving a stream of events
from the X server. The events are buffered inside the server until a request by
the engine is received with instructions for how to handle the events.

When the engine recognizes a gesture primitive, it sends a gesture event to the
server with the set of input event sequence numbers that comprise the gesture
primitive. The server then selects and propagates the gesture event to clients.
If clients were selected for propagation, the input events comprising the
gesture primitive are discarded. Otherwise, the input events are released to
propagate through to clients as normal XInput events.

When the engine does not recognize any gesture primitive for a set of input
events, it sends a request to the server to release the input events to
propagate through to clients as normal XInput events.

The server may set a timeout for receiving requests from the gesture engine. If
no request from the engine is received within the timeout period, the server may
release input events to propagate through to clients as normal XInput events.
The timeout value is implementation specific.

                              ❧❧❧❧❧❧❧❧❧❧❧
6. Errors

Errors are sent using core X error reports.

GestureEngineRegistered
        A client attempts to register a gesture engine while an engine is
        already registered.

                              ❧❧❧❧❧❧❧❧❧❧❧
7. Requests:

The server does not guarantee that the length of a reply remains constant in
future revisions of X Gesture. A client must always retrieve the exact length of
the protocol reply from the connection, even if the reply is longer than defined
for the X Gesture version supported by the client.
Additional bytes in a request may include data supported in later versions of
X Gesture. Clients should ignore this data. Padding bytes in X Gesture protocol
requests are required to be 0.

7.1 General Client Requests

    ┌───
        GestureQueryVersion
        major_version:          CARD16
        minor_version:          CARD16
        ▶
        major_version:          CARD16
        minor_version:          CARD16
    └───

    The client sends the highest supported version to the server and the
    server sends the highest version it supports, but no higher than the
    requested version. Major versions changes can introduce incompatibilities
    in existing functionality, minor version changes introduce only backward
    compatible changes.  It is the client's responsibility to ensure that the
    server supports a version which is compatible with its expectations.

    major_version
        Major X Gesture version.
    minor_version
        Minor X Gesture version.

    If major_version is less than 1, a BadValue error occurs.

    ┌───
        GestureQueryAvailableGestures
        ▶
        num_gestures:           CARD16
        gestures:               LISTofGESTUREMAP
    └───

    GESTUREMAP  { gesture_id:           GESTUREID
                  gesture_type:         ATOM }

    GestureQueryAvailableGestures details information about the gesture types
    that are recognizeable by the gesture recognizer registered with the server.

    Each gesture is detailed as follows:
    gesture_id
        The unique ID of the gesture. Gesture IDs may be re-used when the
        available gestures to be recognized changes.
    gesture_type
        An ATOM specifying the type of gesture. The gesture type should conform
        to the list of standard gesture types if applicable.

    ┌───
        GestureListenForGestureChanges
        listen:                 CARD8
    └───

    GestureListenForGestureChanges registers the client to be sent a
    GesturesChanged event.

    listen
        Inequal to 0 if the client wants to receive GestureChanged events

    On receipt of a GestureChanged event, the client may send the
    GestureQueryAvailableGestures request to receive the new list of available
    gestures.
    Note that all clients listening for any gesture events on any window will
    receive a GestureChanged event regardless of whether they have called this
    request with any value for listen. However, the server will honor the last
    listen value sent in this request whenever the client is not listening for
    gesture events.

    ┌───
        GestureQueryGestureProperties
        gesture_type:           ATOM
        ▶
        num_properties:         CARD16
        properties:             ListofGESTUREPROP
    └───

    GestureQueryGestureProperties details properties of the requested gesture
    type.

    ┌───
        GestureSelectEvents
            window:         Window
            device_id:      CARD16
            flags:          SETofGESTUREFLAG
            mask_len:       CARD16
            init_mask:      GESTUREMASK
            cont_mask:      GESTUREMASK
    └───

    window
        The window to select the events on.
    device_id
        Numerical device ID, or AllDevices.
    flags
        Flags that may affect gesture recognition, selection, or propagation.
    mask_len
        Length of mask in 4 byte units.
    init_mask
        Gesture mask for initiation. A gesture mask for an event type T is
        defined as (1 << T).
    cont_mask
        Gesture mask for continuation. A gesture mask for an event type T is
        defined as (1 << T).

    GestureSelectEvents selects for gesture events on window.

    The mask sets the (and overwrites a previous) gesture event mask for the
    DEVICE specified through device_id. The device AllDevices is treated as a
    separate device by server. A client's gesture mask is the union of
    AllDevices and the per-device gesture mask.
    The removal of a device from the server unsets the gesture masks for the
    device. If a gesture mask is set for AllDevices, the gesture mask is not
    cleared on device removal and affects all future devices.

    If mask_len is 0, the gesture mask for the given device is cleared. However,
    a client requesting for mutual exclusion may register for any valid mask_len
    length of mask with all bits set to 0. This allows a mutual exclusion client
    to prohibit any other client from gaining exclusive privilege.

    ┌───
        GestureGetSelectedEvents
            window:         Window
            ▶
            num_masks:      CARD8
            masks:          LISTofEVENTMASK
    └───

    EVENTMASK { device_id:         DEVICE,
                flags:             SETofGESTUREFLAG,
                mask_len:          CARD8,
                init_mask:         GESTUREMASK
                cont_mask:         GESTUREMASK }

    window
        The window to get selected events for.
    num_masks
        Number of items in masks.
    masks
        Selected event masks by the client.
    device_id
        Input device for the mask.
    flags
        Flags set for the mask.
    length
        Length of mask in 4 byte units.
    init_mask
        Gesture mask for initiation. A gesture mask for an event type T is
        defined as (1 << T).
    cont_mask
        Gesture mask for continuation. A gesture mask for an event type T is
        defined as (1 << T).
    
    GestureGetSelectedEvents requests a list of all gesture events selected by
    the client on the given window.

    Masks are returned on a per-device basis, with the mask for AllDevices
    returned separately. A client can calculate the effective mask for a device
    with a bitwise OR of the AllDevices and the device-specific mask.

    If num_masks is 0, no events have been selected by this client on the
    given window.

    ┌───
        GestureGetAllSelectedEvents
            window:         Window
            ▶
            num_masks:      CARD8
            masks:          LISTofCLIENTEVENTMASK
    └───

    CLIENTEVENTMASK { client_id:   CLIENT,
                      device_id:   DEVICE,
                      mask_len:    CARD8,
                      init_mask:   GESTUREMASK
                      cont_mask:   GESTUREMASK }

    window
        The window to get selected events for.
    num_masks
        Number of items in masks.
    masks
        Selected event masks by all clients.
    client_id
        Client ID of client selecting for the mask.
    device_id
        Input device for the mask.
    flags
        Flags set for the mask.
    length
        Length of mask in 4 byte units.
    init_mask
        Gesture mask for initiation. A gesture mask for an event type T is
        defined as (1 << T).
    cont_mask
        Gesture mask for continuation. A gesture mask for an event type T is
        defined as (1 << T).

    GestureGetAllSelectedEvents retrieves the gesture selections for all clients
    on the given window.

7.2 Gesture engine client requests

    ┌───
        GestureEngineRegister
            num_gestures:   CARD16
            gestures:       LISTofGESTUREINFO
    └───

    GESTUREINFO { gesture_type:      ATOM,
                  num_properties:    CARD16,
                  properties:        LISTofGESTUREPROP }

    GestureEngineRegister is the mechanism by which a gesture engine registers
    with the X Gesture extension to be able to process gesture events. Only one
    gesture engine may be registered to the server at any given time. Further
    registration requests will cause a GestureEngineRegistered error.
    When the gesture engine is registered successfully, a GesturesChanged event
    is sent to all clients registered to listen for the event. The clients may
    then request the new list of available gestures from the server.

    ┌───
        GestureEngineUnregister
    └───

    GestureEngineUnregister unregisters the gesture engine from the server. If
    the client has not registered a gesture engine successfully through the
    GestureEngineRegister request, a BadValue error will result. Otherwise, a
    GesturesChanged event will be sent to all clients registered to listen for
    the event.

    ┌───
        GestureAllowInputEvents
            num_ranges:     CARD16
            ranges:         LISTofEVENTRANGE
    └───

    GestureAllowInputEvents instructs the server to flush the input events to
    clients unmodified. This is used when no gestures are recognized from
    sequences of input events.
    If any of the EVENTRANGE values are invalid, the BadValue error is reported
    and no input events are flushed.

    ┌───
        GestureRecognized
            num_ranges:                 CARD16
            ranges:                     LISTofEVENTRANGE
            gesture_id:                 CARD16
            gesture_instance:           CARD16
            device_id:                  CARD16
            root_x:                     Float
            root_y:                     Float
            event_x:                    Float
            event_y:                    Float
            status:                     STATUS
            num_touches:                CARD8
            num_properties:             CARD8
            touches:                    TOUCHES
            properties:                 LISTofCARD32
    └───

    See the GestureNotify event definition below for gesture data definitions.

    GestureRecognized instructs the server to perform propagation and selection
    for a potential GestureNotify event to be sent to clients. If clients are
    selected to receive events, the input events specified in the event ranges
    must be discarded by the server. Otherwise, the input events must be
    propagated to clients as defined by the XInput protocol.

8. Events:

An event specifies its length in 4-byte units after the initial 32 bytes.
Future versions of the protocol may provide additional information
in the same event, thus increasing the event size. Clients are required to
always read the number of bytes specified by the event, not the size of the
event they may have been compiled against.


The following event types are available in X Gesture.

        GesturesChanged
        GestureNotify

All events have a set of common fields specified as EVENTHEADER.

EVENTHEADER { type:                       BYTE
              extension:                  BYTE
              sequenceNumber:             CARD16
              length:                     CARD32
              evtype:                     CARD16 }

    type
        Always GenericEvent.
    extension
        Always the X Gesture extension offset.
    sequenceNumber
        Sequence number of last request processed by the server.
    length
        Length in 4-byte units after the initial 32 bytes.
    evtype
        X Gesture-specific event type.


    ┌───
        GesturesChanged:
            EVENTHEADER
    └───

    A GesturesChanged event is sent whenever the gestures available from the
    server have changed. A client receiving this event may choose to request the
    new list of available gestures from the server.

    ┌───
        GestureNotify:
            EVENTHEADER
            gesture_id:                 CARD16
            gesture_instance:           CARD16
            device_id:                  CARD16
            time:                       Time
            root:                       Window
            event:                      Window
            child:                      Window
            root_x:                     Float
            root_y:                     Float
            event_x:                    Float
            event_y:                    Float
            status:                     STATUS
            num_touches:                CARD8
            num_properties:             CARD8
            touches:                    TOUCHES
            properties:                 LISTofCARD32
    └───

    STATUS { Begin, Update, Continue }

    TOUCHES { x:                          FLOAT
              y:                          FLOAT
              valuators_len               CARD16
              valuators                   LISTofFP3232 }

    x
    y
        Location of touch in screen coordinates.
    valuators_len
        Number of values in axisvalues.
    axisvalues
        Values of all valuators for the touch. Valuators are defined in the
        XInput protocol specification. The specific meaning of each valuator is
        specific to the input device.

    A GestureNotify event is generated whenever a gesture occurs in a window for
    which the client has requested the gesture ID.

    gesture_id
        Gesture ID of the gesture type.
    gesture_instance
        Unique ID of this gesture instance. The ID will be maintained as the
        gesture progresses from start to end as signified in the status field.
        This value is monotonically increased by the server for every gesture
        causing events to be sent to a client. The ID will only be reused once
        a 32-bit wrap occurs.
    device_id
        X Input device ID of the slave device generating this gesture.
    root
    event
    child
        The root window, event window or subwindow, respectively. See core
        protocol specification for more detail.
    root_x
    root_y
        The position of the focus point of the gesture in screen coordinates.
    event_x
    event_y
        The position of the focus point of the gesture in screen coordinates
        relative to the event window.
    status
        The status of the gesture in its life cycle.
    num_touches
        The number of touches comprising up the gesture.
    num_properties
        The number of properties for the gesture type.
    touches
        Touch data for touches comprising the gesture.
    properties
        The properties of the gesture.

                              ❧❧❧❧❧❧❧❧❧❧❧

A. Acknowledgements:

This protocol has been developed through the ideas of everyone on the Canonical
Multitouch team and the X community. In particular, I would like to thank:

* Peter Hutterer (XInput maintainer)
* Carlos Garnacho (Gtk+ input developer)
* Henrik Rydberg (Kernel multitouch and uTouch-Grail maintainer)
* Rafi Ruben (N-Trig Linux driver maintainer)
* Stephen Webb (uTouch-GEIS library maintainer)
* Neil Patel (Unity maintainer)
* David Siegel (Canonical UI designer)
* Ikbel Boulabiar (Canonical multitouch team)
* Christian Giordano (Canonical UI designer)
* John Lea (Canonical UI designer)
* Cody Russell (Canonical Gtk+ developer)



More information about the xorg-devel mailing list