Xorg Xserver Plug-in Picker Design Deron Johnson deron.johnson@sun.com Version 0.2 (DRAFT) Thu Nov 30 10:38:21 PST 2006 Introduction ------------ This document describes the design of the Xorg X server Plug-in Picker. The purpose of the Plug-in Picker is to allow desktop environments to extend the X server window picking semantics. In the X server, "window picking" is the process of taking a point in screen coordinates and determining in which window the point lies. The normal X server picking process is to hierarchically search the window tree to find the lowest child window in which the point lies. There are several ways in which a desktop environment might wish to extend this behavior. For example, a 3D desktop environment such as Project Looking Glass might use the screen position to perform a pick into a 3D scene graph. The Plug-in Picker interface is an X server internal interface. The X server code which is involved in window picking is replaced with calls through an interface to a Picker module. Different pickers with different semantics can be implemented. Desktop environments can install and activate a picker that is specific to their needs. They use the Picker extension to do this. The Picker extension allows a desktop manager to select a particular Picker module to implement picking for a particular window hierarchy. This design supports different pickers for different windows and screens. For example, the windows on one screen could use normal X server picking and the windows on other screen could use Looking Glass picking. In addition to interacting with the Picker extension, a plug-in Picker module is allowed to interact with other server extensions to accomplish its task. Picker-specific extensions can be defined to allow clients to provide information to help the Picker do its job. In addition, a Picker module can provide its own internal protocols to allow clients to provide information. For example, a Picker could implement a private "back-door" socket protocol to delegate picking to an external process with out going through an X client connection. Pickers are allowed to block the X server but, of course, the amount of time the server is blocked should be minimized. Picker Interfaces ----------------- A. Interface from X Server to Picker Module This section defines the interface that a Picker provides to the rest of X server. The interface is defined as a set of macros. These macros invoke routines through a function vector. PICKER_XY_TO_WINDOW (Window Ptr pWinScr, int x, int y) Given a screen (specified by a window on that screen) and absolute screen point (x, y) this routine calculates the lowest level subwindow in which the point lies. PICKER_FIXUP_EVENT_FROM_WINDOW (WindowPtr pWinScr, xEvent *xE, WindowPtr pWin, Window child, Bool calcChild) Modifies the fields of the given event to be appropriate for the given destination window pWin. PICKER_QUERY_POINTER (WindowPtr pWinScr, INT16 inX, INT16 inY, INT16 *pOutX, INT16 *pOutY, Window *pOutChild, Bool *pOutSameScreen) Implements the XQueryPointer calculation for root window of the screen of the given window. PICKER_TRANSLATE_COORDS (Window pWin, Window pDst, INT16 inX, INT16 inY, INT16 *pOutX, INT16 *pOutY, Window *pOutChild, Bool *pOutSameScreen) Implements the XTranslateCoords calculation for root window of the screen of the given destination window. PICKER_WINDOWS_RESTRUCTURED () Used by the X server to tell the picker that the window tree has changed. B. Interface from Picker Module to X Server A Picker is allowed to reference X server external global variables. C. xEvent Reserved Fields The following fields in the xEvent.keyButtonPointer structure are reserved for use by the Picker from the time the event is first created to after PICKER_FIXUP_EVENT_FROM_WINDOW is called: event child eventX eventY The Picker, and any extensions it cooperates with, may store information in these fields. (A picker will use do this in order to optimize its picking. For example, a picker might use a "pick-ahead" extension to perform early picking and store the results in these fields). The server must not use or modify these event fields until after calling PICKER_FIXUP_EVENT_FROM_WINDOW. [Design Note: this pick-ahead information is crucial for good Looking Glass performance. For smooth interactivity LG needs to be able to process multiple events per rendering frame. Because the unaided Picker extension will only fire off occasional picking requests to the LG Display Server (DS), and not stream them, it is not going to perform well. But if we stream input events to the LG DS and use this as pick-ahead information, and if we toss this pick-ahead information when the window tree changes and use the asynchronous picking protocol instead, we combine the best of both worlds: we have the performance of streaming and the ability to handle changes.] D. Picker Extension Interface The Plug-in Picker interface provides an X extension which is used to activate and control plug-in pickers. This extension is called Picker. It provides the following requests: typedef XID Picker; Picker XPickerAcquirePicker (char *pickerName, int clientMajor, int clientMinor, int *serverMajor, int serverMinor, int nWords, int *pickerControlInfo) If the named picker has not already been loaded into the X server the specified picker shared object library is dynamically loaded and associated with the XID which is returned. The reference count of the picker is initialized to 1. If the named picker has already been loaded into the X server its reference count is merely incremented. XPickerReleasePicker (Picker picker) Decrements the reference count of the given picker. When the reference count reaches 0 any memory associated with the picker can be reclaimed. by the X server. int XPickerBindPicker (Window win, Picker picker) Associates the given picker with the given window. This picker will be used for picking on this window and its descendents. Note that unless the the pickers of all ancestor windows support multi-level picking this picker may be ignored. (See the "Multi-Level Picking section). int XPickerUnbindPicker (Window win) Disassociates the given window's picker from the window. The window's picker is set to the default picker. XID XPickerQueryPicker (Window win) Returns the XID of the picker associated with the given window. If the default picker is bound INVALID is returned. E. Where do the Picker Modules Come From? Picker modules are installed on a machine in a TBD directory. Each is a shared object library (.so). The file names have the format .so.. Multi-Level Picking ------------------- Pickers are allowed to use other pickers to perform picking on subwindows. Pickers which support this are said to support "multi-level picking." Pickers are allowed to support multi-level picking but are not required to do so. A multi-level picker will use the picker associated with a specific window when it reaches that window in its picking search. For example, lets say that the root window is set to use Picker A for its picking. If Picker A determines that the point is somewhere in subwindow B it calls the Picker B, which is the picker of subwindow B. Picker B then continues the search from subwindow B on downward. Along the way, if Picker B supports multi-level, it may call other pickers. Multi-level picking allows desktop environments to be embedded in other desktop environments. For example, imagine Croquet embedded in a window within Looking Glass. However, because multi-level picking is complex to implement it is not required that all pickers support multi-level picking. It is at the discretion of the implementor of the desktop environment. Server Dix Implementation ------------------------- This section describes the changes that must be made to the X server dix code in order to implement the Plug-in Picker interface. In events.c: ComputeFreezes is modified to call PICKER_WINDOWS_RESTRUCTURED before calling XYToWindow. FixupEventFromWindow is modified to only call PICKER_FIXUP_EVENT_FROM_WINDOW. XYToWindow is modified to only call PICKER_XY_TO_WINDOW. CheckMotion is modified to call PICKER_WINDOWS_RESTRUCTURED for the xE == NULL case. ProcQueryPointer is modified to call PICKER_QUERY_POINTER to do its work. In dispatch.c: ProcTranslateCoords is modified to call PICKER_TRANSLATE_COORDS to do its work. Here is the code for the changed routines: void ComputeFreezes() { register DeviceIntPtr replayDev = syncEvents.replayDev; register int i; WindowPtr w; register xEvent *xE; int count; GrabPtr grab; register DeviceIntPtr dev; for (dev = inputInfo.devices; dev; dev = dev->next) FreezeThaw(dev, dev->sync.other || (dev->sync.state >= FROZEN)); if (syncEvents.playingEvents || (!replayDev && !syncEvents.pending)) return; syncEvents.playingEvents = TRUE; if (replayDev) { xE = replayDev->sync.event; count = replayDev->sync.evcount; syncEvents.replayDev = (DeviceIntPtr)NULL; #ifdef PICKER PICKER_WINDOWS_RESTRUCTURED(syncEvents.replayWin); #endif /* PICKER */ w = XYToWindow( XE_KBPTR.rootX, XE_KBPTR.rootY); for (i = 0; i < spriteTraceGood; i++) { if (syncEvents.replayWin == spriteTrace[i]) { if (!CheckDeviceGrabs(replayDev, xE, i+1, count)) { if (replayDev->focus) DeliverFocusedEvent(replayDev, xE, w, count); else DeliverDeviceEvents(w, xE, NullGrab, NullWindow, replayDev, count); } goto playmore; } } /* must not still be in the same stack */ if (replayDev->focus) DeliverFocusedEvent(replayDev, xE, w, count); else DeliverDeviceEvents(w, xE, NullGrab, NullWindow, replayDev, count); } playmore: for (dev = inputInfo.devices; dev; dev = dev->next) { if (!dev->sync.frozen) { PlayReleasedEvents(); break; } } syncEvents.playingEvents = FALSE; /* the following may have been skipped during replay, so do it now */ if ((grab = inputInfo.pointer->grab) && grab->confineTo) { if (grab->confineTo->drawable.pScreen != sprite.hotPhys.pScreen) sprite.hotPhys.x = sprite.hotPhys.y = 0; ConfineCursorToWindow(grab->confineTo, TRUE, TRUE); } else ConfineCursorToWindow(WindowTable[sprite.hotPhys.pScreen->myNum], TRUE, FALSE); PostNewCursor(); } static void FixUpEventFromWindow( xEvent *xE, WindowPtr pWin, Window child, Bool calcChild) { #ifdef PICKER PICKER_FIXUP_EVENT_FROM_WINDOW(pWin, xE, pWin, child, calcChild); #else if (calcChild) { WindowPtr w=spriteTrace[spriteTraceGood-1]; /* If the search ends up past the root should the child field be set to none or should the value in the argument be passed through. It probably doesn't matter since everyone calls this function with child == None anyway. */ while (w) { /* If the source window is same as event window, child should be none. Don't bother going all all the way back to the root. */ if (w == pWin) { child = None; break; } if (w->parent == pWin) { child = w->drawable.id; break; } w = w->parent; } } XE_KBPTR.root = ROOT->drawable.id; XE_KBPTR.event = pWin->drawable.id; if (sprite.hot.pScreen == pWin->drawable.pScreen) { XE_KBPTR.sameScreen = xTrue; XE_KBPTR.child = child; XE_KBPTR.eventX = XE_KBPTR.rootX - pWin->drawable.x; XE_KBPTR.eventY = XE_KBPTR.rootY - pWin->drawable.y; } else { XE_KBPTR.sameScreen = xFalse; XE_KBPTR.child = None; XE_KBPTR.eventX = 0; XE_KBPTR.eventY = 0; } #endif /* PICKER */ } static WindowPtr XYToWindow(int x, int y) { #ifdef PICKER return PICKER_XY_TO_WINDOW(x, y); #else register WindowPtr pWin; BoxRec box; spriteTraceGood = 1; /* root window still there */ pWin = ROOT->firstChild; while (pWin) { if ((pWin->mapped) && (x >= pWin->drawable.x - wBorderWidth (pWin)) && (x < pWin->drawable.x + (int)pWin->drawable.width + wBorderWidth(pWin)) && (y >= pWin->drawable.y - wBorderWidth (pWin)) && (y < pWin->drawable.y + (int)pWin->drawable.height + wBorderWidth (pWin)) #ifdef SHAPE /* When a window is shaped, a further check * is made to see if the point is inside * borderSize */ && (!wBoundingShape(pWin) || PointInBorderSize(pWin, x, y)) && (!wInputShape(pWin) || POINT_IN_REGION(pWin->drawable.pScreen, wInputShape(pWin), x - pWin->drawable.x, y - pWin->drawable.y, &box)) #endif ) { if (spriteTraceGood >= spriteTraceSize) { spriteTraceSize += 10; Must_have_memory = TRUE; /* XXX */ spriteTrace = (WindowPtr *)xrealloc( spriteTrace, spriteTraceSize*sizeof(WindowPtr)); Must_have_memory = FALSE; /* XXX */ } spriteTrace[spriteTraceGood++] = pWin; pWin = pWin->firstChild; } else pWin = pWin->nextSib; } return spriteTrace[spriteTraceGood-1]; #endif /* PICKER */ } static Bool CheckMotion(xEvent *xE) { WindowPtr prevSpriteWin = sprite.win; #ifdef PANORAMIX if(!noPanoramiXExtension) return XineramaCheckMotion(xE); #endif if (xE && !syncEvents.playingEvents) { if (sprite.hot.pScreen != sprite.hotPhys.pScreen) { sprite.hot.pScreen = sprite.hotPhys.pScreen; ROOT = WindowTable[sprite.hot.pScreen->myNum]; } #ifdef XEVIE xeviehot.x = #endif sprite.hot.x = XE_KBPTR.rootX; #ifdef XEVIE xeviehot.y = #endif sprite.hot.y = XE_KBPTR.rootY; if (sprite.hot.x < sprite.physLimits.x1) #ifdef XEVIE xeviehot.x = #endif sprite.hot.x = sprite.physLimits.x1; else if (sprite.hot.x >= sprite.physLimits.x2) #ifdef XEVIE xeviehot.x = #endif sprite.hot.x = sprite.physLimits.x2 - 1; if (sprite.hot.y < sprite.physLimits.y1) #ifdef XEVIE xeviehot.y = #endif sprite.hot.y = sprite.physLimits.y1; else if (sprite.hot.y >= sprite.physLimits.y2) #ifdef XEVIE xeviehot.y = #endif sprite.hot.y = sprite.physLimits.y2 - 1; #ifdef SHAPE if (sprite.hotShape) ConfineToShape(sprite.hotShape, &sprite.hot.x, &sprite.hot.y); #endif sprite.hotPhys = sprite.hot; if ((sprite.hotPhys.x != XE_KBPTR.rootX) || (sprite.hotPhys.y != XE_KBPTR.rootY)) { (*sprite.hotPhys.pScreen->SetCursorPosition)( sprite.hotPhys.pScreen, sprite.hotPhys.x, sprite.hotPhys.y, FALSE); } XE_KBPTR.rootX = sprite.hot.x; XE_KBPTR.rootY = sprite.hot.y; } #ifdef PICKER if (xE == NULL) { #ifdef XEVIE xeviewin = #endif sprite.win = PICKER_WINDOWS_RESTRUCTURED(sprite.win); } #endif /* PICKER */ #ifdef XEVIE xeviewin = #endif sprite.win = XYToWindow(sprite.hot.x, sprite.hot.y); #ifdef notyet if (!(sprite.win->deliverableEvents & Motion_Filter(inputInfo.pointer->button)) !syncEvents.playingEvents) { /* XXX Do PointerNonInterestBox here */ } #endif if (sprite.win != prevSpriteWin) { if (prevSpriteWin != NullWindow) { if (!xE) UpdateCurrentTimeIf(); DoEnterLeaveEvents(prevSpriteWin, sprite.win, NotifyNormal); } PostNewCursor(); return FALSE; } return TRUE; } int ProcQueryPointer(ClientPtr client) { xQueryPointerReply rep; WindowPtr pWin, t; REQUEST(xResourceReq); DeviceIntPtr mouse = inputInfo.pointer; REQUEST_SIZE_MATCH(xResourceReq); pWin = SecurityLookupWindow(stuff->id, client, SecurityReadAccess); if (!pWin) return BadWindow; if (mouse->valuator->motionHintWindow) MaybeStopHint(mouse, client); rep.type = X_Reply; rep.sequenceNumber = client->sequence; rep.mask = mouse->button->state | inputInfo.keyboard->key->state; rep.length = 0; rep.root = (ROOT)->drawable.id; rep.rootX = sprite.hot.x; rep.rootY = sprite.hot.y; rep.child = None; #ifdef PICKER PICKER_QUERY_POINTER(sprite.win, pWin, x, y, &rep.winX, &rep.winY, &rep.child, &rep.sameScreen); #else if (sprite.hot.pScreen == pWin->drawable.pScreen) { rep.sameScreen = xTrue; rep.winX = sprite.hot.x - pWin->drawable.x; rep.winY = sprite.hot.y - pWin->drawable.y; for (t = sprite.win; t; t = t->parent) if (t->parent == pWin) { rep.child = t->drawable.id; break; } } else { rep.sameScreen = xFalse; rep.winX = 0; rep.winY = 0; } #endif /* PICKER */ #ifdef PANORAMIX if(!noPanoramiXExtension) { rep.rootX += panoramiXdataPtr[0].x; rep.rootY += panoramiXdataPtr[0].y; if(stuff->id == rep.root) { rep.winX += panoramiXdataPtr[0].x; rep.winY += panoramiXdataPtr[0].y; } } #endif WriteReplyToClient(client, sizeof(xQueryPointerReply), &rep); return(Success); } int ProcTranslateCoords(register ClientPtr client) { REQUEST(xTranslateCoordsReq); register WindowPtr pWin, pDst; xTranslateCoordsReply rep; REQUEST_SIZE_MATCH(xTranslateCoordsReq); pWin = (WindowPtr)SecurityLookupWindow(stuff->srcWid, client, SecurityReadAccess); if (!pWin) return(BadWindow); pDst = (WindowPtr)SecurityLookupWindow(stuff->dstWid, client, SecurityReadAccess); if (!pDst) return(BadWindow); rep.type = X_Reply; rep.length = 0; rep.sequenceNumber = client->sequence; #ifdef PICKER PICKER_TRANSLATE_COORDS(pWin, pDst, x, y, &rep.dstX, &rep.dstY, &rep.child, &rep.sameScreen); #else if (!SAME_SCREENS(pWin->drawable, pDst->drawable)) { rep.sameScreen = xFalse; rep.child = None; rep.dstX = rep.dstY = 0; } else { rep.sameScreen = xTrue; rep.child = None; INT16 x, y; rep.sameScreen = xTrue; rep.child = None; /* computing absolute coordinates -- adjust to destination later */ x = pWin->drawable.x + stuff->srcX; y = pWin->drawable.y + stuff->srcY; pWin = pDst->firstChild; while (pWin) { #ifdef SHAPE BoxRec box; #endif if ((pWin->mapped) && (x >= pWin->drawable.x - wBorderWidth (pWin)) && (x < pWin->drawable.x + (int)pWin->drawable.width + wBorderWidth (pWin)) && (y >= pWin->drawable.y - wBorderWidth (pWin)) && (y < pWin->drawable.y + (int)pWin->drawable.height + wBorderWidth (pWin)) #ifdef SHAPE /* When a window is shaped, a further check * is made to see if the point is inside * borderSize */ && (!wBoundingShape(pWin) || POINT_IN_REGION(pWin->drawable.pScreen, &pWin->borderSize, x, y, &box)) && (!wInputShape(pWin) || POINT_IN_REGION(pWin->drawable.pScreen, wInputShape(pWin), x - pWin->drawable.x, y - pWin->drawable.y, &box)) #endif ) { rep.child = pWin->drawable.id; pWin = (WindowPtr) NULL; } else pWin = pWin->nextSib; } /* adjust to destination coordinates */ rep.dstX = x - pDst->drawable.x; rep.dstY = y - pDst->drawable.y; #endif /* PICKER */ } WriteReplyToClient(client, sizeof(xTranslateCoordsReply), &rep); return(client->noClientException); }