VNC patch for Xserver 1.3

[Stefan Dirsch]

On Sat, Oct 20, 2007 at 11:43:37AM +0200, Stefan Dirsch wrote:
> On Thu, Oct 18, 2007 at 09:21:21AM +0100, Alan Hourihane wrote:
> > On Wed, 2007-10-17 at 13:08 +0200, Stefan Dirsch wrote:
> > > On Thu, Aug 23, 2007 at 04:31:02PM +0200, Stefan Dirsch wrote:
> > > > Thanks. Unfortunately the Xserver crashes immediately now.
> > > > 
> > > > (gdb) run
> > > > 
> > > > Program received signal SIGSEGV, Segmentation fault.
> > > > [Switching to Thread 0x2ab337bb4290 (LWP 4737)]
> > > > 0x00000000004a0bdc in xf86CollectInputOptions ()
> > > > (gdb) bt
> > > > #0  0x00000000004a0bdc in xf86CollectInputOptions ()
> > > > #1  0x00002ab338d5a019 in xf86rfbKeybInit ()
> > > >    from /usr/lib64/xorg/modules//extensions/libvnc.so
> > > > #2  0x0000000000467fdc in InitInput ()
> > > > #3  0x0000000000439931 in main ()
> > > > (gdb) quit
> > > 
> > > Tried again with X.Org 7.3 and current xf4vnc CVS. Still crashing. :-)
> > > Here are the results of my investigations.
> > > 
> > > Backtrace:
> > > 0: X(xf86SigHandler+0x6a) [0x48ce3a]
> > > 1: /lib64/libc.so.6 [0x2b2b2e271ba0]
> > > 2: /lib64/libc.so.6(strlen+0x40) [0x2b2b2e2b6ba0]
> > > 3: X(xf86ActivateDevice+0x59) [0x47b389]
> > > 4: X(InitInput+0x4d) [0x46785d]
> > > 5: X(main+0x370) [0x436210]
> > > 6: /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b2b2e25eb24]
> > > 7: X(FontFileCompleteXLFD+0x259) [0x435659]
> > > 
> > > Program received signal SIGSEGV, Segmentation fault.
> > > 0x00002b587f48eba0 in strlen () from /lib64/libc.so.6
> > > (gdb) bt
> > > #0  0x00002b587f48eba0 in strlen () from /lib64/libc.so.6
> > > #1  0x000000000047b389 in xf86ActivateDevice (local=0x8be9d0)
> > >     at xf86Xinput.c:158
> > > #2  0x000000000046785d in InitInput (argc=<value optimized out>, 
> > >     argv=<value optimized out>) at xf86Init.c:960
> > > #3  0x0000000000436210 in main (argc=1, argv=0x7fff2ce5c1a8, envp=0x83a090)
> > >     at main.c:397
> > > 
> > > 158             local->atom = MakeAtom(local->type_name,
> > > 159                                    strlen(local->type_name),
> > > 160                                    TRUE);
> > > 
> > > (gdb) p* local
> > > $3 = {next = 0x8be900, name = 0x7ef4d0 "Mouse[3]", flags = 78, 
> > >   device_control = 0x2b0c70993820 <xf86rfbMouseControlProc>, read_input = 0, 
> > >   control_proc = 0, close_proc = 0, switch_mode = 0, conversion_proc = 0, 
> > >   reverse_conversion_proc = 0, set_device_valuators = 0, fd = -1, atom = 0, 
> > >   dev = 0x0, private = 0x0, private_flags = 0, first = 0, last = 0, old_x = 0, 
> > >   old_y = 0, type_name = 0x0, always_core_feedback = 0x2b0c709acb3c, 
> > >   conf_idev = 0x0, drv = 0x7f5000, module = 0x0, options = 0x0, 
> > >   history_size = 256}
> > > 
> > > Looks like the pInfo structure gets not filled properly. I'm not sure,
> > > where this is done. Here is the configuration I'm currently using.
> > > 
> > > Section "InputDevice"
> > >   Driver       "rfbkeyb"
> > >   Identifier   "Keyboard[2]"
> > >   Option       "InputFashion" "VNC"
> > > EndSection
> > > 
> > > Section "InputDevice"
> > >   Driver       "rfbmouse"
> > >   Identifier   "Mouse[3]"
> > >   Option       "InputFashion" "VNC"
> > > EndSection
> > > 
> > > Section "ServerLayout"
> > >   [...]
> > >   InputDevice  "Keyboard[2]" "ExtraKeyboard"
> > >   InputDevice  "Mouse[3]" "ExtraPointer"
> > > EndSection
> > > 
> > > You get different results for pInfo for a regular mouse section and of
> > > course no crashes.
> > > 
> > > Section "InputDevice"
> > >   Driver       "mouse"
> > >   Identifier   "Mouse[1]"
> > >   Option       "Buttons" "5"
> > >   Option       "Device" "/dev/input/mice"
> > >   Option       "Name" "MX310"
> > >   Option       "Protocol" "ExplorerPS/2"
> > >   Option       "Vendor" "Logitech"
> > >   Option       "ZAxisMapping" "4 5"
> > > EndSection
> > > 
> > > (gdb) p* local
> > > $2 = {next = 0x8b9230, name = 0x7ee9d0 "Mouse[1]", flags = 78, 
> > >   device_control = 0x2b61f0cebea0 <MouseProc>, 
> > >   read_input = 0x2b61f0ceb1a0 <MouseReadInput>, control_proc = 0, 
> > >   close_proc = 0, switch_mode = 0, 
> > >   conversion_proc = 0x2b61f0ce86a0 <MouseConvert>, 
> > >   reverse_conversion_proc = 0, set_device_valuators = 0, fd = -1, atom = 0, 
> > >   dev = 0x0, private = 0x8be180, private_flags = 0, first = 0, last = 0, 
> > >   old_x = 0, old_y = 0, type_name = 0x2b61f0cedd55 "MOUSE", 
> > >   always_core_feedback = 0x0, conf_idev = 0x7f4e90, drv = 0x808070, 
> > >   module = 0x8be0a0, options = 0x8bda80, history_size = 256}
> > > 
> > > Alan, can you at least reproduce this problem?
> > 
> > I'll try and get to this asap.
> 
> Could you try this on a x86_64 system? It doesn't seem to happen on a
> 32bit x86 system.

AFAICS the pInfo structure is filled correctly in xf86rfbMouseInit(),
but the content gets lost again in InitInput() before calling
xf86ActivateDevice(). :-(

I hope I'm not the only one, who can reproduce this issue.

Best regards,
Stefan

Public Key available
------------------------------------------------------
Stefan Dirsch (Res. & Dev.)   SUSE LINUX Products GmbH
Tel: 0911-740 53 0            Maxfeldstraße 5
FAX: 0911-740 53 479          D-90409 Nürnberg
http://www.suse.de            Germany 
-----------------------------------------------------------------
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
-----------------------------------------------------------------