[PATCH renderproto] Add floating point transforms

[Siarhei Siamashka]

On Mon, 18 Aug 2014 14:02:13 -0700
Keith Packard <keithp at keithp.com> wrote:

> Siarhei Siamashka <siarhei.siamashka at gmail.com> writes:
> 
> > Do you have a reproducible testcase for this problem? Being off by
> > several pixels seems to be very wrong. Especially if this happens
> > often.
> 
> Non-affine transforms cause problems. I noticed this with RandR when
> doing keystone correction.
> 
> Here's a simple example:
> 
> Desired matrix:
> 
>  33.13915858	15.17037037	-16384.00000000
>  0.00000000	23.73634938	0.00000000
>  -0.00057170	-0.00182142	25.70348287
> 
> Rounded to 16.16 fixed matrix:
> 
>  33.13916016	15.17036438	-16384.00000000
>  0.00000000	23.73634338	0.00000000
>  -0.00056458	-0.00181580	25.70347595
> 
> Actual bits:
> 
>  { 0x00014dba, 0x000098c6, 0xfd7b7d45 },
>  { 0x00000000, 0x0000ef09, 0x00000000 },
>  { 0xffffffff, 0xfffffffb, 0x000102d9 },
> 
> Take the point 2560,1600 and transform:
> 
> desired:        4348.04, 1780.87
> actual:         4342.50, 1778.60
> 
> error:          5.99

It is possible to pre-multiply all the coefficients in the desired
matrix by 1.147703 before converting it to 16.16 fixed point format.
In this case we get the following rounded coefficients in the matrix:

  38.033912   17.411080   -18803.965952
  0.000000    27.242279   0.000000
  -0.000656   -0.002090   29.499964

The following actual bits:

  { 0x002608ae, 0x0011693d, 0xb68c08b7 },
  { 0x00000000, 0x001b3e06, 0x00000000 },
  { 0xffffffd5, 0xffffff77, 0x001d7ffe },

And the 2560,1600 point transforms to 4348.03, 1780.86

I found this particular multiplication constant by a simple bruteforce
search (minimizing the rounding errors for the m[2][0] and m[2][1]
matrix coefficients). But quite likely, a reasonably good constant
can be also found by some better performing algorithm, suitable
for realtime use in applications.

Basically, for an affine transform we just need to do correct rounding.
And for a projective transform, we also need to nudge the matrix a bit
in order to minimize rounding errors when converting to 16.16 fixed
point.

Would this be a suitable solution for the xrandr command line tool?

-- 
Best regards,
Siarhei Siamashka