[PATCH renderproto] Add floating point transforms

[Keith Packard]

Siarhei Siamashka <siarhei.siamashka at gmail.com> writes:

> Do you have a reproducible testcase for this problem? Being off by
> several pixels seems to be very wrong. Especially if this happens
> often.

Non-affine transforms cause problems. I noticed this with RandR when
doing keystone correction.

Here's a simple example:

Desired matrix:

 33.13915858	15.17037037	-16384.00000000
 0.00000000	23.73634938	0.00000000
 -0.00057170	-0.00182142	25.70348287

Rounded to 16.16 fixed matrix:

 33.13916016	15.17036438	-16384.00000000
 0.00000000	23.73634338	0.00000000
 -0.00056458	-0.00181580	25.70347595

Actual bits:

 { 0x00014dba, 0x000098c6, 0xfd7b7d45 },
 { 0x00000000, 0x0000ef09, 0x00000000 },
 { 0xffffffff, 0xfffffffb, 0x000102d9 },

Take the point 2560,1600 and transform:

desired:        4348.04, 1780.87
actual:         4342.50, 1778.60

error:          5.99
  
> Floating point format supports a wider range of values (unnecessary
> if the rest of XRENDER remains restricted to 16-bit coordinates) but
> has much worse accuracy in corner cases and is poorly predictable.
> The 32-bit floating point format does not look like an obvious
> upgrade over 16.16 fixed point.

Transforms are different than specifying geometry though; they actually
need the wide dynamic range offered by floating point.

-- 
keith.packard at intel.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 810 bytes
Desc: not available
URL: <http://lists.x.org/archives/xorg-devel/attachments/20140818/5e3cede6/attachment.sig>