Why is X slow when using a Mac Pro, Ubuntu, and a remote desktop solution (tight vnc and nx)?

[Brian Takita]

Hello, I have Ubuntu Hardy Heron installed on a Mac Pro 8 core 64 bit
machine. The Mac Pro uses an ATI card.
Unfortunately, whenever somebody connects via vnc or nx, X gets very slow.

What happens is it acts normally, but every couple of seconds, it will
pause and not accept any input or refresh the screen.

This does not seem to happen on Ubuntu with a 32 bit 2 core on iMac.
The iMac uses a nVidia graphics card.

Thanks,
Brian