If you have more than 4GB of RAM, you need x64. X32 only reads up to ~3.5 of your RAM.
That is factually incorrect, but mostly by technicality. Most modern Linux distributions compile their 32-bit x86 kernels with PAE support. Pysical Address Extension is a feature of the processor than, when used by the operating system, allows it to map more than 4GB of RAM into a 32-bit address space. So, for example, if Debian Jessie were installed with the linux-image-3.10-3-486 kernel it would be 32-bit and thusly only able to address up to 4 GB of RAM. However Debian Jessie were installed with the linux-image-3.10-3-686-pae kernel it would still be 32-bit but would be able to address up to 4 PiB of RAM thanks to PAE.
Then there is the issue of the x32 architecture versus i386, i486, i586, and i686. Despite the fact that some people use x32 as shorthand for 32-bit x86 processors (i.e. i386-i686), it has distinct meaning in Linux. In the kernel x32 refers to another ABI for the x86 architecture which attempts to squeeze extra performance out of modern AMD64 (64-bit x86) processors by using CPU instructions and registers introduced in AMD64 while retaining the 32-bit pointers found in i686. This theoretically allows it to maintain a lower memory footprint while simultaneously taking advantage of architecture improvements that directly improve performance. However x32 is still very much in the experimental stage, and Debian and Gentoo are the only distributions I am aware of that have working ports of it. Despite its current experimental status, the point stands that it is incorrect to refer to 32-bit x86 processors as x32 when you actually mean to reference i686 (with or without PAE).
It's a 3GB VPS, so perhaps best with 32bit? Is there any difference in performance?
It is a common misconception to think that the one-and-only thing that 64-bit processors offer over their 32-bit counterparts is the ability to address more than 4GB of RAM. There are substantial architectural improvements in AMD64 over i686 that programs compiled for that architecture will be able to take advantage of. While there are still relatively few applications for Windows that take advantage of those architectural improvements outside of heavy computational programs and the operating system itself, that is not the case in Linux. Virtually every program in modern Linux distributions has been compiled to take advantage of the 64-bit x86 ISA. Web servers, such as Apache and nGenx for example, often see a substantial performance boost on 64-bit systems. Despite the fact that CentOS 5 uses an extremely old kernel and userland which will not perform nearly as well as a more modern software stack, I still recommend it above i386 CentOS if your hardware is capable of supporting it. That said, CentOS 6 would be a huge improvement if your VPS provider allows it. Personally, however, I would go with Debian 7 because it generally performs much better than CentOS, especially on the relatively weak machines VPS providers usually supply.