The only difference as you said is the architectural one. The disadvantages are the patch. There is also another good point one could notice is what ever your OS arch., you would be restricted by the I/O bottle necks. :o
Here are some postings that I was able to scavenge from oracle Forum:
The 64 bit vs 32 bit question relates to the operating system not the database. The 64 bit machine has a wider instruction set capability than the 32 bit. Even though the 64 bit machine has the potential to process data faster than 32 bit the over riding bottle neck is still the input/output devices and connections. Oracle offers the database software compatible with the operating system. If your platform is 32 bit the Oracle software is optimized to run on 32 bits. The same is true for 64 bit machines. At this time most software is still written for 32Kbit machines and is merely compiled to run on a 64Kbit platform.
The primary advantage of a 64-bit memory addressing is the improved scalability of the machine. Applications can store more data in the larger amount of memory available and reduce considerably calls to the I/O subsystem. If you are not utilizing the large memory features, then you may be better off running with 32-bit Oracle.