Memory allocated to each process is 64Mb on HP UX.

When we do ipcs -am on our HP UX server, it shows up as each process using 64Mb of memory, and the Oracle process is using 320mb (when the SGA is 240mb). WHY? and How do we calculate the UGA size?

You mean BG process or Server Process?
What is your SORT_AREA_SIZE?

Each user process uses max. 64Mb, not the background processes.
rp7400a-oracle>ipcs -am | grep oracle
m 140 0x00000000 --rw-r----- oracle dba oracle dba 20 67108864 7447 20690 15:06:38 15:08:56 13:00:38
m 141 0x00000000 --rw-r----- oracle dba oracle dba 20 67108864 7447 20690 15:06:38 15:08:56 13:00:38
m 142 0x00000000 --rw-r----- oracle dba oracle dba 20 67108864 7447 20690 15:06:38 15:08:56 13:00:38
m 143 0x00000000 --rw-r----- oracle dba oracle dba 20 67108864 7447 20690 15:06:38 15:08:56 13:00:38
m 144 0x00000000 --rw-r----- oracle dba oracle dba 20 67108864 7447 20690 15:06:38 15:08:56 13:00:38
m 145 0x00000000 --rw-r----- oracle dba oracle dba 20 67108864 7447 20690 15:06:38 15:08:56 13:00:38
m 146 0x4d47f744 --rw-r----- oracle dba oracle dba 20 1544192 7447 20690 15:06:38 15:08:56 13:00:38

The shared memory allocated is 320Mb, and the SGA is 270mb.
rp7400b#[/aodb/oradata]ipcs -bm| grep oracle
m 30 0x4d47f744 --rw-r----- oracle dba 320311296

SQL> @sga

SUM(VALUE)+(1024*1024)
----------------------
286785536

what does "show sga" say?

show sga

Total System Global Area 286072228 bytes
Fixed Size 454052 bytes
Variable Size 201326592 bytes
Database Buffers 83886080 bytes
Redo Buffers 405504 bytes

My guess is the SGA is segmented in 64M segments because your kernel parameters say that the max segment size is 64M.

The shared memory segment is 320Mb,as mentioned above.

rp7400b#[/aodb/oradata]ipcs -bm| grep oracle
m 30 0x4d47f744 --rw-r----- oracle dba 320311296

and each process is 64Mb,

rp7400a-oracle>ipcs -am | grep oracle
m 140 0x00000000 --rw-r----- oracle dba oracle dba 20 67108864 7447 20690 15:06:38 15:08:56 13:00:38
m 141 0x00000000 --rw-r----- oracle dba oracle dba 20 67108864 7447 20690 15:06:38 15:08:56 13:00:38
m 142 0x00000000 --rw-r----- oracle dba oracle dba 20 67108864 7447 20690 15:06:38 15:08:56 13:00:38
m 143 0x00000000 --rw-r----- oracle dba oracle dba 20 67108864 7447 20690 15:06:38 15:08:56 13:00:38
m 144 0x00000000 --rw-r----- oracle dba oracle dba 20 67108864 7447 20690 15:06:38 15:08:56 13:00:38
m 145 0x00000000 --rw-r----- oracle dba oracle dba 20 67108864 7447 20690 15:06:38 15:08:56 13:00:38
m 146 0x4d47f744 --rw-r----- oracle dba oracle dba 20 1544192 7447 20690 15:06:38 15:08:56 13:00:38

therefore the shared memory is not being fragmented.

Is there a working set size parameter on Unix for each process?