try set:use_indirect_data_buffers=true and try again.
Another thing:
You can use orastack to reduce the memory consumed by per connection.Search metalink and you can get it.
It seems that you are using NT4 Enterprise edition and you mentioned the 3GB limit?