OS Error 11, EAGAIN, means you are out of process space at the OS level, not the Oracle level.

You need to either increase the amount of memory available to Oracle or the number of processes the Oracle account can run simultaneously. Probably the number of processes.

It must be a pretty large sort.

By the way, Oracle will try to use memory to sort in SORT_AREA_SIZE chunks, using the TEMP tablespace when the sort set is too large to fit in one chunk, and possibly data files if too large for temp space (I am vague on this, but I seem to remember the 'files' thing from the documentation.)