I've seen problems with slow connections if Oracle Trace is enabled.

Are there any large files under: $ORACLE_HOME/otrace/admin

e.g. process.dat, collect.dat and regid.dat

If so, then you can switch off tracing in the listener.ora by setting EPC_DISABLED

If not, do you have Oracle Net tracing switched on?