How long time does it take to connect via FTP? If long, then the problem might not be within Oracle.
You may try to to transfer the listener to another port. Before doing that however, check that you use the IP address of the database server in the listener.ora and tnsnames.ora files, check that tracing is not on (for the listener, SQL*Net too).
Try to flush the shared pool: ALTER SYSTEM FLUSH SHARED_POOL; and add log_directory_client='null' to sqlnet.ora.