Set the following line into your listener.ora to stop logging any events into listener log file:

LOGGING_LISTENER = OFF

You should of course restart the listener for the changes to take effect. And no, it will not affect the database connections.