Very probably it is like this:

Your query without ORDER BY is returning you the first few rows to your front-end and that's why you have the impression it is so quick. But if you let it return all the rows to the very last one you would probably be surprised that it is not realy so quick as it seemed in the beginning.

With ORDER BY, your query does have no choice but to first process all the rows, then sort them, and only then it is able to return the first set of rows to your front end. But this doesn't mean that the sorting is realy a bottleneck for this query.

But of course, as others have suggested, it is also possible that your sorting is happening on disk, which might realy be a bottleneck.