I think it goes somehow like this.

You do a full table scan.

Oracle reads a set of blocks ( i think using db_file_multiblock_read_count ) and puts the data in the db buffer cache.

Next oracle reads then next set of blocks and puts them in the db buffer cache.
Now the blocks in the buffer cache are put on the list of free/available blocks for the buffer cache.( do not remember the exact name )

So,

when you are alone on the db, the buffer cache will fill with the data from the table, but will be replaced by new blocks during the following reads.

when you are not alone on the db, the select will not monopolise the db buffer cache. It will fill it, but when other sessions need db buffer cache blocks, they will use the blocks used by the previous read.

Regards
Gert