Well, this has nothing to do with buffer cache as a part of a SGA.

This "client swapping" depends on what kind of client application/tool you use. For example, if you use SQL*Plus you won't notice any swapping on client machine, because all the recived rows are immediately shown on the screen and screen scrolls automaticaly when new rows arive.

But if you use some kind of tool that presents returned rows in some kind of "data grid" on the display, only first few rows are displayed at the screen, while others have to be "buffered" somewhere in the memory in case user chooses to scroll up/down the screen to see the rows that are currently not visible. If the amount of the returned rows is huge, client machine have to swap the memory to disk.