vmstat every so often shows the memory page scan rate going through the roof which occasionally brings the box to a standstill.
the problem we are having is marrying up the scan rate usage to a process or indeed a series of processes.
There does not appear to be any unusual activity in the databases. Nor can see how rogue processes or again something unusual.
We have tracked when the issue first started (begining of the month) using grid, and are currently trying to ascertain what has changed on the system. But fundamentally if we can see what process(es) are casuing the scan rate to ramp up we'd have a better idea.
b) "I have a lot of oracle processes" : http://www.dba-oracle.com/t_tuning_cpu_usage_vmstat.htm speak about it, but the fact is that in outage situations (ie, low CPU power remaining, low memory available, etc) the system begins to steal pages for the processes - the assumption here is, you installed something new, this thing (maybe due to bugs) is consuming a lot of resources, this is the root cause... OR maybe not, simply your applications begun to do more work (bigger volumes, new programs doing more work, etc) and the timing is just coincidence , the new installs are not interfering...
My initial advice : check the links, see about configs, AND search for intensive processes (using top, iostat, glance in the OS, and the Oracle tools/dictionary views inside the db), tune them/stop them....
We ended up rebooting the machine, very, very annoying thing to have to do. But now sys mem is down to about 9Gb (in line with the DR machine) so definitely some kind of memory leak. Also the page scan rate is minimal (close to 0), i.e. what it should it be.
I don't know enough of how glance calculates 'system memory' so it is something I need to look into.
We've been through everything that has been recently installed and nothing appears to be the culprit. Saying that, we have noticed an issue with one application that uses dbms_pipes...and doesn't always close them. So that is something to consider, although the amount of memory a pipe uses is tiny.