32Bit oracle is restricted to an SGA of 1.75Gb, any more and you will get errors (you can go up to 3Gb is you set some boot.ini parameters as you have done)

Dont forget that the OS needs memory as well, you may well be blowing all your machines resources

Are you using a lot of PGA as well?