Do you have I/O contention? My bet is that you are building the index in one tablespace that is getting pounded.