I see now. Sorry for not reading careful the question.
Well seems it depends if the to_date will get evaluated first or is_date will be evaluated first. Obviously to_date gets evaluated first :-(
Maybe u can try
select count(*) from (select * from emp where IS_DATE(BIRTHDATE)=0)
where
to_date(birthdate,'mmddyyyy')>='01-JAN-90'

Cheers