Do you have another Ora8i instance to test? - I do not have any 8i left in the shop but I do not remember such a thing ever happened.

As far as I know there is no reason why 'alter database backup controlfile to trace' would create a binary file instead of a text file.

If you are certain this is actually happening I would suggest to open a TAR with Oracle.