I had the exact similar problem. I found no solution. Since I was working on a development server, I decided to create a fresh instance, while ignoring the one already there. By the way I did an upgrade from 8.1.5 to 8.1.6. The old instance was 8.1.5 and trying to connect using svrmgrl failed with this error. All I was trying to do is to upgrade this 8.1.5. instance to 8.1.6. The newly created instance has no such problems.
You must establish an oracle service prior to using svrmgr to connect internal on nt. The oracle service is not the database but it must be running in order to startup the database. To do this set you oracle sid and issue the following
oradim80 -new -sid <sidname> -intpwd <password> -startmode auto -pfile <pathname\filename>
At this point you should be able to connect internal and start up the instance in the nomount mode and then create the database however you wish to.