Tried that, works on my system.

The ^D (carrot D, not ctrl+D) tells grep to look at the first character of the line and ignore the rest.
Code:
dev42:/home/users/jeffh/pando $ ls -l  
total 2
-rw-r--r--   1 jeffh    progr          0 Oct 12 17:23 dir.txt
drwxr-xr-x   2 jeffh    progr        512 Oct 12 17:24 mydir
-rw-r--r--   1 jeffh    progr          0 Oct 12 17:23 readme.txt
dev42:/home/users/jeffh/pando $ ls -l | grep ^d
drwxr-xr-x   2 jeffh    progr        512 Oct 12 17:24 mydir