My experience is the same. Hand drilling with a gun drill works to a point. The deeper you go the less accurate it gets. My first try has been to replace the tailstock with a hunk of metal that has a guide tube through it. Drilling by hand is much more accurate with the guide but the length of the guide reduces the depth the bit can go. The piece is held by the chuck and a steady rest.
I have thoughts in my head about modifying a captured hollowing system to hold gun drills. Someday I'll find the time to try it.