Yes, there are moments when that contributes to making a bit of a bad impression, or having some management troubles. Kindergartners, unlike, say, fourth graders, may be so meek that they won't tell me which way to walk down the hall even when I ask them to. But if we have to turn around ("Why are we going this way?") to correct a mistake, there may be some little issues. ("He got in front of me!" "I was there!")
Most of us aren't homing pigeons, but there are neurons that fire in our brain to help us record locations and find our way back. A normal human (like a normal mouse) will also show an ability to form cognitive maps or neural representations of place. They don't have to walk a path in a particular direction to understand it from that direction. And yet that ability can be missing -- even when cognitive/ reasoning skills are intact.