I have found the "Invisible wall" very annoying in swamp

When infront of the witches house in swamp, inbetween the pond thing, and the house. When I try to enter my house, or someone elses it will come up with the message "You may not enter a house while on a wall!