Only in this way can ghosts hurt you. This is called cause and effect!
There are laws between the dead and the dead, and there are laws between the dead and the dead.
As you said, as the saying goes, people are afraid of ghosts for three points, and ghosts are afraid of people for seven points.
Indeed, an upright man is often protected by a kind god and a ghost of Daford, as well as his dead relatives. Even if the ghost wants to hurt him, he can't do anything.
In addition, people who believe in Buddhism are protected by bodhisattvas and dragons, and ghosts will be very respectful when they see such people!