The source * comes from Chronology of Qing Poetry. It means that you like the bright moon, but the bright moon doesn't understand amorous feelings. I was originally devoted to the bright moon, but the bright moon didn't shine on me, but it filled all the ditches. What I like/want to do is not realized, and what is more helpless is that the people/things I like/want to do are even worse.
Those who know me worry me, and those who don't know what I want worry me.
That is, people who know me will feel worried and worried about my situation, while people who don't know me will think that I am pursuing something in ignorance.