If others don't know the reason, how can he not know?

Fu Yechuan's office is adjacent to his, and he likes to make phone calls on the balcony the most.

As long as he goes to the balcony to make a phone call, his office can hear some of the content.