Because the emperor hated Astelle and if this story was leaked, he would take them back.

Astelle wouldn’t be able to do anything even if she lost the keepsakes.

It was obvious that the Duke was calculating like that.