It wasn’t unfounded.

The emperor might want to keep Astelle and the prince by his side, but also eliminate the Duke that is a threat.

The Duke ordered his henchmen after much deliberation.