Why don't you make the field as suppressed, so that user can't change the field value in MIRO.
Make the Invoice party as suppressed by using screen variant SHD0.
Then user can't able to change the invoice party (it will only flow from PO automatically)