Hi,
To do it only for the shipping invoice you could add such line before your code:
<?php if($this->invoice_type=='full') return; ?>
If you want to remove some fields from the address display, you can change the "address format" setting of the HikaShop configuration.
Note that this will affect both your code and the address display on the checkout, order and invoice too.
If you want it to affect only your code, you would have to not call displayAddress and directly echo the data of the address from $this->element->shipping_address. For example:
echo $this->element->shipping_address->address_city;