Hello,
Directly what you required isn't possible via classic settings, BUT we see a possibility that we will let you judge, because depending on your needs it may not be suitable.
Follow me step by step :
1. You can only restrict one type of custom field, and that is those in the order table.
=> You will find a specific option to restrict per shipping method
2. In your Checkout Workflow configuration part, you can define precisely where you want to have your order custom field
=> Maybe with some Css you can dress it, in order to dress it up to look more like your custom field address.
Note : There are limits to this little hack, and the most important is that this custom field as it's an order custom field
won't be displayed at all next to customer addresses (order listing, order form AND email)
=> If required for email, it's possible to customize the relative email.
Hoping that this will allow your reflection to progress.
Regards