Hi,
Not at all.
An order has to be created before the redirection to the payment gateway.
Then, when the payment is done on the payment gateway, the status of the order is changed to confirmed.
So you should only take into account confirmed orders.
If the buyer click on cancel on the payment gateway, the order status will be changed from "created" to "cancelled" and the user will then be back on the checkout (to select another payment method for example).
Also, you can activate the "auto cancel orders plugin" to automatically cancel created orders after a certain period of time.
To conclude, there is no need to delete orders (and that should actually not be done in many countries where you're supposed to keep a record of an order even if not paid by the customer, and you're not supposed to have holes in the numbering of the orders) since you already have a mean to distinguish between created, cancelled and confirmed orders.