Hi,
I suppose it's possible. However, I'm not sure that the uploader will be able to properly take into account the failed saving of the file.
If you cancel the saving of the file in onBeforeFileCreate with the $do parameter set to false.
That will definitely prevent the file from being added in the hikashop_file table.
However, the file will still be uploaded in the upload folder. And I'm not sure the uploader will be able to properly display an error back to the customer.
So it's worth a try, but I can't guarantee it will work. That would require some development work to confirm that.