Send order confirmation to client

You need to configure your email configuration in the joomla configuration.
Please make sure that you're able to send emails with joomla (and not rsform). Then, you should be able to send emails with HikaShop. For example, you can test that with the user account confirmation email.

Then, you should check the settings in the "mail" configuration of HikaShop.
HikaShop sends the emails using joomla's functions. If you're able to send other joomla emails, it means that it comes from your email server which rejected the email for some reason (encoding not supported, name in the from, etc). I invite you to check these settings with your hosting company so that they can tell you what should be changed so that it works properly with their email server.