Hi,
Well, there is no option for that. We added recently the option to be able to display or not the module independently on the listings and the product pages, but we didn't think about the contact page.
An easy solution which wouldn't require any coding would be to install the extension Advanced Modules Manager. That will give you a lot more options to display or not modules on the different pages of your website and you'll be able to remove the filter module from the contact page thanks to that.