Paypal payment notification refused:invalid response - please advise

-- HikaShop version -- : 3.1.1
-- Joomla version -- : 3.7.4
-- PHP version -- : 5.6.28
-- Error-message(debug-mod must be tuned on) -- : Paypal payment notification refused:invalid response

We have an ongoing issue with a Hikashop installation and would be grateful for your assistance. Payments are made via PayPal Pro Hosted (the payment page is hosted on PP’s own server) and using the Hikashop Paypal Website Payments Pro Hosted Payment plugin plugin.
Normally the a successful order confirmation is returned to the site without any problem. However, as of three weeks ago, we get the message from HS as below:
Subject: Paypal payment notification refused:invalid response
Hello,
A paypal notification was refused because the response from the paypal server was invalid
Check the documentation concerning this issue at www.hikashop.com/index.php?option=com_up...rror#invalidresponse
This notification was for the order JD17-B5D9A8 on the website shop.justducks.co.uk/ You can access the order details directly by clicking on the link below after logging in your back end:
shop.justducks.co.uk/administrator/index...sk=edit&order_id=598

Orders are marked as “Created”, not completed. The user can manually change the order to “Completed”, but this is a hassle and is not how it was working previously.
We have tried the following:
1. Updated both HS and Joomla to the latest stable versions
2. Applied the patch as outlined in a separate post
3. Changed PayPal account to UTF-8
4. Checked the payment logs – nothing appears out of place compared to successful transactions
Please advise on what steps should be taken to resolve this.
Many thanks




ADDITIONAL SERVER INFORMATION
PHP Built On Linux s166-62-85-241.secureserver.net 2.6.32-042stab094.7 #1 SMP Wed Oct 22 12:43:21 MSK 2014 x86_64
Database Version 5.6.35
Database Collation latin1_swedish_ci
Database Connection Collation utf8mb4_general_ci
PHP Version 5.6.28
Web Server Apache
WebServer to PHP Interface cgi-fcgi
Joomla! Version Joomla! 3.7.4 Stable [ Amani ] 25-July-2017 11:11 GMT
Joomla! Platform Version Joomla Platform 13.1.0 Stable [ Curiosity ] 24-Apr-2013 00:00 GMT

Have you checked the latest PayPal security policies?

There may be a UK special version, but this is the PayPal information from the USA announcements:

1) TLS 1.2 Upgrade

The most secure protocol for sharing information on the web today is Transport Layer Security (TLS) version 1.2. PayPal is enabling support for TLS 1.2 for all secure connections and in 2016 will start requiring its use. You will need to verify that your environment supports TLS 1.2 and if necessary make appropriate updates. PayPal is updating its services to require TLS v1.2 for all HTTPS connections in June of 2017. After that time, all TLS v1.0 and TLS v1.1 API connections will be refused.

Under PCI Compliance requirements, TLS 1.2 is becoming mandatory for ALL websites that accept credit cards by January 2018, so anyone experiencing problems won't just be seeing it on your site, it will begin to affect their entire online buying experience more and more over the next few months.

2) IPN Verification Postback to HTTPS

If you are using PayPal’s Instant Payment Notification (IPN) service, you will need to ensure that HTTPS is used when posting the message back to PayPal for verification. After June of 2017 HTTP postbacks will no longer be supported.

The full announcement by PayPal is here (many of these changes have already been done over the last year): devblog.paypal.com/upcoming-security-changes-notice/

I noticed your site isn't using SSL, so that may be the issue right there: cl.ly/2e230f1m1U1N

Thanks for both suggestions.

PolishedGeek, we'll have a look at the SSL and see if this changes anything for us.

Jerome, that thread suggests hat this applies only to the standard PP plugin and not the Hikashop PayPal Website Payments Pro Hosted Payment plugin, so I'm hesitant to apply it.

I can confirm that we're running a different payment module, because the client has PayPal Pro Hopsted.

Separately, the site isn't SSL because the transactions are conducted on PayPal's servers, not ours - when the "complete purchase" button is pressed, the user is taken to the PP site to enter their payment information.

Any more ideas welcome!

Jerome wrote: Hello,

Please check that forum thread : www.hikashop.com/forum/payment-methods/8...ents-pro-hosted.html
Which HikaShop payment plugin are you using ?

Regards,

As an update to this issue, we've been in contact with PayPal and they have confirmed that one of the transactions which was returned to the HS system with the original error message was, as far as PayPal is concerned, a fully valid transaction.

This (slightly expurgated) from their response to us:

Time Created
Jul 31, 2017 06:54:26 PDT
PayPal Account
JustDucks Ltd
Transaction ID
xx25M
Delivery Status
Sent
HTTP Response
200
Last IPN send Attempt
Jul 31, 2017 06:54:28 PDT
Destination URL
shop.justducks.co.uk/index.php?option=co...t&lang=en&Itemid=105
Number of Retries
0
Type
Transaction made
IPN Text
mc_gross=32.00&invoice=635&protection_eligibility=Ineligible&payer_id=SUJW85VBX62ZU&
tax=0.00&payment_date=06:54:12 Jul 31, 2017 PDT&payment_status=Completed&charset=UTF-8&
first_name=timo&mc_fee=1.45¬ify_version=3.8&custom=&payer_status=verified&
business=xxxxx@xxx@xxx&quantity=1&
verify_sign=An7H3kH8.DY56ocyWs9pEuVQUM5wA2seUWPPNlgayLnPGgf-xEVmm2cE&
payer_email=xxxxx@xxx@xxx&txn_id=7Y523873A2521025M&payment_type=instant&
last_name=xxx&receiver_email=xxxxx@xxx@xxx&payment_fee=&
receiver_id=JYQVF7NJL87Q8&txn_type=web_accept&item_name=&mc_currency=GBP&
item_number=&residence_country=DE&handling_amount=0.00&transaction_subject=&
payment_gross=&shipping=0.00&ipn_track_id=9e387cf973036

Thank you.

We'll give it a try and report back on results...

I'm afraid to say that the problem is still occurring - the first order placed after the update was applied failed with the same error message.

The order is now on the system as "Created" and not "confirmed".

Thanks, we'll try this and report back to you when we have some results from live transactions.

In the meantime, PayPal is reporting that the IPNs for the transactions that have generated the error messages are OK. We provided them with several transaction IDs and they came back with the following:

"I can verify the IPN for the transaction you mention is fine"

The fuller details are in an earlier post on this ticket.

More news when we have it...

No good, I'm sorry to say: all the orders that have come into the system since the patch was applied are showing as "Created" and not "completed", so the problem still stands.

Hi nicolas,

I'm not clear on your reponse. PayPal have been telling us that the IP{Ns are fine for the errored transactions. My understanding of hte issue was that the IPN was coming back from PP as OK, but that HS was not processing it correctly, leaving orders as "created" and not "confirmed". The error email quoted at the beginning of the thread is being generated by HS, not PP.

From PP techs: "I understand you are having issues with the IPNs. Actually when I see your logs I do not see any failed attempt to send you an IPN. All of them are marked as received by your server."
and
"I can verify the IPN for the transaction you mention is fine."

However, HS is not completing the order fully, sticking at "created". Are you saying that the IPN text I quoted from PP earlier in the thread indicates that the IPN is not OK? Apologies - this is beyond my pay grade!

As an aside, the shop is currently configured with the payment page being PP's own, hosted on their own servers. On that basis we'd not considered it necessary to set up https on the server as all the transactions were being processed off-site. However, is it possible that the revisions to IPN Verification Postback to HTTPS outlined by PP here ( www.paypal-knowledge.com/infocenter/inde...916&viewlocale=en_US ) might be at the root of the issue?

Ni nicolas,

This is the IPN text that we got back from PP for one of the problem transactions. From what I can see, it doesn't include the text you mention.

IPN Text

mc_gross=32.00&invoice=635&protection_eligibility=Ineligible&payer_id=SUJW85VBX62ZU&
tax=0.00&payment_date=06:54:12 Jul 31, 2017 PDT&payment_status=Completed&charset=UTF-8&
first_name=timo&mc_fee=1.45&notify_version=3.8&custom=&payer_status=verified&
business=xxxxx@xxx@xxx&quantity=1&
verify_sign=An7H3kH8.DY56ocyWs9pEuVQUM5wA2seUWPPNlgayLnPGgf-xEVmm2cE&
payer_email=xxxxx@xxx@xxx&txn_id=7Y523873A2521025M&payment_type=instant&
last_name=xxx&receiver_email=xxxxx@xxx@xxx&payment_fee=&
receiver_id=JYQVF7NJL87Q8&txn_type=web_accept&item_name=&mc_currency=GBP&
item_number=&residence_country=DE&handling_amount=0.00&transaction_subject=&
payment_gross=&shipping=0.00&ipn_track_id=9e387cf973036