Cart table growing in size - 400,000 records!

  • Posts: 298
  • Thank you received: 5
  • Hikashop Business
2 months 1 week ago #365301

-- HikaShop version -- : 5.1.2
-- Joomla version -- : 5.2
-- PHP version -- : 8.2

Looking at a couple of sites realised the cart table is growing to a ridiculous size.
The cart retaining period is setup.

Attachments:

Please Log in or Create an account to join the conversation.

  • Posts: 83487
  • Thank you received: 13510
  • MODERATOR
2 months 1 week ago #365302

Hi,

You want to reduce the frequency as much as possible. That way, the old records will be deleted more often.
With a 1 day delay, only 50 carts will be deleted each day. If, for example, you have 2000 carts being created on the website every day, with 50 old carts being deleted every day, that's 1950 carts piling up every day. And that will quickly grow to 400K records.

Please Log in or Create an account to join the conversation.

  • Posts: 298
  • Thank you received: 5
  • Hikashop Business
2 months 1 week ago #365305

Thanks, just been looking at hikashop_cleanCart().

I see the default for cart_batch_removal_quantity is defaults to 50 and I cannot see anyway in Hikashop code for changing that without adding some coding. That's no problem but I wonder if core Hikashop could be improved.

i.e. Ability to remove the quantity limit (i.e. set it to 0) and purge all carts older than, say, 7 days.

Please Log in or Create an account to join the conversation.

  • Posts: 83487
  • Thank you received: 13510
  • MODERATOR
2 months 1 week ago #365308

Hi,

That's a hidden option.
If you want to change the value, you can do so by adding an entry to the hikashop_config table of your database with the namekey cart_batch_removal_quantity and the value you want via your PHPMyAdmin.
So no code is needed.

Please Log in or Create an account to join the conversation.

  • Posts: 298
  • Thank you received: 5
  • Hikashop Business
2 months 2 days ago #365358

Looking at admin/customers/cart page:
..../administrator/index.php?option=com_hikashop&ctrl=cart&cart_type=cart&limitstart=40&limit=20

Then cross referencing the IP addresses with the server access log I see that all these carts are being created by robots.

216.244.66.231 - - [25/Jan/2025:06:40:32 +0000] "GET ... (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; This email address is being protected from spambots. You need JavaScript enabled to view it.)"
54.36.149.61 - - [25/Jan/2025:06:40:36 +0000] "GET ... (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"
85.208.96.197 - - [25/Jan/2025:06:40:44 +0000] "GET ... (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
74.125.212.193 - - [25/Jan/2025:06:40:45 +0000] "GET ... (X11; Linux x86_64; Storebot-Google/1.0)

Should robots be allowed to add things to the cart, is it necessary for search engine crawlers?

In helper.php there is a function hikashop_checkRobots() which does not appear to be being used anywhere.

There is this Joomla function which does a search through a massive list (incomplete) of known robots.
Joomla\CMS\Environment\Browser\isRobot()

Are there better ways of handling this?

I suppose a plugin which handles the onBeforeCartSave event and calls isRobot() might be one way of preventing the #__hikashop_cart table from filling up with robot visits.

Please Log in or Create an account to join the conversation.

  • Posts: 83487
  • Thank you received: 13510
  • MODERATOR
2 months 2 days ago #365360

Hi,

No, it is not necessary to allow bots the creation of carts.
The first thing to do would be to use the robots.txt file of your website to forbid access to the add to cart URLs for bots.

Indeed, having a small plugin checking the isRobot status with the onBeforeCartSave event would be a more agressive solution.

Please Log in or Create an account to join the conversation.

  • Posts: 526
  • Thank you received: 9
  • Hikashop Business
1 week 6 days ago #366198

Hi, is this the code to add to robots.txt file?

Disallow: /index.php/customerzone

Rgds

Please Log in or Create an account to join the conversation.

  • Posts: 83487
  • Thank you received: 13510
  • MODERATOR
1 week 6 days ago #366201

Hi,

What to add depends on your situation.
What are the URLs used for the add to cart buttons on your website ?
Based on this, you can add disallow rules.

You can also disallow the whole website for some user-agents. For example, if you want to disallow the bots from SemrushBot, you can add:

User-agent: SemrushBot
Disallow: /

Please Log in or Create an account to join the conversation.

  • Posts: 526
  • Thank you received: 9
  • Hikashop Business
1 week 6 days ago #366209

Same situatuon as OP, 1,000's of carts with individual items being created daily.
I disallow mulitple carts but that does not work. only way I can control is is, to set to empty after 2 days but this annoys my regual customers

Don't understand this question? What are the URLs used for the add to cart buttons on your website ? add to cart is the usual method from the popup on the product pages

No idea who's bots are doing this, how do I find that out?

Rgds

Please Log in or Create an account to join the conversation.

  • Posts: 298
  • Thank you received: 5
  • Hikashop Business
1 week 6 days ago #366213

developers.google.com/search/docs/crawli...%20out%20of%20Google .

Please Log in or Create an account to join the conversation.

  • Posts: 83487
  • Thank you received: 13510
  • MODERATOR
1 week 6 days ago #366216

Hi,

First, I had not seen that you were not the same person, sorry.
Also, your question in your initial message was about the robots.txt
But I don't think changing the robots.txt will help you, if your problem is that you have too many carts in the database.
As I said in my first message on this thread, the first thing to do is to reduce the "Frequency of checks for the cart" setting to be as low as possible.
Reducing the retaining period is not a good strategy. As you said, it will annoy your customers. Also, it doesn't help reducing the amount of carts. It's the check frequency you want to reduce.
Also, you can simultaneously increase the cart_batch_removal_quantity as I was explaining in www.hikashop.com/forum/checkout/909044-c...-records.html#365308

Just doing these 2 things should be way enough in most cases.

Regarding the robots.txt I can't say exactly what needs to be done without analyzing the website. Please provide the URL of the shop and I can tell you what you could add to help.

Please Log in or Create an account to join the conversation.

Time to create page: 0.095 seconds
Powered by Kunena Forum