CSV Import Size Limitations

-- url of the page with the problem -- : drc144k.com/
-- HikaShop version -- : 2.2.2
-- Joomla version -- : Tried on 2.5.14 - 2.5.16
-- PHP version -- : Tried on 5.2.x thru 5.4.x ... same problem.
-- Browser(s) name and version -- : Firefox, Chrome, Opera
-- Error-message(debug-mod must be tuned on) -- : PHP Fatal -Allowed memory size of 536870912 bytes exhausted or eventually execution time of 600 seconds exceeded.

We are using the CSV import feature thru cron and we are currently at 40,000 products/variants. We either run into the timeout or the memory issue eventually.

We duplicated the Google Products plugin for each product type we have (t-shirt, jacket, cup, etc) so we could make smaller CSV files. Each plugin we have duplicated and reconfigured works well with the smaller files. This also works better with the Characteristics which are named the same, but have different values. However, the files still eventually grow to being over 16MB which is causing problems. We could continue to make things work by duplicating the plugin, but there has to be a better way, maybe even something we are overlooking.

Is there a way to get the Import plugin to process all CSV files in a certain directory (ex. media/com_hikashop/import)? We could then create as many smaller .CSV files to be processed by cron and it would process them quicker than the 14-16MB files we have now for each category of products.

Thanks.

Thanks for the reply. We just finished up code that will:
- create the csv files when the size will be over 1MB (or whatever size we set as the threshold
- automatically generates a plugin in joomla for the new csv file
- sets the cron "last run" time to a future date (this part doesn't seem to work though).

So now my questions are:
1. Does setting the cron last run date in the plugin XML do anything? It appears ALL the plugins run against the CSV files, or NONE will run. We thought setting the cron last run times at specific intervals would ensure the load on the server is never to heavy performing imports of products.

2. We are now at 1.3 million Products/Variants. The Export breaks pretty quickly with out of memory. Do you know at what number of products/variants will we see our site start to slow down? We can live without the export of the products since we can still backup our entire site with WHM/cpanel.

Thanks.