Hi,
If you want to have the infinite scroll and have it triggered only when the user manually click on the "load more" button can be done with a small view override. For that, go in the Display>Views and edit the product / listing_div view file and remove that code:
window.Oby.addEvent(window, 'scroll', function() {
window.localPage.checkInfiniteScroll('<?php echo $mainDivName; ?>');
});
window.Oby.addEvent(window, 'resize', function() {
window.localPage.checkInfiniteScroll('<?php echo $mainDivName; ?>');
});