Hi,
First you want to turn off the "force size" setting and you want to specify a height but no width in the thumbnails size setting of your menu item.
That way, each image will have the same height, but it width will vary based on the ratio of the image, like you need them to be.
However, each row will still have the same number of products. To change that, you'll have to add custom CSS in order to have them arrange dynamically like you need them.
With the appropriate custom CSS, you should be able to do what you have in your first screenshot.
However, with just the options, it's not possible.