Hi,
The "round prices during calculations" option is to choose whether to round the prices in between the different calculations (between adding taxes to the unit price and the multiplication of the taxed unit price with the quantity, between the sub total calculation and the coupon calculation, etc). This bases itself on the "rounding increment", the "International fractional digits" and "National fractional digits" settings of the currency. So just changing the "round prices during calculations" option won't change the number of decimals of prices, that's normal.
The "rounding increment" is to be able to support odd roundings like rounding every 0.25 cents (you would enter 0.25 in the input).
I would recommend leaving that setting empty in your case and changing the "International fractional digits" and "National fractional digits" settings to 0 instead of 2 decimals.