Hi,
If you restrict a coupon to a category, it will apply to the whole cart is at least a product of that category is in the cart.
The percentage value of a coupon will be calculated based on the total amount of the cart, and if your coupon is limited to a category, on the total amount of the products of that category in the cart.
So if you have product A costing 20€ and product B costing 10€ in the cart, and they are in the same category, if you apply a 50% percentage coupon restricted to that same category, the coupon value will be 50% of 20€ + 50% of 10€ so 15€.
There is no "maximum number of product" setting in the coupons so the only thing you can do to avoid the value increasing based on the number of products in the category in the cart, is to not use the "percentage" field, and instead to use the "flat amount" field so that you enter a fixed value.
If you want to do such things, I'm afraid you'll have to use AWOCoupon:
awodev.com/products/joomla/awocoupon