Where in your case you wanted to round it to nearest upper value which is 5.50 but the rounding rule which you are placing in wrong place
Go to table V_T559L and find out your Quota type with correct groupings then select it click on CTRL+SHIFT+ F2 and then click on accrual entitlement where you will find rounding rule option enter there and check.
Regards
Venkatesh