If I set a commission for a vendor and for a category, which rate is used?

The category rate wins. In Vendors → Commission, the order of priority is: category, then vendor, then global.

• Global default (admin commission %) applies when nothing else matches.
• Per-vendor overrides replace the global rate for that vendor.
• Per-category overrides replace both for products in that category.

So if you want a vendor's special rate to apply everywhere, don't add a category override for the categories they sell in.