Can I stop customers from paying for a whole order with reward points?

Yes. Go to Plugins → Reward Points. In the Redeeming section you can set:

• Max % of order coverable, for example 20, so points can only pay part of an order
• Minimum points to redeem and Max points per order
• Minimum order total to redeem
• Points expire after (days), so old balances don't build up forever

Set Value per point carefully too, since it decides how much each point is worth. Click Save settings when you're done.