What happens if I reject a customer's bank transfer payment proof?

Rejecting a proof is treated as a declined payment: the order is cancelled, reserved stock is released, and any wallet, reward points or coupon used on it are given back. The reason you type is shown to the customer. So if the proof is just unclear, contact the customer first and ask for a better copy. When you are happy, open the order and click Approve (mark paid) in the Payment proof panel, which moves the order to Processing.