Can I ask customers to upload a screenshot of their bank transfer instead of a reference number?

Yes. Go to Settings → Store Settings → Payment Gateway and click Configure on Bank Transfer. Under Payment proof, set What the customer submits to:

• Transaction reference only
• Screenshot only
• Reference + screenshot (both required)
• Reference or screenshot (either)
• No proof (merchant confirms manually)

Use Where they can submit to choose the confirmation page, My Orders, or both. Click Save Changes. Approving a proof marks the order paid; rejecting it cancels the order.