Can I get form submissions by email only, without storing them in the admin?

Yes. In Contact Forms, click the gear icon on the form and open Email and Actions:

1. Switch on Admin Email. Click its settings icon, enter the address in Send admin notifications to and click Save.
2. Switch off Store Submission.

New submissions are then emailed but not saved, so they will not appear under the Submissions tab or in Contact Form Submissions. Submissions already stored stay there. Test the form first to make sure the email arrives, because nothing else is kept.