Will deactivating Price Toggling also remove sale prices I set by hand?

It can. Deactivate Toggling Price clears the Sale price on every product in the chosen scope that has one, whether or not that price came from toggling.

To protect your hand-set sale prices:

1. On Plugins → Price Toggling, set Apply To to cover only the sale products. You can also set Match to Exclude — all except these and exclude the products you want to keep.
2. Click Preview. The table lists each product with its Before and After price.
3. If a product you want to keep is listed, adjust the scope before you click Deactivate Toggling Price.

If one gets cleared by mistake, re-enter its Sale price on the product page.