Precisely! I generally recommend people remove the guards and just stick them in the bags if they ever ride without the bags on. Only takes a few moments to remove and/or reinstall the guards anyway.
My solution of just never removing the bags is just oh so much easier!