I really see no advantage to riding with no bags. I doubt there is increased fuel economy (someone that has noticed a difference chime in) and the bags are not the widest part of the bike - the mirrors are, even when they are folded in. I also use my bags almost every ride.
After two parking lot drops, one with GSG sliders and one without, I can tell you that without the bags on, I suspect the rest of my bike would have been much more heavily damaged that it was.