I mostly ride with them on as I carry more junk than needed, especially this time of year when it can be cold when you start, then get warm and you want to shed or add layers and change up gloves. I just got a tank bag so some of the things I stored in the saddlebags (eyeglass case for prescription sunglasses, etc) can go in there now, so I'll probably ride more without them now. I think the bike looks good either way, potato launcher and all.
I took an old helmet bag and put my rain gear, some extra gloves and some other stuff like that in there to help keep the saddlebag more organized. I have that and my tire repair kit in that bag, the other one is empty to store my helmet when I stop on the road.