I have *NEVER* had that problem, and also have a Sargent seat. But, then, I wear functional/quality underwear...
+1
You have a better chance of crushing the family jewels if you try to lean forward on a flat seat than on a sloped seat because now you are having to roll up on them. Plus, a flat seat puts more strain on your lower back when you are leaning forward for the bars.
Of course if you are putting risers on the bars then you are changing the bike to become more tour than sport. The old Voyager XII was a great sporty touring bike. The concours is a great sport touring bike. I have done 450 mile days with the stock setup and the only real complaint is that the stock foam compresses too much when on a long ride. Since I abhor riding on the interstate, and don't ride at night in unknown areas, I won't be seeing more miles than that in a day.
The beauty of this bike is that you can change it to suit your personal preferences and wants. Seems like some want to make it like a gold wing and some want to make it like a ZX14. IMO it will never be either so why not just buy one of those other bikes?