Mine sometimes acts similar to the way you're describing. I find that I have to hold the set button for a second or two rather than just push it. I think it is just the cruise control taking up all the slack in the cable. There is quite a bit of movement available to the cable, and the CC doesn't make really quick adjustments, as these would make the normal actuation of it quite jerky.
If I just push the button and let go of the throttle, I find that I'll sometimes drop 10mph or more before the cruise starts accelerating. But a one or two second push seems to do it. Then a tap on accel or decel will gain or drop 1mph or so.