I have similar experiences. My prefered option is a pair of hard cases and a LOMO dry holdall on top. This seems to give the best balance between security and flexibility. The only advantage of a top case, over hard panniers is you can store a lid in it.

I also found that a top case made the bike a bit wobbly at speed, particularly when windy. Panniers don't seem to affect handling at all either loaded or unloaded.

__________________