When on holiday I always have a spare key of both the bike and panniers. I usually exchange spare keys with my girlfriend so she can keep mine safe and I hers.
But I never take the red key. You only need it to register new keys on the bike.