I have just tried removing mine as well.
See what you mean.
I couldnt budge mine either!
Tried vice grips and big hammer - nothing!
I am just going to leave them in as I have a new set of renthals to put on. So will use a raw plug or sleeve anchor to put the old bar ends and gaurds back on.
Sorry I couldnt help.
There might be an old thread on the subject.