Have you removed the rubber insert in the bar ends that are the fixing points for the original bar end weights?..
If you have and you now have both ends off you could feed a piece of cable/wire etc through the bars with a rag tied to its end and sweep it out...at least that's what I did.