Yeah, I know that's an option. Googling it there seems to be a fairly even split between those that say that drilling the aluminium weakens the bars, and to file the lugs off, and those that say it's better to drill the holes.
Personally, I just hate it when people break stuff off of bikes. It's why I rarely buy used. I just don't like having bits snapped or cut off.