I think you meant 2004-2006 is a straight swap and 2007+ needs more parts?

MK1 XT660X (2004-2007) has a steel swingarm. I've got one, 2005 XT660X, and the swingarm seems to be the same size. I measured the spacing of the chain guard bolts and the length of the rear of the swingarm to the welds that are almost in the front, they were about the same.

What I'm not sure about is the tabs that hold the brake line and width. On the XT660R the brake line is routed on the "inside" face of the swingarm, on the XT660X it's on the "outside". Then there is a 130 vs 160 rear tire.