Nothing to worry about. More often than not, this sort of clunk is caused by the pads moving slightly in the calipers.
It is particularly common if you wheel your bike backwards to get it out of the shed/garage and use the front brake. The pads shift forward on the pin/retainer groove. As soon as you set off on your ride and apply the brake, the pads clunk backward the other way into their "normal" position.
Often this clunking is more pronounced with old/worn pads, or worn pad retaining pins. A bit of grease on the retaining pin can damp the noise down too.
Just to be sure though, check all the disc and caliper bolts for the correct torque.
Oh, BTW - Welcome to XT660.com David!