Dumbledore explicitly pointed out that the sword has been in the Headmaster's office during the time period when Voldemort was looking for objects to turn into horcruxes (horcruces?). The hat has been in the Headmaster's office since the the founders' days.

I guess we still don't know where exactly the sword was before Harry pulled it out of the hat. The sword could have been in the office if the hat was either a gateway to the sword stored in the office or the actual container of the sword. I'm tempted to go back and read the descriptions of the headmaster's office from PS and CoS pre-climax, and see if Harry notices the sword.