It began in 1269 when the Earl of Atholl returned from Crusade to discover his neighbor, John Cumming (likely assuming the earl wouldn't be back), had built a large tower house on his land. The earl took his land, and the tower house, back from Cumming, and Cumming's Tower still stands as the oldest--and tallest--part of the castle.
Just in case you were interested.
Blair Castle today boasts some 85 rooms, of which 30 are open to the public.
A few interesting facts whilst you peruse the beautiful gardens of Blair Castle:
The Duke of Atholl is the only duke in Scotland allowed to have his own army- though it is today purely ceremonial. Queen Victoria stayed at the castle in 1844. As a mark of loyalty, she granted the Duke the Queen's Colours, and founded the Atholl Highlanders as a private army.
The Duke's family was divided in the Jacobite Uprisings of both 1715 and 1745. In 1745, his oldest son fought on the Jacobite side and sought refuge in France after the Battle of Culloden. The younger son fought for the Crown, and thence became Duke. Years later, the older son's son and the younger son's daughter married, thus keeping the 'rightful heir' (though he was charged with treason) in direct line for the inheritance.