The construction of Richmond Castle (originally called “Riche Mount” or “The Strong Hill”) began in 1071 following the Norman Conquest of England. After the 1069 rebellion, William the Conquerer divided the territory of North Yorkshire among his followers. Alan Rufus of Brittany received Richmond, and he began building the castle as a base from which he could defend against any further rebellions. In 1158, King Henry II would seize control of Richmond, and it was he who would eventually complete the construction of Richmond Castle.

By the end of the end of the 14th century, the castle was no longer being used as a military fortress. It had fallen partially into disrepair, but an increase in tourism led to repairs in the 19th century. The best preserved part of the castle is the Keep, which is a tower that reaches to a staggering height of over 100 feet and was built in the 13th century. The archway at the base of the Keep was built in the 11th Century and is possibly the only remaining portion of the original entrance.

In 1855, Richmond Castle became the headquarters of the North Yorkshire Militia. From 1908-1910, it was the home of Robert Baden-Powell, who would later found the Boy Scouts, while he acted as the commander of the Norther Territorial Army. 

During World War I, it was the home of the Non-Combatant Corps, which consisted of those who refused to fight in the war, as well as a prison for protestors.  The prisoners held at Richmond Castle included the so-called “Richmond Sixteen,” who were known for their moral and absolutist stance on the war, something that caused a serious controversy in Parliament. They were originally sentenced to death for their refusal to fight, but this was later commuted to hard labor.