The forested land was patented in 1726 by Daniel Carroll of Upper Marlboro, who called it "The Mistake". Earliest part of house was built c. 1760 by Nathan and Sophia Dorsey. In 1786 Edward Dorsey sold 650 acres, including "The Mistake" to Colonel John Eager Howard, Revolutionary hero and later governor. When his son, George Howard, married prudence Gough Ridgely of "Hampton" in 1811, "The Mistake", then totaling 1,313 acres, became a wedding present to the couple. They changed the name to "Waverly" after a popular 1814 novel by Sir Walter Scott.