This plantation was once part of Boochawee Hall, owned by Governor James Moore (d. 1706). Moore left 615 acres to his daughter Rebecca, who married Thomas Barker (d. 1715) in 1709. Barker, who planted inland rice here, served one term in the Commons House of Assembly. In 1715, at the outset of the Yemassee War, Barker raised and commanded a company defending Goose Creek. That spring Capt. Barker and 26 of his men were killed in a Yemassee ambush.
Rebecca Moore Barker married planter William Dry (d. 1740), who served six terms in the Commons House of Assembly and was its Speaker 1728-29. In 1785 William Loughton Smith (1758-1812) acquired the plantation; he was a state representative and later U.S. Congressman and U.S. minister to Portugal. Button Hall was owned by two of Smith's grandsons after the Civil War, when it was subdivided and sold or rented to freedmen for small farms.