This site was in the original Crown grant of 200 acres to Colonel Thomas Willoughby in 1636. Located on Freemason Street, so called because the Norfolk Royal Exchange Lodge of Masons erected the "Mason's Hall" on this site in 1764 as America's first Masonic Hall. The building subsequently was destroyed in the bombardment and burning of Norfolk in 1776. William Willoughby (1758-1800), a descendant of Thomas, purchased the property from the Masons in 1794 and built this Federal style town house. An active Mason, William Willoughby was both a successful merchant and in the contracting business. He served the community as a member of the Norfolk Borough Common Council. The Whilloughbys and their descendants, the Sharpe and Baylor families, occupied the house until 1890 when it was sold by Dr. John Baylor. By the 1960s the property had deteriorated. It was acquired by the Norfolk Historic Foundation, restored, and eventually donated to the City of Norfolk as a historic house museum.