Connecticut mariner and wrecker Benjamin Sawyer built the first house on this property by 1844. From 1888-1890, much of Key West's port business took place in Sawyer's home, until the completion of the federal Custom House. Key West native and Deputy Custom Collector William Lowe Delaney (1863-1917) acquired the property from Sawyer's widow and built this ornate Queen Anne Revival residence by 1906. The house featured a two-story verandah, pavilion tower, and an asymmetrical floor plan. Following Delaney's death, a circuit court ruling in 1918 transferred ownership of the mortgaged property to Lavinia Artolozaga for $3,551. The property was sold again in 1925 for $7,000 to Romanian immigrant Theodore Holtsberg (1857-1928), one of the island's first Jewish residents. He was an entrepreneur whose fashion store was a local landmark, and was an active member in the island's Jewish community. The historic Delaney/Holtsberg residence was listed on the National Register of Historic Places in 1971 as a contributing building in the Key West Historic District. In addition, it was selected as one of the thirty-eight architecturally significant Key West structures by the Milo Smith Survey of Historic Buildings in 1974.