Charles A. Hartfield purchased the lot on this site in 1881. A noted area cook, he quickly established "Charley's Restaurant," which included a bakery and boardinghouse. Hartfield was so successful that he planned an elegant rock structure in which to house his business. Construction began in March 1884 amid a flurry of development in the area. Scottish stonemason Patrick McDonnell, who was responsible for much of the stonework of the new courthouse, was foreman of the Hartfield worksite. The project's scope proved too grand for Hartfield's finances, however, and in September 1884 he sold the building to J.C. Lynch. Financially ruined, Hartfield was found dead within the year.
Lynch sold his building in 1885 to three Albany businessmen: Max Blach, N.H. Burns and Sam Webb. Charles Hartfield's widow, Lettie Hartfield, joined them as an equal partner and the group completed the structure, probably using Charles Hartfield's original plans. The building was occupied over time by such businesses as a grocery, a general merchandise store, a bowling alley and an auto repair shop. The Albany Masonic Lodge began meeting in the structure as early as 1893, and it became know as "The Masonic Building" to local residents. Real estate magnate L.H. Hill purchased the building in 1925, and the Masonic Lodge bought it in 1940.
Damage from nesting bats caused part of the building's limestone front to tumble into the street in 1954. The Masons took down the facade and rebuilt it with yellow brick. Sold again in 1996, the building was renovated and its facade was reconstructed to reflect its former grandeur as one of Albany's finest early structures.