In 1849, Antonio Ramirez built a small adobe on the El Camino Real, near the site of the ancient Indian village, Holamna. As a main artery to California's gold fields, the El Camino Real brought many travelers past Ramirez's place. He transformed his adobe into an inn and stage stop and soon enjoyed a brisk business.
Eventually, the Inn became the nucleus of the town Jolon. Ramirez sold the building in 1871 and it changed hands three more times before it was purchased by Lt. George Dutton and Capt. Thomas Tidball in 1876. Dutton, who became the sole owner in 1878, added a second story of adobe and wood frame structures at either end. Jolon's first post office was located at the Inn, and Dutton added a store and saloon. The saloon boasted a huge fireplace, 6 or 8 poker tables, and an ornate bar which came around the Horn.
By 1876, Jolon had grown into a thriving town serving the Los Burros Mines, surrounding ranches, and many travelers. However, when the railroad came to King City in 1886 traffic followed the railroad, and the old Camino Real fell into disuse. Soon after, the Los Burros Mines gave out as well, and business at the Dutton died down.
William Randolph Hearst purchased the property in 1929 and removed the Dutton's outbuildings as well as many other Jolon structures. In 1940, the U.S. Army acquired the property, and the adobe was used as a bivouac area. Vandalism and exposure to the elements finally destroyed the building, leaving only an adobe ruin. The building was placed on the National Register of Historic Places in 1976