In 1865, the Bureau of Refugees, Freedmen, and Abandoned Lands began assistance to former slaves. While efforts at educating freed slaves were strongest in Georgia's larger cities and towns, State Superintendent of Freedmen's Schools G.L. Eberhardt recognized the need to expand to rural areas.
Two years later, Eberhardt rented for a school the Madison Baptist congregation's former church on Academy Street (which was also rented to their black brethren who had formed Calvary Baptist Church led by Rev. Allen Clark). By June 1867, the Bureau acquired a one-acre tract on Hill Street and purchased the rented building for relocation to the site.
Northern aid societies - the most significant among these for Central Georgia being the American Missionary Association - supplied financial support and teachers, mostly Northern white women. As President of the local education association, Rev. Clark worked to employ African-American teachers for Madison Freedmen's School, which operated here until the 1870s when Georgia's dual system of education began.