This church was founded in 1866 by Rev. James Smalls, its pastor for many years. The congregation, which built its sanctuary here on land owned by the Gospel Harp Society, grew to more than one hundred members by 1903. In 1914 trustees S.B. Belin, Neptune Boyd, Seward Dunmore, Joseph Gibson, I.J. McCottrie, W.M. Salters and Samuel White, Jr., purchased this property from the trustees of the Gospel Harp Society.
The first church here, a frame building, was replaced by this brick sanctuary in 1920. Built during the pastorate of Rev. A.E. Tindal, it features elaborate stained glass windows. Mt. Olive was also one of several Georgetown churches hosting graduation exercises for Howard High School in the 1940s.