Established in 1892, the community of Olivia was named for Olivia Haterius, wife of the Rev. Carl J.E. Haterius, a Swedish Lutheran minister who bought land in the area and advertised a new settlement to other Swedish immigrants in the Midwest. Twelve Swedish families responded to the initial advertisement and moved to Olivia with others following in a few years By 1900 a number of non-Swedish families had settled here, as well.
The first public building in Olivia was a one-room schoolhouse where children were taught during the day and parents attended classes at night to learn English. The Eden Lutheran Church held services in the schoolhouse until 1910, when a sanctuary was built.
John Lind built the first store in the community on Carancahua Bay in 1894. The post office was housed in Lind's store and mail was delivered once a week. Lind moved his store to the center of the townsite in 1900. In 1906 Edward Wilson, another Swedish immigrant, bought the Olivia store and was appointed postmaster. Other businesses included a hotel, doctor's office, grocery store, blacksmith shop, and cotton gin. Since 1900 the thriving community has changed and grown, but retains its proud Swedish heritage.