In 1873 Entrepreneur John Harford built a wharf at the westerly end of San Luis Bay to accommodate growing cargo shipping and passenger travel on the Central Coast. The Port Harford pier extended 2,000 feet into the bay and provided horse-drawn rail access from the narrow gauge Pacific Coast Railroad terminus at Avila Beach to docked steamships. In 1876 the three-story Hotel Marre was built at the foot of the pier, which was destroyed by a tidal wave in 1878, but soon rebuilt. In the 1880's stream engines appeared on the pier's rail line.
The Port San Luis breakwater was complete by 1913 to create safe anchorage at the port, though in the following decades a weakened economy and the shifting of shipping to land routes reduced commerce enough that the railroad stopped operating in 1942. By the 1950's the port had primarily become a fishing, boating and tourist destination and in 1954 San Luis Obispo voters established a Harbor District for Port San Luis.
In 1970 a restaurant and bar were first developed in the old railroad building on the pier. After the 1973 replacement of the storm-damaged railroad bridge over San Luis Creek, facilities on and around the pier grew to include additional restaurants, fish markets, and a variety of services for fishing and recreational boating managed by the Port San Luis Harbor District.