The earliest fort on this site was built in 1794. Great Britain's navy had been seizing American merchant ships and forcing their sailors into service on British vessels. Congress decided that the nation's most important harbors should be defended in case war broke between Great Britain and the United States. Charles Vincent, a French engineer, was hired to construct defenses in New York Harbor. He chose tiny Oyster Island (as Ellis Island was then called), known only for its oyster banks, shad fishing and as the location for a battery of eight cannons. Tensions rose again between the two countries when a British frigate fired on the USS Chesapeake in 1807, leading President Thomas Jefferson to further improve the nation's defenses and rebuilt many of its early forts.
Colonel Jonathan Williams, a grandnephew of Benjamin Franklin, oversaw the construction of the brick and stone structure, the remains of which you see here. The new fort was completed only months before the outbreak of the War of 1812. During this war British prisoners were housed on Ellis Island, but the fort was never attacked. In 1814, the fort was named after Colonel James Gibson, a 33-year-old officer killed at the Battle of Lake Erie during the War of 1812.