The church was built on the north bank of the Nemunas (Nieman) River around 1400 for Franciscan monks and foreign merchants. It was ordered and funded by the Grand Duke Vytautas as a commendation to the Blessed Virgin Mary for saving his life after a major defeat in the Battle of the Vorskla River.
Being quite close to the river the church has suffered many times from spring floods. In 1812 it was burned by French troops. The Franciscan monastery was closed by Russian administration after the November Uprising. In 1845 the Catholic Church was closed and later was rebuilt and reopened as an Orthodox church. From 1903 it served as military barracks. In 1915 Germans organized a warehouse here. The building returned to the Catholic Church in 1919.
The church was constructed in Gothic style and is an example of the Lithuanian Brick Gothic architecture. The church layout of the Latin cross is unique in the Lithuanian Gothic. The bell tower was added later and has probably been used to guide ships navigating the Nemunas River. Lithuanian writer Juozas Tumas-Vaižgantas (1869–1933) is buried in the crypt of the church.
