The prison was built from 1429-1434. The fact that it was made out of dark stone and the grime on the walls consequently earned it the name "Zwartstoren". The tower, however, was more a castle or a reinforced fortress where the city council of Grijzestad imprisoned its bandits or criminals. However, in the first centuries of its existence, Zwartstoren was more used where political or noble people, mostly opponents and enemies of the state, King or city. A number of well-known people were imprisoned here, including Founding Fathers Fredrick van der Haas and Lodewijk Steen. In 1663 the prison was burnt down and destroyed by rebels during the Brunanter Civil War, since it symbolized the King's tyranny.
During the 20th century, a lot of excavations were held and remnants of foundations and walls were found. The most famous archeological site is dating from 1974 when a team of Grijzestad University employees and students found the whole courtyard including several items.