After the Citadel was built, Québec's role as a fortress was at its peak. The defence works, buildings, and army grounds occupied one quarter of the city's entire surface area. Military property was unmistakably more concentrated in Upper Town (42 per cent) and dominated by the Citadel. The garrison, which included between 1000 and 1500 soldiers, made up more than a quarter of the population in the area. The troops were stationed around the enceinte, in the Citadel's barracks at Artillery Park and at the Saint-Louis Bastion. They were also placed in the old Jesuits' College which was transformed into barracks and located in the heart of the city. The machinery of the military became more and more important in Québec during the 19th century; the period was a time of great economic development and population growth. However, military requirements often clashed with the interests of the city's inhabitants. The fortifications were incessantly viewed as obstacles to the city's expansion, economic growth, and traffic. During the 19th century, the city and fortifications did not go hand in hand. The stronghold was pitted more and more against the city.

While visiting Québec in 1850, the American naturalist and philosopher Henry David Thoreau wrote:

"A fortified town is like a man cased in the heavy armor of antiquity with a horse-load of broadswords and small arms slung to him, endeavoring to go about his business. "

Because the era of the fortified city drew to a close at the middle of the century, the grip the military had on the city was to last only a few more years. Substantial improvements in artillery and communications caused the military to withdraw from the heart of the city. The detached forts, built on the south shore in Lévis in 1865, and the departure of the British garrison in 1871 indicated that the enceinte was to be abandoned and the old military gates to be demolished.

Québec's transition from fortified city to historic monument and the struggle between the forces of progress and conservation had begun.