The castle became the property of the new canton of Vaud following the Revolution, which drove out the Bernese in 1798, and was sold to the commune of Aigle in 1804. It then housed the cantonal district prisons, which were essentially located in the circular towers, as in the Bernese era. Considered to be the most inhumane and coldest in the canton, and offering no guarantee against escape, they were moved and refurbished in the square tower between 1832 and 1834 under the orders of the canton's architect Henri Perregaux. Many prisoners, both men and women, were locked up there until 1972. The castle also housed the hospital, the asylum for the poor of the commune, for whom several dwellings were created, first in the square tower, then in the central building. They remained there until about 1916. The district court moved to the castle in 1825 and remained there until 1964.
In 1900, the castle was listed as a historical monument in the canton of Vaud. In 1909, a complete survey was carried out by Otto Schmid, architect of Chillon Castle. Aware of the building's historical interest, the municipal authorities of Aigle thought of turning it into a museum in the 1920s. However, it was not until 1943 that the architect Schmid drew up a general restoration project, which was eventually not carried out.
After the prisons' departure in 1972, a great campaign of restoration and transformations allows the castle to become a site of culture and conviviality.
In the middle of the 1990s, the castle's barn was restored and transformed into the Maison de la Dîme. Since 1996, it has been home to the restaurant La Pinte du Paradis and a reception hall.
A new maintenance and restoration campaign of the castle is conducted in the early 21st century. In 2010, the new Vine, Wine and Winelabel Museum is inaugurated and presents a large-scale permanent exhibition.