At the time of the Commonwealth, church property was seized by the state and the site was acquired by Rt Hon Sir John Thurloe, Secretary of State to Oliver Cromwell. He cleared the remains of the Bishop's Palace and built a mansion on the site.
Probably designed by Peter Mills, one of the surveyors appointed to supervise the rebuilding of London after the Great Fire and architect of Thorpe Hall at Peterborough. However Thurloe did not live in it for long as at the Restoration in 1660 it reverted to the See of Ely and became once more the residence for occasional visits by the bishops. The Diocese had little use for the building and it was rented to local families.
The Southwell family of Wisbech occupied the building for over 100 years. In 1778/9 Edward Southwell was paying £30 a year in rent.
In 1793, the Diocese under Bishop York put the Castle Estate on the market, it was divided into several lots. It was purchased in it's entirety by Joseph Medworth for £2305. A local man of humble origins, his father was a ropemaker in the town. Joseph became a bricklayer in Bermondsey and prospered through building houses in ever expanding London. He developed plans for the area which were unacceptable to the local 'capital burgesses' and in a fit of pique promptly demolished the Mansion in which he had lived for 20 years, using the materials to build a Regency Villa within it's gardens in 1816. He also developed the present houses which form York Row, The Crescent, Union Place and Ely Place.