St Helier House will be sold to a private developer.
Parishioners have narrowly voted through the £2.9 million pound deal for the former care home, despite opposition from the Housing Minister.
Senator Sam Mezec had urged ratepayers to block the bid amid concerns the site might be over-developed and the homes sold to foreign buy-to-let-investors.
Town Constable Simon Crowcroft called it 'paranoia'.
"People have assumed on social media that there is going to be tiny, luxury flats created on the site, and that is to completely ignore the fact that we have a planning policy, we have an application process, and I don't believe that Planning would allow something to be built on the site that would produce cramped accommodation."
The vote taken at last night's Parish Assembly was 48 in favour, 43 against of the offer from property firm Columbia Group.
But a late change to the proposal demands that the parish negotiates a betterment clause to restrict the new owner's future profits from the site.
St Helier House closed at the end of 2017 after it was found to have the same cladding as Grenfell Tower.