A group of residents are campaigning against plans for a construction compound on playing fields in the Isle of Man capital.