In the middle of Sydney Harbour, in a tiny windswept cove in Middle Harbour, unknown to the residents around sit seven ramshackle huts built during the period 1923 to 1963.