At the turn of the 19th Century, the Highcliffe Castle estate extended from the castle eastward to the Chewton Bunny and was owned by the Stuart Wortley family, descendants of Lord Bute.
Greystones House, within the Castle estate, was built between 1911 and 1913, at a cost of £4,600 then given by Major General Stuart Wortley to his sister Beatrice and her husband Captain Henry Denison as a seaside residence.
The Architect of Greystones House, Edward Schroder Prior was articled to Norman Shaw an early member of the Arts and Craft movement which influenced the style and design of a building for the 20th Century. Prior amalgamated many different materials and shapes to achieve a contemporary style. The two storey building constructed in Purbeck stone has many interesting features such as external circular brick columns, Tuscan columns in the hall, a domed roof, window seats, fireplaces, oak doors and so on.
The Dennison family lived at Greystones House until 1925 when it was purchased by Mr and Mrs George Kemp who enjoyed residence until 1940. It was then requisitioned by the War Office and used by the garrison manning the beach defences.
In 1944, the House was bought by Mrs Ruck-Keene who sold it to the Christchurch Housing Society in 1947. The land to the south of the building was bought by the Christchurch Borough Council. The Housing Society commenced a flat building programme in the grounds in 1960 and converted the house for use as a Nursing Home until 1973. When different regulations came into force, it became uneconomic to use and it was eventually left empty.
The Housing Society hoped to demolish the House to make way for more flats, but was prevented from doing so by the timely intervention of an American graduate, Lynne Walker, who was studying the work of the architect Edward Schroder Prior. Appalled by the state of disrepair Greystones House had been left in, she appealed to the Victorian Society to prevent the house being demolished. It was then left uncared for until 1980, until an approach was made by the Highcliffe Community Association. It subsequently became their headquarters.