The manor house was built on a marble foundation and was strongly influenced by American colonial architecture, tempered by the Georgian style. Like other large residences in fashionable Lenox, the estate was, in effect, a private hotel designed as much for the comfort and amusement of friends as for the owners. The Sloanes used the property from the spring through late October, and their staff numbered approximately 125 employees. In high season, a daily schedule of activities was posted in the Manor Hall and might include a lawn tennis tournament, horseback riding, a picnic in a field with sterling silver candelabras, and an after-dinner concert in the music room staged under crystal sconces and intricate carvings of cherubs. At the end of the season, Mrs. Sloane gave each staff person gold coins so they could have financial means to weather the winter.

The property has always been highly picturesque, enhanced by the rolling landscape of the Berkshires Mountains. Its grounds were designed by Frederick Law Olmsted, the “father of landscape architecture” and the landscape architect of New York’s Central Park and the U.S. Capital. The grounds include woods, open fields, foliage, and framed vistas culminating with frontage on Lily Pond. 

It took 13 months, some 70 designs for Olmsted, to plan the property’s gardens. The estate housed 32 greenhouses, including one greenhouse sized to accommodate full-grown palm trees. The greenhouse supplied flowers and fruits for the estate and the Sloane’s New York City residence and, at one time, was the largest acreage of private greenhouses in the world. The Sloanes used the property from the spring through late October. The staff numbered about 125. At the end of the season, Mrs. Sloane gave each staff person gold coins so they could have financial means to weather the winter. The estate was reported to have 4,000 chickens, produce 40,000 pounds of hay yearly, and have 25 coaches and horse teams. While the setting was somewhat rural, it is a one-mile walk to the downtown/retail area of Lenox.

In 1919, the “Elm Court Talks” were held on the massive hewn marble porch of the Manor House and the talks ultimately led to the creation of the Treaty of Versailles and the League of Nations. These agreements ended World War 1 with peace between the Allied Forces and Germany and held Germany responsible for the war. Henry White, Emily’s second husband, was a prominent American diplomat and one of the three American signatories to the treaty. The other two were President Wilson and General Pershing. Also signing - Winston Churchill, Neville Chamberlin, General Goftree, and Marshall Foch.

In 1948, Vanderbilt descendants began operating an inn (The Elm Court Club) to preserve the estate and provide summer employment for the area's many teachers. It was open to the public on Saturday nights for dancing and dinner and made for a popular spot for many years. The Inn eventually faltered, and due to overwhelming operational costs, the house shuttered in 1957, and the property was abandoned for 42 years.

The house was left locked up but intact. Unoccupied, secluded, and only lightly patrolled, it fell prey over the years to massive vandalism, outright looting, some arson, and in a general derelict state until the end of the 20th century. In 1953, the extraordinary elm tree on the grounds for which the property was originally named, died of disease.

The house was passed down through generations, and in 1999, it was passed to Robert Berle, great-great-grandson of the Sloanes, and his wife, Sonya, for a token amount of just under $1,000,000. They undertook more than a $5,000,000 restoration and upgrading effort, repairing, and replacing original details and adding state-of-the-art new wiring, plumbing, heating, and structural upgrades. They then opened the property as a luxury Inn.

In 2002, the Berle family repositioned Elm Court as a luxury destination for weddings; however, it was too costly to operate, and in 2005, the property was briefly for sale before ultimately being taken off the market at the end of 2006. After some work on the property that included the installation of new roofs on several buildings and a new septic system for the entire property. In 2012, the couple sold the estate to Amstar/Travaasa Experiential Resorts, a small collection of destinations, resorts, and spas.

In 2020, Amster/Travassa put the house up for sales before taking it off the market at the end of 2021. In 2022, Elm Court was purchased off-market by Vanderbilt Berkshires Estate, LLC. Throughout the next incarnation of this legacy property, the new owners remain committed to restoring the vibrancy of this treasured landmark. A considerable amount of research on the architecture and design of the Gilded Age and the history of the Vanderbilt family will continue, along with a tremendous responsibility to pay homage to its legendary past. Equally as important, utmost attention will be paid to the historical importance of Elm Court as its own entity, as well as its position and prominence in the Lenox and Stockbridge communities.