Edward intended to build a new house with beautiful raised walks and vistas through the woods, an amphitheatre and deer park. His designs were drawn up but the building of the proposed house was never started. The plans for the grounds incorporated a canal or basin of 10 or 12 acres to be filled either from a stream a mile off, or a spring which was above the level of the canal. The gardens however were completed by Vanbrugh's protege and associate Charles Bridgeman and led through blocks of woodland below the house to the long canal and ponds. It is said the finished landscape bore a strong resemblance to the gardens of Blenheim and Bridgeman received payments totalling £654 from Edward Rolt for this work.

It is thought Edward Rolt and Vanbrugh disagreed over some aspect of the house plans and Edward subsequently approached James Gibbs, the architect of St. Martin-in-the-Fields to submit plans for the completion of the project, however before more could be done Edward Rolt died of Smallpox on the 22nd December 1722 and his eldest son, Thomas, inherited his estate and the family home at Sacombe Park and was quite content to live in the old house. This house and walled garden were eventually demolished some time in 1780 and another house built nearby.

After her husband's death, Anne Rolt and her children moved back into Spye Park House and made some alterations to the mansion.

James, the 13th Lord Somerville of Scotland first came to England in 1721 at the age of 23 for the purpose of prosecuting his claim to the Barony of Somerville which he established in 1723. His family had been settled in Scotland for six centuries and during his time in Wiltshire, he became acquainted with the widow, Anne Rolt.

On the 18th September 1724 he married Anne, at Bath Abbey, Somerset (pictured right). The couple had received a special licence from the Bishop to marry at the Abbey. The microfiche of the parish registers for Bath Abbey give no further details of this marriage, nor who was present that day.

In a letter from Bath, the English poet and dramatist, John Gay (1685 - 1732) mentions this wedding. He writes: "The talk of Bath is the marriage of Lord Somerville and Mrs Rolt. She left Bath yesterday. He continues here but is to go away today or tomorrow, but as opinions differ I cannot decide whether they are married or not".

After their marriage they resided at Spye Park House with Anne's six children. One daughter from Anne's first marriage – Anna Marie – had died a year earlier. A daughter was born in 1725, which they christened Anne and in 1726 the couple left Bromham to live in Scotland.

It is not known if all of her children accompanied her to Edinburgh. The eldest two, Thomas and Edward, both named successors to the Bayntun and Rolt estates, may have remained at Spye Park and Sacombe Park under guardianship, but the remaining five children - ranging in age from 15 to 5 years old - would most likely have gone with their mother.

James, the 13th Lord Somerville was 24th in direct male line to Sir Gualter, the First Lord of Whichenour and was 26 years old at the time of the marriage. Ann Rolt was 35 years of age. Between 1726 and 1734, he built the elegant House of Drum - a fine Palladian mansion in the Glenmerton District, on the outskirts of Edinburgh. They had two more children, James and Hugh, born in Scotland.

In 1731 the couple came to Bristol on account of Anne's health, but returned to Scotland again the following year. However Anne's health did not improve and two years later she died on the 24th October 1734 at Drum. Lord Somerville brought Anne's body back to England and to surrender her estate to her son, Sir Edward Bayntun Rolt. Although there is no record of her burial, she was most likely laid to rest in the family crypt beneath the Bayntun chapel in the Church of St. Nicholas in Bromham.

The younger of Anne's children, James, Elizabeth and Wilmot, not of age at the time, were looked after by their elder brother, Edward, and resided with him at Spye Park.

Between the years 1689 and 1734, deeds for the Enmore estate, or Manor of Enmore, saw many name changes to leases, as the various members of the Bayntun family inherited the property. In 1689 Henry Bayntun is mentioned in a lease involving a cottage held of the Manor of Enmore. In 1705, his son and heir, John Bayntun (then aged 17 years), is named alongside his guardian, Walter Grubbe of Potterne, Wiltshire. By 1714, John is listed as the owner of the Manor in another lease, but two years later he died and his sister, Anne and her husband Edward Rolt of Sacombe Park, Hertfordshire are listed in a reversionary lease dated 1716. It is clear from this deed that Anne inherited the Manor of Enmore after John's death and a further reversionary lease in 1730 lists she and her second husband as The Honourable Lord and Lady Somerville. But by 1734 Anne had also died and her son, Edward Rolt would receive the remainder of the Bayntun estates, including the Manor of Enmore, as was laid down by his uncle John's will.

After Anne's death James Somerville married again in 1736 to Frances, the daughter of John Rotherham and they had one daughter, who died young. He died on the 14th December 1765, aged 67 years at his home in Drum.

As a result of the above mentioned 1715 Indenture, or agreement, in 1739, John, Henry and Wilmot Rolt each received £2,000, plus the rate of 5 per cent per annum for every year since reaching their respective ages of 21. The same rate of interest applied to Elizabeth, who received £6,000 – as her only sister, Anna-Maria, died very young in 1923.

At this time however all three parties to the Indenture of 1715 were deceased and Edward Bayntun-Rolt, their successor, was named executor and he sold and leased part of the Bayntun estates to raise the necessary money to honour the deed heard in the High Court of Chancery.