Construction 1762 -1764
Settled in 1684, Perth Amboy was described on the official city seal as portus optimus—Latin for "best port." Settlers envisioned it developing into a commercial port and gateway to the New World to rival New York. While it never achieved that status, Perth Amboy nevertheless remained a prominent port town into the early 20th century. Indeed, for a while, it was the capital of East Jersey.
Between 1674 and 1702, governance of the British colony of New Jersey was divided into the two political entities of East Jersey and West Jersey. Each was under control of a Board of Proprietors, wealthy landowners who leased and subdivided their properties.
The East Jersey Proprietors chose Perth Amboy as their capital, while West Jersey selected Burlington. In 1702, governing power was turned over to the crown, though the Proprietors remained an influential real estate corporation. The Royal Governors that followed would split their time between the twin capitals of Perth Amboy and Burlington, renting lodgings and local spaces in which to conduct government business.
​
On March 25, 1761, the Board of the Proprietors of the Eastern Division of New Jersey (to give them their full title) proposed to construct a fine mansion worthy of serving as the residence of the Royal Governors. They hired the English architect and builder John Edward Pryor to design and build what they called the "Proprietary House in Amboy." This was Pryor's first commission in America, though he would go on to design and oversee construction of a number of important buildings in colonial New Jersey and New York. He arrived from England on July 14, 1761, and was able to record in his accounts book for April 3 of the following year: "Set the labourers to digging."
​
The original structure consisted of four levels. In the basement was a kitchen, wine cellar, servants' hall, butler's quarters, and housekeeper's room. The main floor encompassed an entrance hall, drawing room, governor's study, dining room, study, breakfast parlor, and housekeeper's quarters. Upstairs was the master bedroom, dressing room, and guest bedroom. Tucked under the eaves were more servants' quarters. Sixteen fireplaces warmed it in winter and it could boast such advanced architectural features as lead gutters and the protection of a lightning rod.
Troubled by cost overruns and delays that almost ruined Pryor, major construction was at last completed in September of 1764. Unfortunately, this was just in time to coincide with one of the worst economic depressions in the history of the colonies and the first rumblings of unrest that would lead to revolution. The house may have been ready for the Governor, but he wasn't ready for the house. New Jersey's royal governor at the time was William Franklin. London was slow to support his plan to buy the mansion in Perth Amboy and he had heavily invested in a fine estate in Burlington, closer to Philadelphia where his family still resided.
In the meantime, between 1766 and 1771, Proprietary House was leased to New Jersey’s chief justice, Frederick Smyth, and then to another lawyer until the illustrious resident for whom the mansion had been intended was at last able to move in.