Aaron Brown was born in 1779, the eldest of eight children of Job Brown II and Jane Tompkins of South Orange. Brown was a direct descendant of John Brown and his wife May who had come from Connecticut in 1666 with the original settlers of Newark. Aaron Brown received a sum of money from his uncle which he used to purchase property in South Orange Village. Early in the nineteenth century, Brown married Dorcas Ball, from another prominent early Maplewood family. He lived the rest of his life in this house, where he and Dorcas raised their six children.
There is some indication (so far unsubstantiated) that the existing house was constructed around a small preexisting house dating to c. 1749, that includes the area on the western section of the house. This section of the house has a rough stone cellar with a large hearth. Originally two rooms with two adjoining fireplaces that shared a chimney, it would also have had a small loft. Aaron Brown and Dorcas Ball either built the original house in 1805, or enlarged the existing structure to a one-and one-half story building to have a center hall with living rooms on one side and dining room and kitchen on the other.