Brown attended Middlebury College in Vermont, where he became the first African American member of Sigma Phi Epsilon, collegiate fraternity and as result, the national charter of SPE at Middlebury was rescinded and the fraternity became known as Sigma Epsilon, the Harlemworldmagazine.com