While at Baylor, Jim Brown earned 13 varsity letters in soccer, football, wrestling, baseball, and track. In 1963, he was named All-City, All-Mid-South, and was an honorable mention selection to the All-Southern football team. Jim also served as Brigade Commander at Baylor and was a member of The Round Table.
Brown attended Georgia Tech, where he was a three year football letterman. He played in the 1966 Gator Bowl and the 1966 Orange Bowl on Tech teams ranked as high as eighth in the nation. Jim still holds the Georgia Tech punt return record of 22.6 yards per return for the 1965 season.