While at Baylor, Joe compiled an impressive record. He holds 13 Baylor letters in five different sports. A three-year starter on the varsity football team, he captained the team in 1943 and was named to both the All-Mid-South and the All-City Football teams for two consecutive years. He was also a varsity wrestler and a Mid-South wrestling champion in 1944. He also captained the track team and was a boxer and weight lifter. Joe was the treasurer of the Student Council in 1943 and received the Headmaster's Award of Exceptional Merit in 1944.
After lettering one year at Tennessee (where he played in the 1945 Rose Bowl), Joe received an appointment to West Point. In 1947, he was captain of the Cadets' football team, was named to the All-America Team, and received the Outland Trophy, which is given to the best interior lineman in collegiate football.
Joe received the Baylor Distinguished Service Award in 1973. He was named to the All-Time Army team and was inducted into the National Football Foundation Hall of Fame in 1987.