While at Baylor, Ellis was the second fastest high school sprinter in the nation. He has now held the Baylor 100-yard dash record of 9.75 seconds for over 40 years. He owns the school mark in the 220-yard dash, as well. Ellis also lettered in football at Baylor and was named to the All-Prep Football Team in 1953 and 1954.
He attended the University of Florida and was the leading scorer for the track team from 1957 through 1959. He won three different SEC 100-yard dash titles, participated on two SEC championship relay teams, and was a member of the 1956 SEC champion team. Ellis was inducted into the University of Florida Athletic Hall of Fame in 1986.