The first Olympic games occurred over 1,250 years before the fall of the Roman Empire.

The first Olympic Games occurred in 776 B.C. They continued until A.D. 393, when Emperor Theodosius declared that these "pagan cults" must be banned.
Most historians agree that the Roman Empire "officially" ended in A.D. 476.
Source: Olympic.org