The low risk of progression of Barrett's esophagus (BE) to esophageal adenocarcinoma can lead to over-diagnosis and over-treatment of BE patients. This may be addressed through a better understanding of the dynamics surrounding BE malignant progression. Although genetic diversity has been characterized as a marker of malignant development, it is still unclear how BE arises and develops. Here we uncover the evolutionary dynamics of BE at crypt and biopsy levels in eight individuals, including four patients that experienced malignant progression. We assay eight individual crypts and the remaining epithelium by SNP array for each of 6-11 biopsies over 2 time points per patient (358 samples in total). Our results indicate that most Barrett's segments are clonal, with similar number and inferred rates of alterations observed for crypts and biopsies. Divergence correlates with geographical location, being higher near the gastro-esophageal junction. Relaxed clock analyses show that genomic instability precedes and is enhanced by genome doubling. These results shed light on the clinically relevant evolutionary dynamics of BE.