Since the beginning of the study of evolution, people have been fascinated by recent human evolution and adaptation. Despite great progress in our understanding of human history, we still know relatively little about the selection pressures and historical factors that have been important over the past 100,000 years. In that time human populations have spread around the world and adapted in a wide variety of ways to the new environments they have encountered. Here, we investigate the genomic signal of these adaptations using a large set of geographically diverse human populations typed at thousands of genetic markers across the genome. We find that patterns at selected loci are predictable from the patterns found at all markers genome-wide. On the basis of this, we argue that selection has been strongly constrained by the historical relationships and gene flow between populations.