Immunoglobulin heavy chain rearrangement (V(H)-to-DJ(H)) occurs only in B cells, suggesting it is inhibited in other lineages. Here we found that in the mouse V(H) locus, methylation of lysine 9 on histone H3 (H3-K9), a mark of inactive chromatin, was present in non-B lineage cells but was absent in B cells. As others have shown that H3-K9 methylation can inhibit V(D)J recombination on engineered substrates, our data support the idea that H3-K9 methylation inhibits endogenous V(H)-to-DJ(H) recombination. We also show that Pax5, a transcription factor required for B cell commitment, is necessary and sufficient for the removal of H3-K9 methylation in the V(H) locus and provide evidence that one function of Pax5 is to remove this inhibitory modification by a mechanism of histone exchange, thus allowing B cell-specific V(H)-to-DJ(H) recombination.