Idiopathic infantile nystagmus (IIN) is a genetically heterogeneous disorder of eye movement that can be caused by mutations in the FRMD7 gene that encodes a FERM domain protein. FRMD7 is expressed in the brain and knock-down studies suggest it plays a role in neurite extension through modulation of the actin cytoskeleton, yet little is known about its precise molecular function and the effects of IIN mutations. Here, we studied four IIN-associated missense mutants and found them to have diverse effects on FRMD7 expression and cytoplasmic localization. The C271Y mutant accumulates in the nucleus, possibly due to disruption of a nuclear export sequence located downstream of the FERM-adjacent domain. While overexpression of wild-type FRMD7 promotes neurite outgrowth, mutants reduce this effect to differing degrees and the nuclear localizing C271Y mutant acts in a dominant-negative manner to inhibit neurite formation. To gain insight into FRMD7 molecular function, we used an IP-MS approach and identified the multi-domain plasma membrane scaffolding protein, CASK, as a FRMD7 interactor. Importantly, CASK promotes FRMD7 co-localization at the plasma membrane, where it enhances CASK-induced neurite length, whereas IIN-associated FRMD7 mutations impair all of these features. Mutations in CASK cause X-linked mental retardation. Patients with C-terminal CASK mutations also present with nystagmus and, strikingly, we show that these mutations specifically disrupt interaction with FRMD7. Together, our data strongly support a model whereby CASK recruits FRMD7 to the plasma membrane to promote neurite outgrowth during development of the oculomotor neural network and that defects in this interaction result in nystagmus.