We show by pump-probe spectroscopy that the optical response of a fishnet metamaterial can be modulated on the femtosecond time scale. The modulation dynamics is dominated by pump-induced changes in the constituting dielectric medium, but the strength of modulation is dramatically enhanced through the plasmon resonance. The pump-induced spectral responses of the metamaterial provide understanding on how the resonance is modified by pump excitation. Our study suggests that metamaterials can be used as high-speed amplitude/phase modulators with terahertz-bandwidth.