We present a high-speed and phase-sensitive reflectance line-scanning confocal holographic microscope (LCHM). We achieved rapid confocal imaging using a fast line-scan CCD camera and quantitative phase imaging using off-axis digital holography (DH) on a 1D, line-by-line basis in our prototype experiment. Using a 20 kHz line scan rate, we achieved a frame rate of 20 Hz for 512x512 pixels en-face confocal images. We realized coherent holographic detection two different ways. We first present a LCHM using off-axis configuration. By using a microscope objective of a NA 0.65, we achieved axial and lateral resolution of ~3.5 micrometers and ~0.8 micrometers, respectively. We demonstrated surface profile measurement of a phase target at nanometer precision and the digital refocusing of a defocused confocal en-face image. Ultrahigh temporal resolution M mode is demonstrated by measuring the vibration of a PZT-actuated mirror driven by a sine wave at 1 kHz. We then report our experimental work on a LCHM using an in-line configuration. In this in-line LCHM, the coherent detection is enabled by moving the reference arm at a constant speed, thereby introducing a Doppler frequency shift that leads to spatial interference fringes along the scanning direction. Lastly, we present a unified formulation that treats off-axis and in-line LCHM in a unified joint spatiotemporal modulation framework and provide a connection between LCHM and the traditional off-axis DH. The presented high-speed LCHM may find applications in optical metrology and biomedical imaging.