Terahertz (THz) time-domain imaging is an emerging modality and has attracted a lot of interest. However, existing THz imaging systems often require a long scan time and sophisticated system design. Recently, a new design incorporating compressed sensing (CS) leads to a lower detector cost and shorter scan time, in exchange for computation in an image reconstruction step. In this paper, we develop two reconstruction algorithms that can estimate the underlying scene as accurately as possible. First is a single-band CS reconstruction method, where we show that by making use of prior information about the phase and the correlation between the spatial distributions of the amplitude and phase, the reconstruction quality can be significantly improved over previously published methods. Second, we develop a method that uses the multi-frequency nature of the THz pulse. Through effective use of the spatial sparsity, spectroscopic phase information, and correlations across the hyperspectral bands, our method can further enhance the recovered image quality. This is demonstrated by computation on a set of experimental THz data captured in a single-pixel THz system.