Frustrated magnets hold the promise of material realizations of exotic phases of quantum matter, but direct comparisons of unbiased model calculations with experimental measurements remain very challenging. Here we design and implement a protocol of employing many-body computation methodologies for accurate model calculations-of both equilibrium and dynamical properties-for a frustrated rare-earth magnet TmMgGaO4 (TMGO), which explains the corresponding experimental findings. Our results confirm TMGO is an ideal realization of triangular-lattice Ising model with an intrinsic transverse field. The magnetic order of TMGO is predicted to melt through two successive Kosterlitz-Thouless (KT) phase transitions, with a floating KT phase in between. The dynamical spectra calculated suggest remnant images of a vanishing magnetic stripe order that represent vortex-antivortex pairs, resembling rotons in a superfluid helium film. TMGO therefore constitutes a rare quantum magnet for realizing KT physics, and we further propose experimental detection of its intriguing properties.