The mechanism of the rhodium-catalyzed cascade oxidative annulation of benzoylacetonitrile with alkynes is investigated using density functional theory calculations. The result shows that the reaction undergoes a stepwise annulation process, wherein the 1-naphthol acts as an intermediate. The first-step annulation involves the sp(3) C-H bond cleavage, sp(2) C-H bond cleavage, alkyne insertion into the Rh-C(sp(2)) bond, ketone enolization, and reductive elimination to produce the 1-naphthol intermediate. The second-step annulation involves the O-H cleavage, sp(2) C-H bond cleavage, alkyne insertion into the Rh-C(sp(2)) bond, and C-O reductive elimination to generate the final product naphtho[1,8-bc]-pyran. The sp(3) C-H bond cleavage rather than the sp(2) C-H bond cleavage is found to be the rate-determining step of the catalytic cycle. The ketone enolization should occur before the reductive elimination. The substituent effects on the reactivities and regioselectivities of reactions are also analyzed. These calculation results shed light on some ambiguous suggestions from experiments.