Pertussis toxin, an irreversible inhibitor of some G proteins, inhibits endothelium-dependent relaxations to certain agonists in porcine coronary arteries. In the present study, the effects of the toxin were examined on endothelium-dependent and -independent relaxations of hypercholesterolemic and atherosclerotic porcine coronary arteries to assess the functional state of the endothelial pertussis toxin-sensitive G protein. Male Yorkshire pigs were maintained on either a regular diet (control group, n = 7) or a 2% high-cholesterol diet (cholesterol-fed group, n = 7) for 10 weeks. After the initial 2 weeks of maintenance, animals in both groups underwent balloon catheter removal of the endothelium of the left anterior descending or left circumflex coronary arteries. Endothelium-dependent responses were examined in vitro after 10 weeks of maintenance; at this time, a full lining of endothelial cells in both left coronary arteries was confirmed histologically. In arteries with endothelium of the control group (normal responses), pertussis toxin significantly inhibited the endothelium-dependent relaxations to serotonin, UK14304 (a selective alpha 2-adrenergic receptor agonist), and thrombin but not those to ADP, bradykinin, or the calcium ionophore A23187. In previously denuded arteries of the control group (effects of endothelial regeneration alone) or intact arteries of the cholesterol-fed group (effects of hypercholesterolemia alone), the relaxations to serotonin, UK14304, and thrombin were impaired significantly; those relaxations were impaired further in previously denuded arteries of the cholesterol-fed group (effects of atherosclerosis). The inhibitory effects of pertussis toxin were significantly reduced after endothelial regeneration and in hypercholesterolemia and were almost absent in atherosclerosis.(ABSTRACT TRUNCATED AT 250 WORDS)