Executive Summary

Firms are hospital-based and consist of more than one group of physicians, nurses, and allied staff, who provide ongoing medical care for a defined patient population. These teams of parallel caregivers are similar and provide inpatient or outpatient care. They are more likely to be found in general internal medicine services in large hospitals. They were often started to promote continuity of care, better residency training and to share equally the patient care work load.

Firms research is based on the concept of ongoing random assignment of new patients or new physicians to one of these parallel provider groups. An experimental change is made in one firm, leaving another as an unchanged control. The effect of the change is evaluated by comparison and, if beneficial, applied to both groups. Then the next change is introduced in an ongoing process. This can increase ethical, low-cost, high-quality medical care and improve the organization of care.

This summary report will describe the organization of several hospitals' firms, along with the clinical trials carried out so far, and will present the methodological, organizational, ethical issues, and future directions discussed at the workshop. There is now enough experience with this innovative approach to efficient clinical research to show that it can be widely adopted. Firms research is particularly appropriate for the evaluation and use of clinical guidelines and policies.

The Basic Concept

Research using firms applies three core concepts: parallel providers of care; ongoing patient random assignment to providers; and continuous experimental changes, evaluation, and, if successful, implementation. Each of these concepts has been observed in the past by themselves, but their combination is new.(37)

When the volume of patients is sufficiently large, more than one provider or group of providers is needed to provide care. When there is more than one provider group, there has to be some way of allocating patients to these providers. In large organizations such as hospitals, large group practices, or government health services, various methods are often used. The new patient who contacts a large elite, private clinic for an evaluation may be assigned to the next available internist. Depending on the day of the week, there may be different teams covering the hospital emergency room. Several American teaching hospitals have affiliations with two medical schools and new patients are assigned alternately to these two separate services.(1, 32) In some hospitals, there are separate inpatient or outpatient care teams, often called firms, who are similar in staffing and provide care to similar patient populations. Haphazard or rotational assignment of patients occurs in many settings.(10, 30) The next step is to change haphazard assignment to random assignment of patients to parallel providers. If the parallel providers (teams, firms) are already in existence, and assignment of new patients is now haphazard, then the change to random assignment is a small step.

In health care organizations like hospitals, changes in administration, education, organization, information flow, and staffing are many and occur constantly. These changes occur out of necessity or out of the belief that they will result in better care. Rarely are such changes ever examined in any way, much less carefully evaluated. If there are parallel teams, random assignment of patients should be made to provide the best assurance of similar patient populations. Changes made for one provider group can be compared to the provider group that was not changed. If the change is demonstrably beneficial, it can be made for all providers and the next change begun in a continuous process.

What Are Firms?

Firms have mostly started for reasons other than research. They are seen where there are large volumes of patients coming for care to one institution. In large urban teaching hospitals, where care has been provided by residents and subspecialty clinics have multiplied, the patient never saw the same doctor twice, the resident physicians never saw the outcome of the care they provided, and there was no coordination or continuity of care across many subspecialty units.

This situation argued for creating teams of generalist providers of care out of a large amorphous clinical service. These teams would include residents, attending physicians, nurses, and other ancillary staff. Because this group would work together, they would know each other's work habits. Patients were assigned to one of these groups and therefore have a greater change of seeing the same physicians and staff over time. These teams have been created in inpatient units, outpatient units, or both.

Attending physicians assigned to a firm would get to know the residents better and understand their educational needs. General internal medicine firms have become the coordinating center base for their patients' care. Patients remain with the firms unless referred to subspecialty clinics.

Similar teams needed a fair distribution of the patient work load. Therefore, patients were assigned to a firm based on the next available bed, by rotation, or by the whim of a clerk.

On the inpatient side, hospital architecture that creates separate floors or nursing units leads to the creation of separate teams associated with each unit. In some hospitals these teams have been called firms, a term taken from British hospital usage.

Given the existence of these similar parallel teams and haphazard assignment of new patients, the step to formal randomization is a relatively easy one.(29)

A table of random numbers can be generated and the admissions office, emergency room, and other points of entry use them to assign new patients. Stratified random assignment of resident physicians also can be used. Once patients are assigned to a firm they remain with that firm to promote continuity.

Randomization has to be monitored and maintained. There is a form of housekeeping required to keep such systems going.(39)

Examples of Firm Systems

Examples of Research

The types of research carried out to date include trials of test usage, promoting preventive care, changing provider roles, organizational change, and provider education. Many of the trials could fit into more than one of these categories. It is not our purpose here to describe these trials in detail. The interested reader is referred to the original publications.

The first Metro trial consisted of feeding back to house staff information on the costs of the laboratory tests they ordered in two randomly chosen firms and on the costs of x-ray tests in the other two firms. Total costs per resident per week were posted on the inpatient office wall. Costs for lab and x-ray tests were tracked before and during the trial. This information, which the residents could use or ignore, resulted in a significant reduction in testing. A special feature of this study was an effort to control and monitor the Hawthorne effect by having two concurrent experimental interventions.(3)

The second trial used chart reminders and education to promote preventive procedures in the ambulatory clinic.(4) This resulted in a modest, yet statistically significant, improvement in preventive measures. Observing that resident use of preventive measures is woefully limited in spite of a belief by the residents that these procedures (mammography, colon cancer screening) should be done led to two trials, one at Harborview Hospital in Seattle and one at University Hospitals of Cleveland. In each of these trials, clinical responsibilities were shifted from physicians to nurses. In Seattle, nurses were given the responsibility for screening for alcoholism and, if appropriate, making referral to an alcoholism counselor. The result was a significant increase in appropriate referrals.(43)

At University Hospitals of Cleveland, nurse practitioners took charge of colon cancer screening using occult blood tests. This caused a major increase in the percent of patients who returned their tests.(35)

At Cleveland MetroHealth Center, intravenous therapy had been managed by the residents and nursing staff.(5, 6) A firm trial introduced a specialized IV therapy team. Infections (phlebitis) were counted for the team and compared with firms that remained unchanged. The IV teams had a significantly lower infection rate and as a result were introduced in all the firms. Later, in a budget cutting effort, the elimination of these IV teams was proposed. By bringing forth the persuasive evidence from the trial, the IV teams have survived to this day.

The costs of medical education in teaching hospitals have been a major issue for many years. A number of nonexperimental and often expensive studies have been carried out to measure the costs of residency training in hospitals.

In an inpatient general internal medicine unit at Henry Ford Hospital, patients were randomly assigned either to a traditional team of residents with an attending advisor or to a team of full-time attending physicians. Henry Ford Hospital has a full-time salaried medical staff, some of whom were assigned to the team in this unit. Length-of-stay, test usage, salary costs, and mortality were compared in this trial between the resident team and the attending team. Despite their higher salaries, the attending group had costs that were up to 25 percent lower with no difference in mortality. The savings resulted from a shorter length-of-stay and less testing.(41)

As far as is known this is the only randomized clinical trial of the costs of residency education. Because Henry Ford Hospital is at the center of a 420,000-member HMO, these cost savings directly affect the economics of the organization. To the degree that teaching costs can be passed on to third-party payers, it lessens the pressure on the hospital to worry about these added costs.

The most elaborate firm trial carried out to date at Cleveland Metro Health Center was a major reorganization of the firms' ambulatory services.(14, 15, 20, 22, 23, 24) Ambulatory clinic registration was decentralized, clinic hours were increased to five days a week, and off-hour access to physicians by ambulatory patients was made easier. These changes resulted in a significant reduction in inpatient length- of-stay and improved patient satisfaction. It showed that organization of ambulatory care can influence inpatient care utilization.

Methodological Issues

While carrying out these firm trials, researchers have been working on and thinking about the methodological issues involved.(10, 16, 30, 38, 39) Some of these include:

Ethics of Firms Research

Although all firm trials have been reviewed and approved by the hospitals' human subjects committees, they do raise new questions about the ethics of research. A substantial amount of thought has been given to these issues.(34, 42)

Under what circumstances can general organizational changes be made that do not require individual patient consent? The changes that have been studied can be viewed as the type of administrative and educational changes that occur in all hospitals without careful evaluation or without written informed consent from the patients.

What makes it possible to introduce such changes without formal individual informed consent is that these changes do not restrict decisionmaking by the patient and the physician. The firm trials that include changing organization, increased information provided to physicians, or more education do not compel change in treatment. These changes still allow physician, nurse, and patient to make their own agreed-upon decisions.

Dr. Goldberg has provided the following notice to all patients in his clinics. This is a general statement addressed to all patients. "The staff of the Adult Medicine Clinic is always interested in improving the care we provide. We sometimes try what we think is a better way of doing things on one of our group practices but not on another. We might, for example, try a different method of scheduling appointments. By comparing groups after a short period, we can decide whether any changes work well enough to be worth adopting for all our patients. For this kind of evaluation we only select administrative changes that do not constrain the clinical process; that is, you and your physician are always able to decide what medical tests and treatments you do or do not receive. If information contained in your medical record is used in such an evaluation, your anonymity will be carefully protected."(42)

A firm trial that includes a change in required testing or treatment is different. In one proposed trial, a new way to organize a diabetic consult service was to be put in place in one firm to see if this resulted in better patient outcome. The outcome would be based on a blood test to measure control of this problem. The human subjects committee at the Cleveland MetroHealth Center decided that each patient being tested in this way should give specific permission for this test. This ruling met with general approval. In this case, the diabetic patient could directly benefit from the knowledge gained from the test results. Firm trials that constrain patient and physician choice have otherwise not been conducted.

Efficiency in Research

One of the most important features of firm trials is that they can be done inexpensively at a fraction of the cost of traditional randomized clinical trials. If we are to have more good evidence of what works and what does not in clinical medicine, we will need to have many more randomized trials. If each one will cost millions of dollars, that number will be limited.

If we can drive the costs of trials toward zero, we can afford more and more of them. By having a research team in place, having randomization as part of the ongoing care system, and having computer-based management information systems that can easily aggregate the needed data, then the costs can be vanishingly small for some trials.

Estimating the true costs of a firm trial can quickly entangle one into accounting and economic controversy (joint product problems, microcosting analysis, costs to whom, etc.).

We have used a more simple approach to cost measurement that is directly relevant to researchers. How much new money did the researcher have to go out and find to do this study? Using this rough but relevant rule of thumb, two of the firm trials have been carried out for add-on-costs of under $1,000.(11, 12, 43)

That is not to say that all envisioned trials will cost so little. Some will cost much more. Two of the largest trials have drawn on external funding of over $100,000. However, the general commitment to efficiency in clinical research is a guiding principle of firms research.

Management Information Systems

One key to efficient firms research is the development of computer-based information systems that routinely record and aggregate information about care provided on the firms. How many male patients from 50-60 years of age had high blood pressure in February? What is the total price of x-rays ordered by this resident physician? What is the mean DRG score for this firm? How many people cared for by the firms in 1989 are alive in 1991?

The routine, easy-to-get, low-cost availability of data to answer such questions is essential for efficient trials. The development of such systems does not happen immediately. The Regenstrief Group in Indianapolis has been working to develop such a system for ambulatory care over the last 17 years. The University of Michigan Hospital in Ann Arbor has developed a detailed inpatient costing information system for their general internal medicine inpatient firms.

For the future, if there are to be multicenter firm trials, each hospital information system should be compatible with the others so that the same questions can easily be answered for all hospitals participating in the trial.

Clinical Guidelines Trials

The new Federal Government initiative to develop clinical practice guidelines will call for a large number of clinical trials to evaluate physician compliance, problems with compliance, costs, patient satisfaction, and outcome.

To evaluate several hundred such guidelines by traditional multicenter trials at millions of dollars per trial will be impossible. Firm systems may be an answer to this need for evaluation. Firm trials can be done at a fraction of the cost and are ideal for evaluating compliance with guidelines for frequently occurring medical conditions. In one firm, the use of the guideline is encouraged, while the other firm practices "business as usual" medicine. The second control firm could be the experimental firm for a different guideline about a different illness, thus controlling for the Hawthorne effect and doubling the research productivity.

The Future: Multicenter Trials

Already on the drawing board are multicenter firm trials. Such trials could increase the power of trials by increasing the number of subjects. Such trials can be a step toward answering the question of generalizability of results. If it works in Seattle, will it work in Cleveland? Each hospital will be slightly different and this can add strength to the argument of generalizability.

Applying basic ideas behind firm trials can result in different approaches to good management. There are 100 primary care clinics in Stockholm, Sweden, 173 Veterans Administration Medical Centers, and 82 regional health clinics in Costa Rica. These multiple sites can be used as the basis for experimental changes. In Stockholm, 40 clinics may ask to participate in an educational program to improve diabetic care. Only 20 can be accommodated this year. Fairness suggests randomly choosing 20 clinics this year. The effects of the program can be measured by the health status of diabetics cared for in these clinics. The Karolinska Hospital has carried out such projects. Management based on ongoing randomization is a rational strategy, and this idea may someday spread beyond medical care and be a contribution to the general theory of management.

So far no trial has forced a change in patient-physician decisions. The mammography, alcohol counseling, x-ray, or IV can be offered (or rejected) by every doctor (and every patient). Under what circumstances would it be acceptable to conduct a firm trial for alternative drugs, for a new drug, for surgery or not? At this writing it is not known how these questions will be answered.

Firm trials are not useful for the study of rare diseases but are better for the common problems that occupy so much of medical care.

Conclusion

Parallel providers of care (inpatient or outpatient firms), the random assignment of new patients, and a commitment to efficient, continuous, careful evaluation and improvement are the basic ideas behind firm trials. There is now enough experience with firm trials to say that they are a viable addition to our tools for improving the delivery of medical care.

Speakers

Workshop Day 1

• Frederick Mosteller, Ph.D.
• Chairperson
• Roger I. Lee Professor of Mathematical Statistics Emeritus
• Director of Technology Assessment Program
• Harvard University
• Boston, Massachusetts
• Randall D. Cebul, M.D.
• "Research Using the Metro Firm System"
• Associate Professor of Medicine
• Case Western Reserve University
• Chief of General Internal Medicine
• Division of General Medicine
• Department of Medicine
• MetroHealth System
• Cleveland, Ohio
• Neal V. Dawson, M.D.
• "Organizing the Metro Firms for Research"
• Associate Professor of Medicine
• Case Western Reserve University at MetroHealth Medical Center
• Cleveland, Ohio
• Thomas Ebert, M.D.
• "Ongoing Randomization in a Community Hospital"
• Associate Chief of Medicine
• Program Director
• The Medical Center of Central Massachusetts--Memorial
• Worcester, Massachusetts
• Harold Goldberg, M.D.
• "The Ethics of Ongoing Randomization Trials: The Harborview Experience"
• Assistant Professor of Medicine
• University of Washington
• Harborview Medical Center
• Seattle, Washington
• Laurence McMahon, M.D., M.P.H.
• "Computerized Clinical and Cost Information Systems for Research"
• Chief Director of General Medicine
• University of Michigan Medical Center
• Department of General Medicine
• Ann Arbor, Michigan
• David Nerenz, Ph.D.
• "Resident or Special Staff Attendings: Patient Management on an Internal Medicine Inpatient Unit"
• Director
• Center for Health System Studies
• Henry Ford Health System
• Duncan Neuhauser, Ph.D.
• "Ongoing Randomization, Parallel Providers of Care, and Continuous Improvement of Care"
• Department of Epidemiology and Biostatistics
• Case Western Reserve University Medical School
• Cleveland, Ohio
• William Tierney, M.D.
• "Ongoing Randomization and the Research Agenda in Indianapolis"
• Assistant Director
• Computer Science Research Group
• Regenstrief Institute
• Indiana University
• Indianapolis, Indiana

Workshop Day 2

• Frederick Mosteller, Ph.D.
• Chairperson
• Roger I. Lee Professor of Mathematical Statistics Emeritus
• Director of Technology Assessment Program
• Harvard University
• Boston, Massachusetts
• Michael Barry, M.D.
• "Status of New and Evolving Programs: Goals, Plans/Progress, and Problems"
• Assistant Chief of Medicine for Residency Training
• Massachusetts General Hospital
• General Internal Medicine Unit
• Boston, Massachusetts
• Neal V. Dawson, M.D.
• "Results of a Survey"
• Associate Professor of Medicine
• Case Western Reserve University at MetroHealth Medical Center
• Cleveland, Ohio
• Thomas Ebert, M.D.
• "Status of New and Evolving Programs: Goals, Plans/Progress, and Problems"
• Associate Chief of Medicine
• Program Director
• The Medical Center of Central Massachusetts--Memorial
• Worcester, Massachusetts
• Daniel W. Rahn, M.D.
• "Status of New and Evolving Programs: Goals, Plans/Progress, Successes and Problems"
• Assistant Professor of Medicine
• Yale University Medical School
• New Haven, Connecticut
• Peter Rudd, M.D.
• "Status of New and Evolving Programs: Goals, Plans/Progress, and Problems"
• Associate Professor of Medicine
• Division of Internal Medicine
• Stanford University Medical School
• Stanford, California
• James P. Whelan, M.D.
• "Status of New and Evolving Programs: Goals, Plans/Progress, and Problems"
• Program Director for Internal Medicine
• Associate Professor of Clinical Medicine
• University of Chicago Medical School
• Chicago, Illinois

About the NIH Technology Assessment Program

NIH Technology Assessment Conferences and Workshops are convened to evaluate available scientific information related to a biomedical technology when topic selection criteria for a Consensus Development Conference are not met. The resultant NIH Technology Assessment Statements are intended to advance understanding of the technology or issue in question and to be useful to health professionals and the public.

Some Technology Assessment Conferences and Workshops adhere to the Consensus Development Conference format because the process is altogether appropriate for evaluating highly controversial, publicized, or politicized issues. Other Conferences and Workshops are organized around unique formats. In this format, NIH Technology Assessment Statements are prepared by a nonadvocate, nonfederal panel of experts, based on: (1) presentations by investigators working in areas relevant to the consensus questions typically during a 1-1/2-day public session; (2) questions and statements from conference attendees during open discussion periods that are part of the public session; and (3) closed deliberations by the panel during the remainder of the second day and morning of the third. This statement is an independent report of the panel and is not a policy statement of the NIH or the Federal Government.

Preparation and distribution of these reports are the responsibility of the Office of Medical Applications of Research, National Institutes of Health, Federal Building, Room 618, Bethesda, MD 20892.

Bibliography

1.

Halperin, W, Neuhauser, D. MEU: A Way of Measuring Efficient Utilization of Hospital Services. Health Care Manage Rev. 1976; 1(1):63-70. [PubMed: 10316807]

2.

Waggoner, DM, Frengley, JD, Griggs, RC, Rammelkamp, CH. A "Firm" System for Graduate Training in General Internal Medicine. J Med Educ. 1979; 54:556-561. [PubMed: 448712]

3.

Cohen, DI, Jones, P, Littenberg, B, Neuhauser, D. Does Cost Information Availability Reduce Physician Test Usage? Med Care. 1982a; 20:286-292. [PubMed: 7078287]

4.

Cohen, DI, Littenberg, B, Wetzel, C, Neuhauser, D. Improving Physician Compliance with Preventive Medicine Guidelines. Med Care. 1982; 20:1040-1045. [PubMed: 7132464]

5.

Tomford, JW, Hershey, CO, McLaren, CE, Porter, DK, Cohen, DI. Intravenous Therapy Team and Peripheral Venous Catheter-Associated Complications: A Prospective Controlled Study. Arch Intern Med. 1984; 144:1191-1194. [PubMed: 6375611]

6.

Hershey, CO, Tomford, JW, McLaren, CE, Porter, DK, Cohen, DI. The Natural History of Intravenous Catheter-Associated Phlebitis. Arch Intern Med. 1984; 144:1373-1375. [PubMed: 6732398]

7.

McDonald, CJ, Hui, SL, Smith, DM, et al. Reminders to Physicians from an Introspective Computer Medical Record: A Two-Year Randomized Trial. Ann Intern Med. 1984; 100:130-139. [PubMed: 6691639]

8.

Cohen, SJ, Weinberger, M, Hui, SL, Teirney, WM, McDonald, CJ. The Impact of Reading on Physicians' Nonadherence to Recommended Standards of Medical Care. Soc Sci Med. 1985; 21:909-914. [PubMed: 4071124]

9.

Neuhauser, D, Cohen, D, Austin, S, et al. Ongoing Patient Randomization for Better Health Care. Bull Clev Med Libr. 1985; 31:5-7.

10.

Cohen, DI, Neuhauser, D. The Metro Firm Trials: An Innovative Approach to Ongoing Randomized Clinical Trials. In: Assessing Medical Technology , Institute of Medicine, Washington, DC, National Academy Press, 1985; 529-534.

11.

Hershey, CO, Porter, DK, Breslau, D, Cohen, DI. Influence of Simple Computerized Feedback on Prescription Charges in an Ambulatory Clinic: A Randomized Clinical Trial. Med Care. 1986; 24:472-481. [PubMed: 3520177]

12.

Neuhauser, D, Editorial. Med Care. 1986; 24:471.

13.

Cohen, DI, Breslau, D, Porter, DK, Hershey, CO, et al. Academic Group Practice: The Patient's Perspective. Med Care. 1986; 24:990-998. [PubMed: 3773581]

14.

Cohen, DI, Breslau, D, Porter, DK, Goldberg, HI, et al. The Cost Implications of Academic Group Practice: A Randomized Controlled Trial. N Engl J Med . 1986; 314:1553-1557. [PubMed: 3713748]

15.

Cohen, DI, Dawson, NV, Goldberg, HI, Hershey, CO. Correspondence: Academic Group Practice. N Engl J Med. 1986; 315:1420.

16.

Cargill, V, Cohen, D, Kroenke, K, Neuhauser, D. Ongoing Patient Randomization: An Innovation in Medical Care Research. Health Serv Res. 1986; 21:663-678. [PMC free article: PMC1068980] [PubMed: 3546202]

17.

Tierney, WM, Hui, SL, McDonald, CJ. Delayed Feedback of Physician Performance versus Immediate Reminds to Perform Preventive Care: Effects on Physician Compliance. Med Care. 1986; 24:659-666. [PubMed: 3736141]

18.

Tierney, WM, McDonald, CJ, Martin, DK, Hui, SL, Rogers, MP. Computerized Display of Past Test Results: Effect on Outpatient Testing. Ann Intern Med. 1987; 107:569-574. [PubMed: 3631792]

19.

Vinicor, F, Cohen, SJ, Mazzuca, SA, et al. DIABEDS: A Randomized Trial of the Effects of Physician and/or Patient Education on Diabetes Patient Outcomes. J Chron Dis .1987; 40:345-356. [PubMed: 3549757]

20.

Hershey, CO, Cohen, DI, Goldberg, HI, McLaren, CE, et al. Effect of an Academic Group Practice on Patient Show Rates: A Randomized Controlled Trial. Med Care. 1987; 25:72-77. [PubMed: 3807448]

21.

Hershey, CO, Dawson, NV, McLaren, CE, et al. Resident Knowledge of Charges: Are We Asking the Right Questions? Am J Med Sci. 1987; 293:182-186. [PubMed: 3565464]

22.

Cohen, DI, Breslau, D, Porter, DK, Goldberg, HI, et al. Academic Group Practice: The Physician's View. Med Care. 1987; 25:686-694. [PubMed: 3695671]

23.

Goldberg, HI, Cohen, DI, Hershey, CO, Hsiue I-L, et al. A Randomized Controlled Group of Academic Group Practice: Improving the Operation of the Medicine Clinic. JAMA. 1987; 257:2051-2055. [PubMed: 3560381]

24.

Goldberg, HI, Cohen, DI, Hershey, CO. Correspondence: A Randomized Controlled Trial of Academic Group Practice. JAMA. 1987; 258:613-614. [PubMed: 3612984]

25.

Cebul, RD, Dawson, NV. Translating Theory into Practice: The Metro Firm System. Soc Gen Intern Med Newsletter. 1987; 10(5):2-3.

26.

Tierney, WM, McDonald, CJ, Hi, SL, Martin, DK. Computer Predictions of Abnormal Test Results: Effects on Outpatient Testing. JAMA. 1988; 259:1194-1198. [PubMed: 3339821]

27.

Goldberg, HI. Innovations and Institutions: The Metro Firm System: A Primer. Soc Gen Intern Med Newsletter. 1988; 11(2):4-5.

28.

Hershey, CO, Goldberg, HI, Cohen, DI. The Effect of Computerized Feedback Coupled with a Newsletter upon Outpatient Prescribing Charges. Med Care. 1988; 26:88-93. [PubMed: 3336248]

29.

Dawson, NV.The Metro Firm System: Meeting the Challenges of a Changing Health Care Environment. Health Matrix. 1988; 5:27-33 . [PubMed: 10286435]

30.

Neuhauser, D. The Metro Firm Trials and Ongoing Randomization. In: JM Tanur, F Mosteller, WH Kruskal, EL Lehman, RF Link, RS Pieters, GR Rising (Eds). Statistics: A Guide to the Unknown. Pacific Grove, CA, Wadsworth and Brooks, 1989; 25-30.

31.

Cohen, SJ, Stookey, GK, Katz, BP, Drook, CA, Smith, DM. Encouraging Primary Care Physicians to Help Smokers Quit. A Randomized, Controlled Trial. Ann Intern Med . 1989; 110:648-652. [PubMed: 2930095]

32.

Gillespie, KN, Romeis, JC, Virgo, KS, et al. Practice Pattern Variation Between Two Medical Schools. Med Care. 1989; 27:537-542 . [PubMed: 2498584]

33.

Tierney, WM, Miller, ME, McDonald, CJ. The Effect on Test Ordering of Informing Physicians of the Charges for Outpatient Diagnostic Tests. N Engl J Med. 1990; 322:1499-1504 . [PubMed: 2186274]

34.

Goldberg, HI. Ethical Issues in Administrative Continuous Improvement: Applying the Concept of Prior Notification to the Conduct of Firm Trials. Med Care (in press) 1991 . [PubMed: 2205768]

35.

Cargill, V, Conti, M, Neuhauser, D, McClish, D. Improving the Effectiveness of Screening Colorectal Cancer by Involving Nurse Clinicians. Med Care. 1991; 29:1. [PubMed: 1986175]

The following papers (36-45), presented at this workshop, were published in a supplement of Medical Care (1991, Vol. 29, No. 7)

36.

Mosteller, F. The Contributions of Firms: A Fresh Movement in Medicine. [PubMed: 1857133]

37.

Neuhauser, D. Parallel Providers, Ongoing Randomization, and Continuous Improvement. [PubMed: 1857136]

38.

Cebul, R. Randomized, Controlled Trials Using the Metro Firm System. [PubMed: 1857139]

39.

Dawson, N. Organizing the Metro Firm System for Research. [PubMed: 1857131]

40.

Nerenz, DR, Zajac, BM, Miron, LL, Romans, J, Simmer, TL. Inpatient Firms in a Teaching Hospital: The Henry Ford Hospital Experience. [PubMed: 1857132]

41.

Simmer, TL, Nerenz, DR, Rutt, WM, Newcomb, CS, Benfer, DW. A Randomized, Controlled Trial of An Attending Staff Service in General Internal Medicine. [PubMed: 1906962]

42.

Goldberg, HI, McGough, H. The Ethics of Ongoing Randomization Trials: Investigation Amongst Intimates. [PubMed: 1857134]

43.

Goldberg, HI, Mullen, M, Ries, RK, Psaty, BM, Ruch, BP. Alcohol Counseling in a General Medicine Clinic: A Randomized, Controlled Trial of Strategies to Improve Referral and Show Rates. [PubMed: 1857135]

44.

Tierney, WM, Miller, ME, Hui, SL, McDonald, CJ. Practice Randomization and Clinical Research: The Indiana Experience. [PubMed: 1857137]

45.

Ebert, TH. Initiating a Firm System at a Community Teaching Hospital. [PubMed: 1857138]

This statement was originally published as: Health care delivery research using hospital firms. Workshop summary; 1990 Apr 30-May 1. Bethesda (MD): National Institutes of Health, Office of Medical Applications of Research; [1990]:17.

For making bibliographic reference to the statement in the electronic form displayed here, it is recommended that the following format be used: Health care delivery research using hospital firms. NIH Technol Assess Statement Online 1990 Apr 30-May 1[cited year month day];(5).