Estimation and Inference

Thomas A Trikalinos; David C Hoaglin; Kevin M Small; Christopher H Schmid

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Trikalinos TA, Hoaglin DC, Small KM, et al. Evaluating Practices and Developing Tools for Comparative Effectiveness Reviews of Diagnostic Test Accuracy: Methods for the Joint Meta-Analysis of Multiple Tests [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2013 Jan.

Cover of Evaluating Practices and Developing Tools for Comparative Effectiveness Reviews of Diagnostic Test Accuracy

Evaluating Practices and Developing Tools for Comparative Effectiveness Reviews of Diagnostic Test Accuracy: Methods for the Joint Meta-Analysis of Multiple Tests [Internet].

Show details

Contents

< Prev Next >

Estimation and Inference

Separate (one test at a time) and joint meta-analysis models using the normal approximation can be fit using (restricted) maximum likelihood.

Separate meta-analyses models that use the binomial distribution can be fit in the generalized linear mixed models framework using routines readily available in general statistical packages such as xtmelogit in Stata or lmer in R. However, the joint meta-analysis models using the multinomial likelihood cannot be fit in these general routines. The available generalized linear mixed model (GLMM) packages in R, Stata and SAS do not allow the user to specify the random effects distribution in (24), where the random effects pertain to sums of the probabilities in Table 5. Optimizing the likelihood for joint meta-analysis using the multinomial likelihood outside a GLMM package is nontrivial, because it involves calculating complicated integrals numerically. Thus we did not develop routines for fitting this model. Instead we fitted the model using Markov Chain Monte Carlo (MCMC) methods in the Bayesian framework, as described later in this section.

Maximum Likelihood Estimation (Model Using the Normal Approximation)

To fit the normal approximation model, optimize the log likelihood

L o g L = \frac{1}{2} \sum_{k = 1}^{K} (log (| W_{k} |) - {D_{k}}^{'} W_{k} D_{k}),

where W_k = (∑_k + T)⁻¹ and $D_{k} = (\begin{matrix} {\hat{η}}_{k} - H \\ {\hat{ξ}}_{k} - Ξ \end{matrix})$ ; |W_k| denotes the determinant of W_k. The parameters to be estimated are the summary effects H and Ξ and the elements of the between-study covariance matrix T. Alternatively, one can optimize the restricted likelihood, which was the approach we used in the applied example:

{L o g L}^{*} = \frac{1}{2} \sum_{k = 1}^{K} (log (| W_{k} |) - {D_{k}}^{'} W_{k} D_{k}) + \frac{1}{2} log (| \sum_{k = 1}^{K} W_{k} |)

As mentioned previously, it is typical meta-analytic practice to consider the elements of ∑_k known, but calculate them from the data. Appendix A provides formulas for these calculations. The matrix equations for the log likelihood remain the same for bivariate meta-analysis of one test and for the joint meta-analysis of two or more tests.

By optimizing (35) or (36) we obtain the (restricted) maximum likelihood estimators Ĥ, $\hat{Ξ}$ and $\hat{T}$ . We also obtain the (2^M⁺¹ − 2)×(2^M⁺¹ − 2) estimated covariance matrix C = (c_ij) of the (Ĥ′, $\hat{Ξ}$ ′)′ as the inverse Hessian matrix.

Confidence Intervals

Confidence Intervals for the Summary Estimates H_m and Ξ_m

Confidence intervals for summary estimates are obtained in a similar manner for bivariate analyses of one test and for joint meta-analyses of two or more tests. Therefore, the formulas below are for M tests.

The 100(1−α)% simultaneous confidence interval (usually a 95% confidence interval) for H_m (the summary logit-TPR in test m) is given by:

({\hat{H}}_{m} - q_{α} \sqrt{c_{m m}}, {\hat{H}}_{m} + q_{α} \sqrt{c_{m m}}),

where c_mm is the variance of Ĥ_m, and q_α is the square root of the 100(1−α) percentile of the chi-squared distribution with 2^M⁺¹ − 2 degrees of freedom. This simultaneous confidence interval is a special case of Scheffé’s F-projections for multiple comparisons; it controls type I error for the family of all possible linear combinations of the estimated parameters.⁴⁷ The simultaneous confidence interval for Ξ_m (the summary logit-FPR in test m) is given by:

({\hat{Ξ}}_{m} - q_{α} \sqrt{c_{m + 2^{M} - 1, m + 2^{M} - 1}}, {\hat{Ξ}}_{m} + q_{α} \sqrt{c_{m + 2^{M} - 1, m + 2^{M} - 1}}),

where c_m_+2^M−1,m_+2^M−1 is the variance of $\hat{Ξ}$ _m.

Confidence Intervals for Differences H_i − H_j and Ξ_i − Ξ_j Between Summary Estimates of Two Tests

For two tests that have been applied to the same patients, one can either perform a meta-analysis for Test 1 and a separate one for Test 2, or a joint meta-analysis for the two tests. In either case, one can compare the diagnostic accuracy of the tests by calculating the difference between the logit-TPRs H₁ − H₂ and the difference between the logit-FPRs Ξ₁ − Ξ₂. The confidence intervals for such differences are calculated in different ways for separate versus joint meta-analyses of the two tests.

Confidence Intervals for Differences Based on Separate Meta-Analyses Per Test

Separate bivariate meta-analyses of the two tests ignore within-study correlations and treat the summary estimates of the two tests as independent. The resulting asymptotic confidence interval for the difference in logit TPR of tests i and j is

({\hat{H}}_{i} - {\hat{H}}_{j} - z_{α / 2} \sqrt{var ({\hat{H}}_{i}) + var ({\hat{H}}_{j})}, {\hat{H}}_{i} - {\hat{H}}_{j} + z_{α / 2} \sqrt{var ({\hat{H}}_{i}) + var ({\hat{H}}_{j})}),

where z_α_/2 is the upper α/2 percentile of the standard normal distribution. Because the confidence intervals in (39) ignore within-study correlations, their coverage differs from the nominal 100(1−α)%. Bonferroni’s inequality offers a simple adjustment to control the type I error. One substitutes z_α_/(2_f₎ for z_α_/2 in (39), where f is the number of comparisons of interest. It may be reasonable to consider f = 2M + M(M−1), which equals the number of estimated mean logit-TPRs and mean logit-FPRs plus the total number of pairwise differences among the mean logit-TPRs and plus the total number of pairwise differences among the mean logit-FPRs. (The above considers all other modeled quantities, such as the logit-JTPR and the logit-JFPR, as nuisance parameters that are not of interest.)

Confidence Intervals for Differences Based on Joint Meta-Analyses of All Tests

For joint multivariate meta-analyses of all tests, differences and simultaneous confidence intervals are obtained as follows. For convenience, write $β = (\begin{matrix} H \\ Ξ \end{matrix})$ ; that is, arrange the true summary logit-transformed quantities in a column vector. For a vector a = (a₁, …, a_2(2^M−1))′ let L(a, β) = a′β be a linear combination of the true summaries, and $L (a, \hat{β}) = a^{'} (\begin{matrix} \hat{H} \\ \hat{Ξ} \end{matrix})$ its estimate. Then 100(1−α)% simultaneous confidence intervals for all possible linear combinations are given by

(L (a, \hat{β}) - q_{α} \sqrt{(a^{'} Ca)}, L (a, \hat{β}) + q_{α} \sqrt{(a^{'} Ca)}) .

In particular, to estimate differences between the summary logit-TPRs of tests i and j, set a_i = 1, a_j = −1, and all other elements of a to 0. Then L(a, $\hat{β}$ ) = Ĥ_i − Ĥ_j, and the confidence interval in (40) becomes

({\hat{H}}_{i} - {\hat{H}}_{j} - q_{α} \sqrt{c_{i i} + c_{j j} - 2 c_{i j}}, {\hat{H}}_{i} - {\hat{H}}_{j} + q_{α} \sqrt{c_{i i} + c_{j j} - 2 c_{i j}}) .

In an analogous manner, to estimate differences between summary logit-FPRs for tests i and j, set a_i_+2^M−1 = 1, a_j_+2^M−1 = 1, and all other elements of a to 0, and proceed as in (40) to obtain

\begin{matrix} ({\hat{Ξ}}_{i} - {\hat{Ξ}}_{j} - q_{α} \sqrt{c_{i + 2^{M} - 1, i + 2^{M} - 1} + c_{j + 2^{M} - 1, j + 2^{M} - 1} - 2 c_{i + 2^{M} - 1, j + 2^{M} - 1}}, \\ {\hat{Ξ}}_{i} - {\hat{Ξ}}_{j} + q_{α} \sqrt{c_{i + 2^{M} - 1, i + 2^{M} - 1} + c_{j + 2^{M} - 1, j + 2^{M} - 1} - 2 c_{i + 2^{M} - 1, j + 2^{M} - 1}}) . \end{matrix}

MCMC Estimation and Credible Intervals for Models Using Discrete Likelihoods

We fit models using the binomial and multinomial distributions at the within-study level with MCMC methods. To this end, and in addition to equations in the Models and Estimation chapter, we specified vague prior distributions for the following modeled parameters.

The true means were assigned independent vague normal priors:

(\begin{matrix} H \\ Ξ \end{matrix}) ~ N (0, 10^{6} \cdot I_{6}),

where I₆ is the 6 × 6 identity matrix.

To assign priors for the covariance matrix T we use the factorization T = diag(τ)R diag(τ) where diag(τ) is the diagonal matrix whose diagonal elements are the square roots of the variances of the η_k and ξ_k and R is the correlation matrix corresponding to the covariances of the η_k and ξ_k. We assign independent uniform priors to the elements of τ, i.e., the standard deviations of the random effects):

τ_{m} ~ U (10^{- 4}, 5)

The priors for R must guarantee that the matrix is positive definite with elements between −1 and 1. We follow Lu and Ades⁴⁸ in factorizing R using the Cholesky decomposition for square symmetric matrices R = LL′, and in assigning specially constructed priors to the elements of the lower triangular matrix L (this is the spherical parameterization of Pinheiro and Bates⁴⁹):

L = [\begin{matrix} 1 & 0 & 0 & 0 & 0 & 0 \\ cos (ϕ_{21}) & sin (ϕ_{21}) & 0 & 0 & 0 & 0 \\ cos (ϕ_{31}) & \begin{array}{l} cos (ϕ_{32}) \cdot \\ sin (ϕ_{31}) \end{array} & \begin{array}{l} sin (ϕ_{32}) \cdot \\ sin (ϕ_{31}) \end{array} & 0 & 0 & 0 \\ cos (ϕ_{41}) & \begin{array}{l} cos (ϕ_{42}) \cdot \\ sin (ϕ_{41}) \end{array} & \begin{array}{l} cos (ϕ_{43}) \cdot \\ sin (ϕ_{42}) \cdot \\ sin (ϕ_{41}) \end{array} & \begin{array}{l} sin (ϕ_{43}) sin (ϕ_{42}) \cdot \\ sin (ϕ_{41}) \end{array} & 0 & 0 \\ cos (ϕ_{51}) & \begin{array}{l} cos (ϕ_{52}) \cdot \\ sin (ϕ_{51}) \end{array} & \begin{array}{l} cos (ϕ_{53}) \cdot \\ sin (ϕ_{52}) \cdot \\ sin (ϕ_{51}) \end{array} & \begin{array}{l} cos (ϕ_{54}) sin (ϕ_{53}) \cdot \\ sin (ϕ_{52}) sin (ϕ_{41}) \end{array} & \begin{array}{l} sin (ϕ_{54}) sin (ϕ_{53}) \cdot \\ sin (ϕ_{52}) sin (ϕ_{41}) \end{array} & 0 \\ cos (ϕ_{61}) & \begin{array}{l} cos (ϕ_{62}) \cdot \\ sin (ϕ_{61}) \end{array} & \begin{array}{l} cos (ϕ_{63}) \cdot \\ sin (ϕ_{62}) \cdot \\ sin (ϕ_{61}) \end{array} & \begin{array}{l} cos (ϕ_{61}) sin (ϕ_{63}) \cdot \\ sin (ϕ_{62}) sin (ϕ_{64}) \end{array} & \begin{array}{l} cos (ϕ_{65}) sin (ϕ_{64}) \cdot \\ sin (ϕ_{63}) sin (ϕ_{62}) \cdot \\ sin (ϕ_{61}) \end{array} & \begin{array}{l} sin (ϕ_{65}) sin (ϕ_{64}) \cdot \\ sin (ϕ_{63}) sin (ϕ_{62}) \cdot \\ sin (ϕ_{61}) \end{array} \end{matrix}]

Setting uniform independent priors for ϕ’s in the interval 0 to π = 3.14159... yields a prior for R in which all elements are between −1 and 1 and positive definiteness is guaranteed

ϕ_{i j} ~ U (0, π) .

See Lu and Ades for a short discussion on the density of the elements of R using the priors above.⁴⁸ See Pinheiro and Bates for a discussion of additional parameterizations.⁴⁹

95% Credible Intervals

With MCMC it is straighfrorward to obtain credible intervals for any quantity or any function of quantities explicitly, by simulation. In particular, we used 95% central credible intervals as the 2.5 and 97.5 percentile of the MCMC simulations.

Software and Computation

For the normal approximation models, the log likelihood in (35) and (36) for the unstructured variant of the T matrix can be optimized using routines such as mvmeta in Stata. We have developed our own Stata routines to optimize both the structured and the unstructured variant of T. (mvmeta uses a simple imputation of zero point estimates and large variances or covariances to simplify programming when handling studies with missing data; our routines do not need such imputations.) For convergence, starting values from fixed effect meta-analysis estimates appear to suffice. Note however, that the routine for the structured covariance matrix is not as robust: it failed to converge in the dataset used in this example (but does converge in other datasets). The optimization uses a modified Newton-Raphson algorithm. The routines are available from the authors upon request (see also www.cebm.brown.edu).

We ran MCMC analyses using JAGS version 3.1.0 through the R package rjags. We used three chains with a burn-in of at least 100,000 iterations and between 100,000 and 800,000 iterations for recording results. We monitored convergence with the Gelman-Rubin diagnostic for stochastic nodes corresponding to the meta-analysis means and the elements of their between-study covariance matrices. We declared convergence when the 97.5 percentile of the diagnostic was 1.10 or less for all monitored stochastic nodes, and provided that on visual inspection the traceplots of the MCMC chains were suggestive of good mixing.

Bookshelf ID: NBK148805

Contents

< Prev Next >

PubReader
Print View
Cite this Page
Trikalinos TA, Hoaglin DC, Small KM, et al. Evaluating Practices and Developing Tools for Comparative Effectiveness Reviews of Diagnostic Test Accuracy: Methods for the Joint Meta-Analysis of Multiple Tests [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2013 Jan. Estimation and Inference.
PDF version of this title (3.4M)

In this Page

Maximum Likelihood Estimation (Model Using the Normal Approximation)
Confidence Intervals
MCMC Estimation and Credible Intervals for Models Using Discrete Likelihoods
Software and Computation

Other titles in these collections

Recent Activity

Clear Turn Off Turn On

Estimation and Inference - Evaluating Practices and Developing Tools for Compara...
Estimation and Inference - Evaluating Practices and Developing Tools for Comparative Effectiveness Reviews of Diagnostic Test Accuracy

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Bookshelf

Evaluating Practices and Developing Tools for Comparative Effectiveness Reviews of Diagnostic Test Accuracy: Methods for the Joint Meta-Analysis of Multiple Tests [Internet].

Estimation and Inference

Maximum Likelihood Estimation (Model Using the Normal Approximation)

Confidence Intervals

Confidence Intervals for the Summary Estimates Hm and Ξm

Confidence Intervals for Differences Hi − Hj and Ξi − Ξj Between Summary Estimates of Two Tests

Confidence Intervals for Differences Based on Separate Meta-Analyses Per Test

Confidence Intervals for Differences Based on Joint Meta-Analyses of All Tests

MCMC Estimation and Credible Intervals for Models Using Discrete Likelihoods

95% Credible Intervals

Software and Computation

Views

In this Page

Other titles in these collections

Recent Activity

Confidence Intervals for the Summary Estimates H_m and Ξ_m

Confidence Intervals for Differences H_i − H_j and Ξ_i − Ξ_j Between Summary Estimates of Two Tests