a meta-analysis of weak studies is a weak meta-analysis

variance_hunterSenior Analyst

Meta-analysis sits high on most evidence hierarchies and gets treated as authoritative regardless of what went into it.

What it does well: increases precision when combining comparable, unbiased studies of the same question.

What it can't do: remove bias present in every input (if all the studies share a flaw the pooled estimate has it too, with a tighter interval, precision without accuracy). Fix publication bias (pooling a selected literature preserves the selection). Combine incomparable things (different models, doses, routes and endpoints pooled into one number describes nothing in particular).

So the question is always what went in. A meta-analysis of twelve small unblinded animal studies is twelve small unblinded animal studies with a summary statistic attached. Methodological point, applies to any meta-analysis you run into.

4 replies5 upvotesStudies & Evidence

4 replies

  • quiet_quantSenior Analyst

    precision without accuracy. a narrow interval around a biased estimate is more misleading than a wide one

    5 upvotes

  • newvialwhodisSenior Analyst

    i thought meta analysis was the top tier

    3 upvotes

  • MethodsSectionSenior Analyst

    Heterogeneity statistics are supposed to flag the third problem and get reported then ignored. High heterogeneity means the pooled estimate answers a question nobody asked.

    4 upvotes

  • same_tbhSenior Analyst

    garbage in garbage out with a confidence interval

    4 upvotes

Join the discussion โ€” reply to this thread or start your own.

Open in Community Q&A

Community discussions are educational and cover published research. Nothing here is individualised medical advice. Community guidelines