what if three level turned out to be unnecessary?

4 replies [Last post]
James Lee's picture
Offline
Joined: 06/17/2013

After model comparison with 'anova()' command, variation in level 3 turned out to be not significant which means that 3 level approach can be inappropriate for the data.

In this case, do I have to adopt a level 2 mixed approach? The problem is that i guess some variables are seemingly higher level variables (e.g., institution),which was proven to be insignificant, while other variables are relatively lower level variables (e.g., female ratio). In my guess, integrating different characteristics into a mixed model can be problematic. Additionally, since more than one effectsize were reported in a study, some variable like 'outcome type' can be a covariate explaining within study variation.

Succinctly, is it acceptable to put different characteristics variables in a equation in 2 level approach, where 3 level test was insignificant?

Mike Cheung's picture
Offline
Joined: 10/08/2009
You may refer to pp. 28-29 of

You may refer to pp. 28-29 of Cheung (in press) for a discussion on whether a 2-level or a 3-level model should be used. Simply put, you should decide which model, 2-level or 3-level, is more appropriate. The decision is NOT based on the test of the heterogeneity.

Both level-2 and level-3 predictors are treated as a design matrix (fixed values) in 3-level meta-analysis. Unless they are highly correlated, I do not see any issue on including them in the same analysis. Could you explain in more details why it is a problem to include both level-2 and level-3 predictors in the same analysis?

Cheung, M.W.-L. (in press). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods. Available at https://dl.dropboxusercontent.com/u/25182759/Modeling%20dependent%20effe...

James Lee's picture
Offline
Joined: 06/17/2013
theoretical issue

Dear prof. Mike

Thanks for your kindness.

I thought it could cause theoretical rather than statistical problem. As far as i know, three level or two level should be determined by strong theoretical supports as well as statistical requirement. However, my situation is tricky in that theoretical supports are validated but statistical requirements are not.

I can choose either of them, but interpretation can be difficult if i choose two-level instead of three-level model including school and students variables. On the contrary, if i choose three-level in spite of invalid heterogeneity test,in practice, I guess, I can come across a rough situation to proceed the analysis with invalid class heterogeneity, which is nullify additional deviance test comparison etc.

So this is dilemma I have. Perhaps, i didn't catch the notion of design matrix you mentioned exactly. I will try to work on it and it will be appreciated if you give me a little hints.

Regards

Mike Cheung's picture
Offline
Joined: 10/08/2009
Dear James, Due to my limited

Dear James,

Due to my limited knowledge, I am not aware of any literature saying that a non-significant heterogeneity means that the random-effects model is invalid. Since you have read the literature, you should be in a better position than I do in making the judgement.

In regression analysis, the predictors are treated as fixed values. We called it a design matrix (http://en.wikipedia.org/wiki/Design_matrix). They are not treated as random variables. In SEM, the predictors are usually treated as random variables. In my previous reply, my point was that both level-2 and level-3 predictors are treated as fixed values in a design matrix. The model does not care whether the predictors are level-2 or level-3. Thus, I couldn't see why it is a problem to include both level-2 and level-3 predictors in the same analysis.

Mike

James Lee's picture
Offline
Joined: 06/17/2013
Dear Mike Honestly,I guess i

Dear Mike

Honestly,I guess i didn't catch the notion of design matrix exactly.
By the way, I didn't refer to any specific reference. It was more like my conjecture.

I just thought that if heterogeneity test can not solely determine which model is to be used
so if it is 'ok' to adopt 3 level meta analysis even if heterogeneity test turns out to be not heterogeneous in 3 level, it seems practically weird to me to add more fixed variables into the model and see if the heterogeneity is explained by added variables.

So although adopting 3 level model should not be solely determined by statistical test, but practically
I might as well switch 3 level model to 2 level model for the interpretation purpose.

This is a summarized point that I thought of.
Anyway, I will keep working on it. thanks for helpful advice

James