I applied the "multilevel.fit" function to a dataset of ~220 clusters, with an average cluster size of ~40. I compared the fit measures of the between-cluster level to those obtained when fitting a single-level CFA on the unbiased estimate of the between-cluster covariance matrix. The SRMR, CFI and TLI corresponded well, but there was a huge discrepancy between the two rmsea measures: For the two-level model, between-cluster rmsea was around .02. For the single-level model, it was around .25. I would expect for these methods to give slightly different estimates, but this seemed rather large to me.
I'd tend to say that the issue is with the multilevel.fit function, as: (1) the fit measures of the single-level model all indicated bad fit and (b) for the two-level model, only between-level rmsea indicated very good fit, while the other measures (especially CFI, TLI) indicated very bad fit. The average between-cluster intercorrelation of variables is relatively high, so I do not believe this discrepancy is due to CFI and TLI's downward "bias" in case of sparsely correlated variables.
If this line of reasoning is correct, one possibility might be that "multilevel.fit" divides with the incorrect sample size estimate. If I understand equations 14 and 15 by Ryu (2014) correctly, rmsea(between) is calculated by dividing by the number of clusters (J), while rmsea(within) is calculated by dividing by the total sample size minus the number of clusters (N-J).
To check this possibility, I multiplied the single-level rmsea estimate with the sqrt(J/N-J), which lead to an estimate around .04, much closer to the output of multilevel.fit.
I would be very thankful for some feedback, whether this might be the case.
Generally, thank you a lot for the great package!
Best wishes,
Maurizio
Ryu (2014): https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2014.00081/full
I applied the "multilevel.fit" function to a dataset of ~220 clusters, with an average cluster size of ~40. I compared the fit measures of the between-cluster level to those obtained when fitting a single-level CFA on the unbiased estimate of the between-cluster covariance matrix. The SRMR, CFI and TLI corresponded well, but there was a huge discrepancy between the two rmsea measures: For the two-level model, between-cluster rmsea was around .02. For the single-level model, it was around .25. I would expect for these methods to give slightly different estimates, but this seemed rather large to me.
I'd tend to say that the issue is with the multilevel.fit function, as: (1) the fit measures of the single-level model all indicated bad fit and (b) for the two-level model, only between-level rmsea indicated very good fit, while the other measures (especially CFI, TLI) indicated very bad fit. The average between-cluster intercorrelation of variables is relatively high, so I do not believe this discrepancy is due to CFI and TLI's downward "bias" in case of sparsely correlated variables.
If this line of reasoning is correct, one possibility might be that "multilevel.fit" divides with the incorrect sample size estimate. If I understand equations 14 and 15 by Ryu (2014) correctly, rmsea(between) is calculated by dividing by the number of clusters (J), while rmsea(within) is calculated by dividing by the total sample size minus the number of clusters (N-J).
To check this possibility, I multiplied the single-level rmsea estimate with the sqrt(J/N-J), which lead to an estimate around .04, much closer to the output of multilevel.fit.
I would be very thankful for some feedback, whether this might be the case.
Generally, thank you a lot for the great package!
Best wishes,
Maurizio
Ryu (2014): https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2014.00081/full