No standard error

10 replies [Last post]
Mike Cheung's picture
Offline
Joined: 10/08/2009

Thanks. It is another milestone!

Unfortunately, it broke my metaSEM package. I am trying to fix it. I am testing a random-effects meta-analysis model. This model is equivalent to the one in the sources /openmx/models/passing/RAM-FIML-defvars.R There are only two parameters---one mean and one variance. The models are specified in OpenMx and in metaSEM that generates the code for OpenMx. The R syntax, outputs, and R objects are test2.0.R, “test2.0 output.txt”, and test2.0.txt, respectively.

As shown in “test2.0 output.txt”, the -2log likelihood and the parameter estimates are the same for them. However, there is no standard errors reported in the version created by metaSEM.

When I check the output slots, the calculated Hessian are different:
Specified in OpenMx:
$calculatedHessian
test.Tau[11] test.M[1,1]
test.Tau[1,1] 371.31703 -16.31094
test.M[1,1] -16.31094 181.77602

Specified in metaSEM:
$calculatedHessian
Intercept1 Tau2_1_1
Intercept1 -207769350869 144970978723
Tau2_1_1 144970978723 -404613761388

I have no idea on how to proceed. Any suggestions are highly appreciated! Thanks in advance.

AttachmentSize
test2.0 output.txt9.62 KB
test2.0.R1.37 KB
test2.0.txt4.56 KB
tbates's picture
Online
Joined: 07/31/2009
mxStandardizeRAMpaths

hi Mike, and thanks for metaSEM! It's a wonderful in its own right, as well as a great addition to the Openmx ecosystem!

SE reporting has changed in 2.0.beta

OpenMx 1.4 returned SEs in summary() only for free parameters - which meant that for models with equalities, only the SE for one arbitrary path was returned.

OpenMx 2.0 no longer gives the Std.SEs in summary, as it now computes them better in the new function

?mxStandardizeRAMpaths

This reports Std.Est and SEs for all non-zero parameters, doing the correct separate standardization for each equated path.

Is that the issue in your code?

Mike Cheung's picture
Offline
Joined: 10/08/2009
Hi, Timothy. Thanks. There is

Hi, Timothy.

Thanks. There is no parameters with equalities in the model. It does not seem to be the causes. When I tried mxStandardizeRAMpaths(), it did not work as it's not a RAM model.

Mike

RobK's picture
Offline
Joined: 04/19/2011
Maybe CSOLNP problem?

Hi, Dr. Cheung. I ran the R syntax in your test2.0.R (after uncommenting the necessary lines), and was not able to reproduce the problem you're reporting. I tried using both the OpenMx beta release and the most recent revision of the repository. FWIW, I'm running R 3.0.2 on 32-bit Windows. Which OS and R version are you using?

In any event, I suspect that you may have encountered a problem with the new optimizer, CSOLNP, which we know to be kind of buggy at present. Try inserting the following line,

mxOption(NULL,"Default optimizer", "NPSOL")

into your syntax, prior to any calls to mxModel() or meta(), and see if the problem persists.

EDIT: To clarify, the code I ran in an attempt to reproduce the problem was the following, which did allow me to compare the results of metaSEM.fit and OpenMx.fit:

yi <- c(-0.264,-0.230,0.166,0.173,0.225,0.291,0.309,0.435,0.476,0.617,0.651,0.718,0.740,0.745,0.758,0.922,0.938,0.962,1.522,1.844)
vi <- c(0.086,0.106,0.055,0.084,0.071,0.078,0.051,0.093,0.149,0.095,0.110,0.054,0.081,0.084,0.087,0.103,0.113,0.083,0.100,0.141)
 
library(metaSEM)
metaSEM.model <- meta(y=yi, v=vi)$mx.model
 
##### mx model generated from metaSEM
metaSEM.fit <- mxRun(metaSEM.model)
summary(metaSEM.fit)
 
##### mx model specified in OpenMx
OpenMx.model <- mxModel("test", type="default",
 mxMatrix("Full", ncol=1, nrow=1, free=F, values=0,   labels="data.vi", name="V"),
	mxMatrix("Full", ncol=1, nrow=1, free=T, values=0.1, lbound=0.0000001, name="Tau"),
	mxMatrix("Full", ncol=1, nrow=1, free=T, values=0,   name="M"),
	mxAlgebra(V+Tau, name="S"),
	mxFitFunctionML(),mxExpectationNormal(covariance="S", means="M", dimnames=c("yi")),
	mxData(observed=cbind(yi,vi), type="raw")
)
OpenMx.fit <- mxRun(OpenMx.model)
summary(OpenMx.fit)

Mike Cheung's picture
Offline
Joined: 10/08/2009
Hi, Robert. Thanks for your

Hi, Robert. Thanks for your help. I should have included more info. on the settings. I'm running R 3.1.0 on a 32-bit Linux with OpenMx 2.0.0-3529.

You are correct that the model in the uncommented lines works fine. The issue is that I have written a package called metaSEM to conduct meta-analysis based on the OpenMx. The metaSEM package translates models of meta-analysis into models in OpenMx. Many of my models do not work for OpenMx 2.0.0-3529.

I have just found that OpenMx reports SEs again when I rerun the model. I have also tried both CSOLNP and NPSOL. Please refer to the files test2a. I have tried a few sample datasets. Rerunning the models seems to work fine. But there should be something wrong here since these datasets work fine in OpenMx 1.4 without rerunning the models.

I have also attached a new example (test2b). test2b.txt stores the workspace of the data and OpenMx models. test2b.R and "test2b output.txt" are the R code and output, respectively. This is a multivariate meta-analysis with two effect sizes and a 2x2 covariance matrix of the random effects. CSOLNP works fine after rerunning the model, whereas NPSOL throws an error when rerunning the model.

AttachmentSize
test2a.R 1.76 KB
test2a output.txt 14.8 KB
test3.txt 4.14 KB
test3 output.txt 9.49 KB
test3.R 829 bytes
RobK's picture
Offline
Joined: 04/19/2011
Looks like a problem with the Linux beta

I just tried running your syntax in R 3.0.2 on Linux, with the beta version of OpenMx. This looks like a bug. Thanks for letting us know!

jpritikin's picture
Offline
Joined: 05/23/2012
fixed?

I don't think this is Linux specific. Can you try with SVN 3570?

RobK's picture
Offline
Joined: 04/19/2011
Yes, I think it's fixed

Hmm, yes, looks much better with revision #3570. I just installed #3570 under Linux and ran Dr. Cheung's code from his test2.0.R and test2a.R files.
EDIT: Although, re-running NPSOL.fit in test3.R still does fail. I think that's probably because the solution is perilously close to the boundary of the parameter space for certain values of the definition variables. Interesting that it doesn't trip up CSOLNP.

I agree that it is not Linux-specific, in that it is not platform-specific per se; instead, I think it's a problem that existed in the revision used to make the srcbuilds for the Linux installation of the beta. As I said in an email, we probably just need to roll out new srcbuilds (and binaries) of the beta to reflect the bug fixes we've made since we publicly announced it.

Dr. Cheung, are you able to build OpenMx from a local checkout from the source repository?

jpritikin's picture
Offline
Joined: 05/23/2012
when introduced?

> instead, I think it's a problem that existed in the revision used to make the srcbuilds for the Linux installation of the beta. As I said in an email, we probably just need to roll out new srcbuilds (and binaries) of the beta to reflect the bug fixes we've made since we publicly announced it.

I doubt that Rob. I don't see any changes in src/ComputeGD.cpp that could account for the bug for the last month, at least.

It might be the case that the optimizers do different things on different platforms due to precision differences. It is surprising that both optimizers behave the same way in this case.

RobK's picture
Offline
Joined: 04/19/2011
If you've checked and haven't

If you've checked and haven't seen any commits that would account for the bug, then I trust your judgment. The important thing is that it be fixed in trunk, which it now appears to be.

Mike Cheung's picture
Offline
Joined: 10/08/2009
Hi Robert and Joshua. Thanks

Hi Robert and Joshua.

Thanks for your assistance. I just installed #3572 in a 32-bit Linux. Test2a.R worked fine. Test3.R also worked fine except the same error in reruning it with NPSOL. I also tried it with version 1.4-3532. It also generated the same error message. Thus, it is at least consistent with version 1.4.

> ## Rerun the analysis
> NPSOL.fit <- mxRun(NPSOL.fit)
Running Meta analysis with ML
Error: The job for model 'Meta analysis with ML' exited abnormally with the error message: MxComputeGradientDescent: fitfunction Meta analysis with ML.fitfunction evaluated to nan (Expected covariance matrix is not positive-definite in data row 2)
Execution halted

Since I am going to fix my packge to reflect the changes in OpenMx 2.0, I would like to check whether the output slots are fixed. I will probably post more questions later.

Looking forward to basing my package in OpenMx 2.0!

Cheers,
Mike