Versions 0.9.0 vs. 0.5.2

10 replies [Last post]
Mike Cheung's picture
Offline
Joined: 10/08/2009

Hello all,

I am fitting some variance known models (akin to meta-analysis). To handle models with possible covariates, I use a RAM model. The results based on 0.5.2 are comparable to those based on other statistical packages. After upgraded to 0.9.0, the results look odd.

The followings are an example. The -2LL of the 0.5.2 and 0.9.0 versions are 27.79916 and 29.02565, respectively. Any suggestions are highly appreciated. Thanks.

Regards,
Mike

yi <- c(-0.264,-0.230,0.166,0.173,0.225,0.291,0.309,0.435,0.476,0.617,0.651,0.718,0.740,0.745,0.758,0.922,0.938,0.962,1.522,1.844)
vi <- c(0.086,0.106,0.055,0.084,0.071,0.078,0.051,0.093,0.149,0.095,0.110,0.054,0.081,0.084,0.087,0.103,0.113,0.083,0.100,0.141)
my.df <- cbind(yi,vi)
test <- mxModel("test", mxData(observed=my.df, type="raw"),
mxMatrix("Zero", ncol=1, nrow=1, free=FALSE, values=0, name="A"),
mxMatrix("Full", ncol=1, nrow=1, free=FALSE, values=0, labels="data.vi", name="V"),
mxMatrix("Full", ncol=1, nrow=1, free=TRUE, values=0.1, lbound=0.0000001, name="Tau"),
mxMatrix("Iden", ncol=1, nrow=1, name="F", dimnames=list(c("yi"), c("yi"))),
mxMatrix("Full", ncol=1, nrow=1, free=T, values=0, name="M"),
mxAlgebra(V+Tau, name="S"),
mxRAMObjective("A", "S", "F", "M")
)
summary(mxRun(test))

######################## Version 0.9.0
## Running test
## data:
## $test.data
## yi vi
## Min. :-0.2640 Min. :0.05100
## 1st Qu.: 0.2745 1st Qu.:0.08025
## Median : 0.6340 Median :0.08650
## Mean : 0.5999 Mean :0.09120
## 3rd Qu.: 0.7990 3rd Qu.:0.10375
## Max. : 1.8440 Max. :0.14900

## The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).

## free parameters:
## name matrix row col Estimate Std.Error
## 1 Tau 1 1 0.1000 NaN
## 2 M 1 1 0.5999 9.115587e-10

## observed statistics: 20
## estimated parameters: 2
## degrees of freedom: 18
## -2 log likelihood: 29.02565
## saturated -2 log likelihood: NA
## number of observations: 20
## chi-square: NA
## p: NA
## AIC (Mx): -6.974355
## BIC (Mx): -12.44877
## adjusted BIC:
## RMSEA: NA
## timestamp: 2010-09-11 09:48:48
## frontend time: 0.3790069 secs
## backend time: 0.003436089 secs
## independent submodels time: 0.0001730919 secs
## wall clock time: 0.3826160 secs
## cpu time: 0.3826160 secs
## openmx version number: 0.9.0-1417

## Warning message:
## In model 'test' NPSOL returned a non-zero status code 1. The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).

######################## Version 0.5.2
## Running test
## data:
## $test.data
## yi vi
## Min. :-0.2640 Min. :0.05100
## 1st Qu.: 0.2745 1st Qu.:0.08025
## Median : 0.6340 Median :0.08650
## Mean : 0.5999 Mean :0.09120
## 3rd Qu.: 0.7990 3rd Qu.:0.10375
## Max. : 1.8440 Max. :0.14900

## The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).

## free parameters:
## name matrix row col Estimate Std.Error
## 1 Tau 1 1 0.1315197 0.07353606
## 2 M 1 1 0.5790348 0.10510040

## observed statistics: 20
## estimated parameters: 2
## degrees of freedom: 18
## -2 log likelihood: 27.79916
## saturated -2 log likelihood: NA
## number of observations: 20
## chi-square: NA
## p: NA
## AIC (Mx): -8.200837
## BIC (Mx): -13.06201
## adjusted BIC:
## RMSEA: NA
## timestamp: 2010-09-11 09:53:24
## frontend time: 0.3790886 secs
## backend time: 0.01191211 secs
## independent submodels time: 0.0001611710 secs
## wall clock time: 0.3911619 secs
## cpu time: 0.3911619 secs
## openmx version number: 0.5.2-1376

## Warning message:
## In model 'test' NPSOL returned a non-zero status code 1. The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).

mspiegel's picture
Offline
Joined: 07/31/2009
RAM and raw data is evaluated

RAM and raw data is evaluated as a FIML model. We had a bug in the -2 LL calculation that was corrected in 0.9.0. This bug affected all the 0.5.x releases (unless my memory serves me wrong). Could someone run this model with 0.3.3 and report the results? I'm not making a claim as to which value is correct, only that a third datapoint will help. Old releases can be downloaded at openmx.psych.virginia.edu/packages.

Mike Cheung's picture
Offline
Joined: 10/08/2009
Thanks for the prompt reply.

Thanks for the prompt reply. Attached are the output for 0.3.3. They are the same as those in 0.5.2. I have also run the analyses (but with a different model parametrization) in Mx 1.70a and Mplus 6. They are similar to those in 0.5.2.

Mx:
Parameter Estimate Std Error Lower 95% Upper 95%
------------------------------------------------------------
2 0.579035E+00 0.105101E+00 0.373036E+00 0.785034E+00
1 0.131520E+00 0.737999E-01 -0.131281E-01 0.276168E+00

Mplus 6:
MODEL RESULTS
Two-Tailed
Estimate S.E. Est./S.E. P-Value
Means
U 0.579 0.107 5.406 0.000
Variances
U 0.132 0.078 1.689 0.091

######################## Version 0.3.3
## Running test
## data:
## $test.data
## yi vi
## Min. :-0.2640 Min. :0.05100
## 1st Qu.: 0.2745 1st Qu.:0.08025
## Median : 0.6340 Median :0.08650
## Mean : 0.5999 Mean :0.09120
## 3rd Qu.: 0.7990 3rd Qu.:0.10375
## Max. : 1.8440 Max. :0.14900

## The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).

## free parameters:
## name matrix row col Estimate Std.Error
## 1 Tau 1 1 0.1315197 0.07353606
## 2 M 1 1 0.5790348 0.10510040

## observed statistics: 20
## estimated parameters: 2
## degrees of freedom: 18
## -2 log likelihood: 27.79916
## saturated -2 log likelihood: NA
## number of observations: 20
## chi-square: NA
## p: NA
## AIC (Mx): -8.200837
## BIC (Mx): -13.06201
## adjusted BIC:
## RMSEA: NA
## timestamp: 2010-09-11 11:40:24
## frontend time: 0.3829899 secs
## backend time: 0.01260304 secs
## independent submodels time: 9.202957e-05 secs
## wall clock time: 0.3956850 secs
## cpu time: 0.3956850 secs
## openmx version number: 0.3.3-1264

## Warning message:
## In model 'test' NPSOL returned a non-zero status code 1. The final iterate satisfies the optimality conditions to the accuracy requested, but the sequence of iterates has not yet converged. NPSOL was terminated because no further improvement could be made in the merit function (Mx status GREEN).

tbrick's picture
Offline
Joined: 07/31/2009
Thanks for catching this so

Thanks for catching this so quickly.

These discrepancies resulted from a bug in the latest optimization for FIML RAM models with definition variables.

The latest source revision, r1419, contains the fix.

Would you mind if we add your script to the test suite to make sure we catch any similar bugs that crop up in the future?

mspiegel's picture
Offline
Joined: 07/31/2009
This has been fixed in OpenMx

This has been fixed in OpenMx 0.9.1. Thanks for spotting the bug so quickly.

Mike Cheung's picture
Offline
Joined: 10/08/2009
Thanks a lot for all the

Thanks a lot for all the responses and fixes. The latest version (r1420) works fine!

Please feel free to use the script. The original data were adopted from Joop Hox's "Multilevel analysis: Techniques and Applications" at http://www.ats.ucla.edu/stat/hlm/examples/ma_hox/chapter8.htm

By the way, I built OpenMx r1420 from source. When I ran the analysis, it still indicates "0.9.0-1417." It would be better to reflect the correct versions being used.

mspiegel's picture
Offline
Joined: 07/31/2009
When OpenMx is compiled, the

When OpenMx is compiled, the version number is hard coded into the Makefile and the DESCRIPTION file. I've thought about using the command-line program 'svnversion' to extract the subversion revision number. It would be easier to place this number into the Makefile, and more difficult to place this number into the DESCRIPTION file. The only schemes we have come up with are way too complicated for the problem. So we just tell OpenMx developers to realize they are compiling from source and ignore the version number.

Mike Cheung's picture
Offline
Joined: 10/08/2009
Thanks. It is not a big

Thanks. It is not a big issue.

Ryne's picture
Offline
Joined: 07/31/2009
First, thanks for running

First, thanks for running your code with the new version. Finding problems like these really helps the project.

I've attached full output for runs under 0.5.2 and 0.9.0. While I can't identify the problem, I find the difference in numbers of iterations across the two versions interesting. The successful 0.5.2 run goes for 9 major iterations, while 0.9.0 runs for three. This is interesting because it means that 0.9.0 (a) got stuck, as indicated by the non-invertible hessian at iteration 3 and the invalid standard errors, but (b) successfully ran through two complete iterations before hitting the problem. Its unfortunate that we get the same NPSOL message for both runs, though the hessian/SE differences should indicate a problem to users who look at them.

Ryne

AttachmentSize
Results090.txt 1.44 KB
Results052.txt 1.53 KB
neale's picture
Offline
Joined: 07/31/2009
Hmmm. Seems like a problem

Hmmm. Seems like a problem concerning definition variables, but there is a definition variable example in the test suite which still runs fine. However, that example does not use a continuous definition variable.

Steve's picture
Offline
Joined: 07/30/2009
Thanks for the input, Mike!

Thanks for the input, Mike! Any chance we can add your script to our test suite so that we can catch any future discrepancies?