Should version 0.5 be slower on some models?

19 replies [Last post]
rlucas's picture
Offline
Joined: 11/17/2009

I have been running some models that decompose multi-wave data into a stable trait component, an autoregressive trait component, and a state component. I have eight waves, with three indicators at each wave, 600 observations, and I'm using raw data (and there are missing data). With the data I have, these models (and some variants) can take a little while to run. But I've noticed that they actually run considerably slower with version .5.0 than with .3.0 (I hadn't upgraded until .5.0 came out). The results I get are identical (at least to about 7 decimals), but it takes more time. For instance, one of the models took 2.41 minutes in .3.0, versus 12.13 minutes in .5.0. Just curious whether with the performance optimizations, some model characteristics actually lead to a slowdown.

rlucas's picture
Offline
Joined: 11/17/2009
The speedup is definitely

The speedup is definitely great; 0.9 is much faster! However, I noticed one other issue. When I was using the earlier versions, I would often compare my results to Mplus, and everything was usually the same. However, I thought that some of the recent results that I'd been getting with 0.9.1 looked different than what I had remembered. So I installed 0.3.3 and compared the results from the script attached above in 0.3.3 and 0.9.1 and the estimates are very different. I don't have this exact same model setup in Mplus, so I didn't test it there, but I looked at the results for a similar model, and the results from 0.3.3 are different from those from 0.9.1, but those from 0.3.3 match what I get from Mplus. Unless I'm making some stupid mistake, I think you should be able to see the difference between 0.3.3 and 0.9.1 by running the script and data from my earlier message in each version.

mspiegel's picture
Offline
Joined: 07/31/2009
Hi Rich. You are definitely

Hi Rich. You are definitely in my list for the top five OpenMx testers on the planet. In models/nightly/startsTestNoMissing.R, the -2 log likelihood jumps from 3035.196 to 3143.61. The bug was introduced in revision 1384. We'll get working on fixing it.

mspiegel's picture
Offline
Joined: 07/31/2009
So, I introduced the bug into

So, I introduced the bug into the code base. And tbrick fixed the bug this evening. The patch will be a part of OpenMx 0.9.2, which will be released sometime over the weekend.

rlucas's picture
Offline
Joined: 11/17/2009
Great! I'll look for the

Great! I'll look for the update and will compare some of the models to the results I'm getting in Mplus. It's great to see these issues fixed so quickly!

neale's picture
Offline
Joined: 07/31/2009
Yes, the -2 log likelihood is

Yes, the -2 log likelihood is back to 3035.196, phew! Serious kudos to: rlucus for finding it, to mspiegel for admitting its introduction, and to tbrick for its fix. Its correction seems to come with a performance hit, is that so?

This one must go in the models/passing suite. I'm surprised that none of the others were affected. And yes, we need more ordinal examples as well.

mspiegel's picture
Offline
Joined: 07/31/2009
I reran the performance test

I reran the performance test suite and the results show no change in performance. I'll post the results later today.

mspiegel's picture
Offline
Joined: 07/31/2009
It's possible that our

It's possible that our performance test suite is missing some test cases. Is it possible to attach the script with either the real data or some simulated data? Here are the numbers of our current performance test suite: http://openmx.psyc.virginia.edu/wiki/performance-testing

rlucas's picture
Offline
Joined: 11/17/2009
Okay, I've done some more

Okay, I've done some more testing. I generated data with the same sample size, covariances, and means, and reran the models. I need to check with the people who collected the data whether it's okay to send these data (I would think it should be, but they are picky about even releasing summary stats without approval), but I had an update in the meantime. It looks like when there are no missing data or very little missing data 0.5.0 is much faster than 0.3.0. However, the more missing data there are, the bigger the slowdown for 0.5.0. So with a missing data pattern that is identical to the real data (in which about 1/3 of the data are missing--it's longitudinal data over 10 years), 0.5.0 is much slower than 0.3.0. See below (everything is in seconds, the first value is for 0.3.0, the second is for 0.5.0; sorry, I don't know how to make columns in the forum post):

No missing 237; 27
Some missing 247; 73
30% missing 453; 2715

I should note that these are different machines, but the hardware, OS (Ubuntu 10.04), and version of R (2.11.1) are identical. Also, I did run one of the models with 0.3.0 and then upgraded to 0.5.0 on the same machine and saw the same slowdown.

mspiegel's picture
Offline
Joined: 07/31/2009
That's good news in the sense

That's good news in the sense that we're narrowing in on the problem. My fingers are crossed that we can use the synthetic data in our open-source test suite. As a failsafe, we could always just test the script privately. But we advocate transparency in the development of OpenMx, and the public test suite allows others to verify the performance of our project.

rlucas's picture
Offline
Joined: 11/17/2009
Well, worst case scenario is

Well, worst case scenario is that I create synthetic data that is different enough from the real data to release but has the same issues. The only problem is that the model is a little finicky, so if the measures are too stable it gets difficult to separate a stable trait from a very stable autoregressive trait, which leads to estimation problems. So it's a lot easier to use data that have covariances that are similar to how the real measures behave. In any case, I'll check on using these synthetic data, but will move forward creating others if that doesn't work.

mspiegel's picture
Offline
Joined: 07/31/2009
There is now a wiki page with

There is now a wiki page with instructions on generating simulated data: http://openmx.psyc.virginia.edu/wiki/generating-simulated-data

rlucas's picture
Offline
Joined: 11/17/2009
Okay, sorry for the delay.

Okay, sorry for the delay. I'm attaching the script, a set of fake data with nothing missing, and a set that reproduces the pattern of missing data in the original. These are the files I used to get the numbers reported in my earlier post, so I assume that you should be able to reproduce the slowdown. This is one of my first attempts at writing a script in OpenMx, so if you have any suggestions, I'd love to hear them.

AttachmentSize
startsTest.r 9.81 KB
fakeSTARTSNoMissing.csv 198.3 KB
fakeSTARTSMissing.csv 145.29 KB
mspiegel's picture
Offline
Joined: 07/31/2009
We found the bug. It's been

We found the bug. It's been fixed in subversion r1387. We're looking at some test cases to eliminate any remaining correctness bugs in the library. Expect a 0.9 OpenMx release that will be implementation-complete very soon.

mspiegel's picture
Offline
Joined: 07/31/2009
Oh, there's definitely a

Oh, there's definitely a difference in performance. The execution times for OpenMx 0.3.0 are 6.16 minutes with no missing data, and 4.34 minutes with missing data. The execution times for the subversion repository are 38 seconds with no missing data, and 52 minutes with missing data. I think I'll start doing a binary search on the OpenMx releases to determine which release started slowing things down. Thank you (very much) for this test case.

mspiegel's picture
Offline
Joined: 07/31/2009
I've tracked down the bug to

I've tracked down the bug to somewhere between 0.3.3 and 0.4.1. That's narrowed it down to 60 subversion commits. Which should be 6 steps in a binary search. I'll take a look at it tomorrow.

neale's picture
Offline
Joined: 07/31/2009
Agreed. IMO there's

Agreed. IMO there's something wrong with the sorting by missingness patterns.

rlucas's picture
Offline
Joined: 11/17/2009
Sure, I could try to do that.

Sure, I could try to do that. I noticed in your call for test scripts that you said you had workarounds for data that can't be released. I can't release these data, so if you have a straightforward way of generating fake raw data that is similar to the real data, I could try that. Also, the script is a bit unwieldy right now because it is designed to handle data with any number of waves and any number of indicators, along with completely missing waves (through phantom variables). But I think I could come up with a stripped down version that just runs one model that I know has this issue.

mspiegel's picture
Offline
Joined: 07/31/2009
That sounds like a good idea.

That sounds like a good idea. We don't have a very sophisticated method for generating artificial data. In the past, I've determined the upper and lower bounds for each column in a data frame. Then I just start generating random numbers from a uniform distribution with the same upper and lower bounds on a column-by-column basis. A final step may be to introduce random NA values if you have missing data. Test whether this technique (that ignores any patterns in the actual data) is good enough to replicate the performance slowdown.

neale's picture
Offline
Joined: 07/31/2009
For most structural equation

For most structural equation models, latent class, and mixture distribution models I find mvrnorm() in the MASS library to be very useful. With empirical=TRUE you can generate data with exactly the same covariance structure and means as your sample. Or, the same covariance structure as the estimates from a model fitting run. Doing this with missing data is a bit more tricky, of course.