Tutorials start too fast

32 replies [Last post]
Steve's picture
Offline
Joined: 07/30/2009

As we begin to get comments, I think we may be surprised at how rapidly our tutorials tend to move relative to the general public of SEM'ers. We run the risk of turning people off before they even begin.

The tutorial currently starts with far too much assumed knowledge. I think we will see that as we begin to hear comments from our beta testers. If we don't change our style from the current Mx (high level, lots of features, but no concessions for beginners), there is not much point in having written OpenMx. We will continue to appeal to a niche audience, primarily BG researchers who are already willing to learn Mx.

The main complaint I hear about Mx, and I hear this alot: "It must be a great program, but it's too hard for someone like me." I hear this from PhDs who are successfully running and publishing SEM. I even have heard it from people who teach SEM. I think we need to remember that on a day-to-day basis we operate in a rarified atmosphere. We could increase the user base of OpenMx a great deal by keeping its introductory tutorials simple enough that people who will may never hear of a Cholesky decomposition in their entire research career will not think "This is just like Mx, it's too hard for me".

Dorothy Bishop's picture
Offline
Joined: 02/04/2010
I'd be very grateful for some

I'd be very grateful for some feeback on the attached. (It's also on a Wiki, but I'm not sure where is best to put new modifications so that people are aware of them).

AttachmentSize
db_SMB.doc 208 KB
Dorothy Bishop's picture
Offline
Joined: 02/04/2010
As someone who has a rather

As someone who has a rather basic mastery of Mx and is totally new to R, I agree the documentation is too hard.
I am working through it slowly and annotating heavily for my own benefit and that of my students.
If someone can let me have a Word version of the documentation that I can edit, I'd be happy to make suggestions of how make it more friendly for the complete novice - and also to flag up points where I don't understand it.

mspiegel's picture
Offline
Joined: 07/31/2009
Unfortunately our

Unfortunately our documentation system exports to postscript, PDF, or HTML. I have been successful at downloading a page of the documentation in HTML format and opening the file in Microsoft Word. Saving the edited document in Word format should be OK, we'll be able to open it. Make sure to turn on track changes if editing the file in Word.

Dorothy Bishop's picture
Offline
Joined: 02/04/2010
I think that using HTML->txt

I think that using HTML->txt would be too fiddly for both of us, because formatting gets disrupted.
I will attach a document with suggested changes that I generated this afternoon. I could continue if people think this kind of additional material would be useful - I guess it could be irritating for the experts.
I am not sure whether I will persist - I had hoped that OpenMx would be more straightforward than Mx, and that I'd be able to get quickly up to speed, but my experience today suggests not - I'm only up to page 5 and had to spend a lot of time googling to work out R commands. I do find R pretty frustrating after working in Matlab. Colleagues tell me it's good once you get used to it, so I am willing to persevere a bit longer, but I'll wait to see what you think of the attached.

AttachmentSize
db comments on manual.doc 86 KB
Steve's picture
Offline
Joined: 07/30/2009
Your comments are really

Your comments are really helpful. By reading through, I can see some of where you are frustrated, but also can see how expert you are in other ways.

As I see it, we have three audiences for this documentation. Some, like you, are new to R but expert in Mx and SEM modeling in general.

A second group is familiar with R, but is new to SEM and/or Mx. Maybe they've used Amos or MPlus a little. Or maybe they don't know SEM modeling at all.

The third group knows neither R nor SEM.

I hope you continue with your comments, but it is very valuable that you make these comments as you go through the frustration of learning. It is hard for an expert to remember what frustrated them as a novice. And if you keep going, you'll be an expert soon.

One other comment. It is clear from your documentation that you are running on Windows and using the R GUI. We have another challenge, and that is for the documentation to work just as well for PC users as Mac users as Linux users.

I think you'll find that as you run R from the R GUI, it will be a good habit to do the following: First create a project directory that will contain all of your scripts, data, and workspace for a project. Second, copy your data and any pre-existing scripts into that directory. Third, start up a session of the R GUI just for that project and use the "Change Working Directory" option to that directory. Now, as you work, you won't need to use any paths in your read.table() commands or write.table() commands. Plus, when you quit, you can Save Workspace and you will be able to take up where you left off. Also, your script will continue to work even if you move your project directory.

Dorothy Bishop's picture
Offline
Joined: 02/04/2010
Thanks Steve. All advice most

Thanks Steve. All advice most welcome. I'm tempted to try and write a 'missing manual for real beginners' who don't know R, Mx, SEM, or behavior genetics, as that's the state a typical student is in. Would be great to get something structured so they could learn about all of these things by gradual introduction through examples. Trouble is, (a) like most people, I doubt I have the time to do it, and (b) I am largely self-taught and sometimes discover big holes in my knowledge, or worse, instances where I thought I understood something and find I got it wrong. So anything I wrote would need checking over by an expert.
I'll press on working through the manual and see if this is going to be a realistic thing to aim for. I have to make the decision whether to move to R for my own analyses and those of a couple of students and am not yet sure it will be worth the investment of effort.

Jeff's picture
Offline
Joined: 07/31/2009
Hi, Dorothy, Thanks for your

Hi, Dorothy,

Thanks for your input thus far. If you would like to start writing documentation that people can use (and check), you could create a page in our Wiki. We would love to have user-contributed documentation/tutorials.

I hope you don't give up on R quite yet. It does take some time to get the hang of, but here are some references that may be of value to you and your students:

http://cran.r-project.org/other-docs.html

Dorothy Bishop's picture
Offline
Joined: 02/04/2010
I've nearly finished some

I've nearly finished some tweaks to p 11-14 of manual to create a version for real beginners. Will follow your suggestion re putting it on a Wiki - provided I can understand how to do it - am also a Wiki novice.
The example of using a simulated dataset with correlated data is nice from the point of view of introducing people to simulations and to model-fitting in a relatively simple situation, but I'm having trouble explaining to the uninitiated exactly what it is that we are doing with the likelihood estimation, I think because I am confused myself. I can follow how to do it, but not why.
I think lots of novices will find it odd that you generate a dataset with a specified intercorrelation, but then evaluate the likelihood of the data. Obviously, it makes sense if you want to compare fit to models with different covariance between X and Y, but the initial ML estimation seems odd. I have no trouble with the notion that there will be a standard error around an estimate of a correlation etc - should this example be introduced in that kind of context? Advice would be welcome on this.

Steve's picture
Offline
Joined: 07/30/2009
The basic idea behind using

The basic idea behind using ML is that, assuming that the structure of our model is correct, we are looking for the parameter estimates for which the the likelihood of the data is at a maximum. This is one way to define "best" for obtaining "best estimates".

I'm hoping I'm understanding your question.

If the data are simulated, we know what the correct answer is, but OpenMx doesn't know the answer. OpenMx is estimating an answer, just the same as if it were given data from a real experiment.

Does that clear things up? Hopefully I'm not just further muddying the waters.

Dorothy Bishop's picture
Offline
Joined: 02/04/2010
Just to say that I have

Just to say that I have created a Wiki page called ideas for a simplified manual for beginners, which I will be adding to over this week. Your input welcome.

mspiegel's picture
Offline
Joined: 07/31/2009
My opinion is that since

My opinion is that since "pass by value" is the #1 cause of errors in OpenMx scripts, it should appear early in the user guide. I will admit the section could be re-written, but I do not agree that it needs a "hand wave" coverage that does not explain in detail the semantics. The #2 cause of errors is the R namespace versus the OpenMx namespace. This detail is actually a by-product of the fact that R does not pass-by-reference, but it is necessary to explain the namespace issue.

Steve's picture
Offline
Joined: 07/30/2009
I'd say that there is a

I'd say that there is a difference between "Hand Wave", which carries a negative connotation and "Minimally Sufficient in Order to Prevent Beginner Errors".  Like it or not, most of our target users just want to know the minimum thing they need to memorize and don't care why it is the case.

So, I'd recommend that we have something like:

We have found that many people make error X.  Here is all you need to remember in order to avoid X: "blah blah".  

For those of you who want to know why this is the way it is: "blah blah blah blah blah blah blah.... etc. etc. etc. Blah Blah"

Wish everyone were programmers and cared why, but I'd estimate that 90% don't care.  However, the 10% are more likely to post long posts on the forums... 

tbates's picture
Offline
Joined: 07/31/2009
Your point about common

Your point about common errors made me wonder if we know what the causes of error are?

For me it was (is :-) ) a whole set of things. I'd say

  1. Object name != "name" parameter
  2. Not knowing what needs to fill in the "..." part of slot-based functions.
  3. Things like not including an objective.
  4. Getting used to where @ and $ apply.

Then a pretty flat space of things like

  • Changing and re-assembling sub-models
  • How equating applies to labels and objects
  • How to use the name space inside algebras
  • mxEval needing a model name to specify the right object
  • Using labels to create definition variables ...

Just the normal learning of a complex system :-) but very very rewarding!

tbates's picture
Offline
Joined: 07/31/2009
Hi Michael, I think you're

Hi Michael, I think you're right about pass-by-value not being a hand-wave issue, but ideally the quick start shouldn't have much of a learning curve.

So my comment was in line with helping people get to issues as they need to and to avoid setting things up as hurdles when people don't even know enough to make an error.

For the quick start (as opposed to an in-depth study guide), I'd begin by cutting and pasting in the 6-line factor analysis and getting people to push return: then people know this thing works and motivation to learn goes up.

With something working under their belt, where name="innerName" is used in and algebra (say) while name=object is used to bring the object into a model, People will be thinking (if previous comments are a guide) that that seems a bit unwieldy - so a good time to explain that that's how it's done and is worth making into a habit - perhaps that the easiest way to get used to this is to adopt a personal rule for naming objects based on the "name" parameter (like pre-pending the type to the internal name to make memorable objectName: matrix "A" --> "matrixA".

Then have some pages or sidebars in the pdf on design philosophy, constraints imposed by pass-by-value, the name !="name" issue, and pass-by-value in depth.

 

I'm sure much of this is something where there is no easy answer, and the same issues come up no matter what, so no big deal.

Jeff's picture
Offline
Joined: 07/31/2009
Perhaps this should a

Perhaps this should a subsection in a "things you need to know before starting" section at the top.

pdeboeck's picture
Offline
Joined: 08/04/2009
After meeting with several

After meeting with several people here at KU, it is clear that across the board that the feeling is that there needs to be some large changes to the initial documentation. The first few examples don't seem particularly helpful for learning OpenMx (not to mention that the example don't run properly, and the data sets are not packaged with OpenMx). The impression I got was that the examples with regression/correlation and the CFA model on the website were much more helpful for starting to learn OpenMx. I'm not sure if it is a matter of going too fast, or whether the initial example do not build off of what is commonly known by people using other SEM packages.

msturgeapple's picture
Offline
Joined: 08/26/2009
Along these lines: As an

Along these lines:

As an extremely naive R user but a somewhat experienced SEM software user...can I make a suggestion on the quick start tutorial.

I installed OpenMx and it went through fine. Then I went to the quick start tutorial and of course pasted in the first command to see how it ran...

mxMatrix(
type="Full",
nrow=3,
ncol=1,
values=c(1,2,3),
name='A'
)
mxMatrix(
type="Full",
nrow=3,
ncol=1,
values=c(1,2,3),
name='A'
)

The problem is...it didn't run. I got this...

Error: could not find function "mxMatrix"
> mxMatrix(
+ type="Full",
+ nrow=3,
+ ncol=1,
+ values=c(1,2,3),
+ name='A'
+ )
Error: could not find function "mxMatrix"
> mxAlgebra(
+ A %*% t(A),
+ name='q1'
+ )
Error: could not find function "mxAlgebra"
> mxAlgebra(
+ t(A) %*% A,
+ name='q2'
+ )
Error: could not find function "mxAlgebra"
> mxAlgebra(
+ A * A,
+ name='q3'
+ )
Error: could not find function "mxAlgebra"
> mxAlgebra(
+ A + B,
+ name='q4'
+ )
Error: could not find function "mxAlgebra"
>

So that made me think that OpenMx didn't install properly. So I spent a while trying to re-install it and kept getting the same error message. Frustrating. Then...I found the library of commands in my OpenMx files that were installed and open one up and it ran. So it looks like I need to type in the require(OpenMx) command. So, just a thought that the first set of code in the users manual might be confusing to R-newbies such as myself.

Steve's picture
Offline
Joined: 07/30/2009
Good point!

Good point!

mspiegel's picture
Offline
Joined: 07/31/2009
I've added an OpenMx

I've added an OpenMx Beginners Guide to the web documentation. Or rather, I've checked it into the repository, so some magic needs to happen to get it on the website. It occurs after the OpenMx Introduction but before the mathic and pathic examples. The beginners guide at the moment has a mathic style example. I want to add a path style example, but I can't do it this evening.

mspiegel's picture
Offline
Joined: 07/31/2009
Beta testers: please try the

Beta testers: please try the Beginners Guide to OpenMx: http://openmx.psyc.virginia.edu/docs/OpenMx/latest/BeginnersGuide.html. It is our first attempt at slowing down the pace of the tutorials. Feedback is always appreciated.

msturgeapple's picture
Offline
Joined: 08/26/2009
Since I'm not versed in

Since I'm not versed in R...I'll try to help with the newbie documentation!

Comments on the new guide:

1. For me, the pass by value section is a little redundant because i do get some of the code there, but for someone not versed in R, that could be helpful.

2. The section after that is great. This might sound really needy...but I wonder if some sort of diagram to go with the five factor model might be helpful. Particularly since this is the first tutorial that a new user will encounter - the figure might help to solidify the concepts in the code.

What is an "s4 class"?

Finally...this thought might get towards what the purpose of OpenMx is. If the purpose is to be accessible to SEM users who typically might try AMOS, MPlus, ect...then I do get lost in this section of the quick users guide. Primarily because these users are probably not versed in the language of matrices and SEM. However, if OpenMx is for users who are more advanced in the workings of SEM and get matrices, etc..then the section reads fine for me. What type of person do you want the quick users guide to appeal to?

smcquillin's picture
Offline
Joined: 08/28/2009
I agree with msturgeapple's

I agree with msturgeapple's "finally" statement. Depending on the audience this guide may or may not be appropriate—although the recent changes are appreciated.

My opinion is that the quick users guide should be primarily introducing users to the applications of the statistical package, rather than explaining the logic behind the SEM & Matrices.

I would prefer to see an agenda for the wiki documentation so that users can begin providing some expository and interactive (i.e. using OpenMX) instruction on the logic and basics of SEM/Matrices. Perhaps some of this can be using OpenMX while other portions are just instructive. However, I think it is going to be important to set an agenda for the wiki so that users can begin to contribute in an orderly fashion.

tbates's picture
Offline
Joined: 07/31/2009
I'd say just dig in:make some

I'd say just dig in:make some wiki pages: empty ones if you want answers, or have questions. and lists of things that need saying.

I think it's a barrier that the wiki seems limited to html rather than mediawiki syntax - it's a lot more work to work on pages when everything has to have closing tags and such.

Can we get something here that does wiki syntax?

msturgeapple's picture
Offline
Joined: 08/26/2009
Does the quick start users

Does the quick start users guide repeat things...or am I going batty on a Saturday night?

Also, the first section of the quick start guide - honestly, I have no idea what that means or how to follow it and in reading it, I get the feeling that I need to read up on R before I can even attempt OpenMx. Which might be a bit discouraging. I'm happy to work with someone on the users guild outside of the boards so not to take up board space - try to make it more user friendly from the non-R sophisticated perspective, if that's the goal of the team. Let me know.

Also, I'm not sure if this is something I need to program in my computer or if it's a function of the website - but is it possible to have some of the documentation links open to a new page when I click on them? Otherwise I have to go back and forth when it would be easier to have a new page and the old page. I suppose I could just paste the URL in my browser...but if it's an easy thing just thought I would mention it.

tbates's picture
Offline
Joined: 07/31/2009
> the first section of the

> the first section of the quick start guide - honestly, I have no idea what that means

The quick-start dives into "pass by value" from the get go, which might be off-putting.

I wonder if we could say:

A key insight into using OpenMx is that you assemble your model out of a set of pieces (paths or matrices, data, perhaps some algebra), and only subsequently run the model using mxRun().

Because of this, it becomes important to understand how R treats the parts of a model that you are assembling. In particular, that R uses a style of information passing called "pass by value" as opposed to "pass by reference". You can proceed with the quick start guide following how to get a basic model run, but you should also spend some time reading the "Guide to Pass by value" in order to get the most out of OpenMx and avoid surprises that might trip you up later on.

... a nice model that can be typed in and run to get a feeling for how OpenMx works

tbates's picture
Offline
Joined: 07/31/2009
hi, > does the quick start

hi,
> does the quick start repeat things?
Can you point out the specific pieces that were repeated? (note that things are covered twice: once for writing a path-based model, and once for a matrix-based model, so those sections are similar, not repetitive by design)

> is it possible to have some of the documentation links open to a
> new page when I click on them?

Just right-click (or cntrl-click on a mac) the link, and say "open in new page"

Steve's picture
Offline
Joined: 07/30/2009
Good idea!!

Good idea!!

mspiegel's picture
Offline
Joined: 07/31/2009
Good feedback! I am hearing

Good feedback! I am hearing the Beginners Guide needs a path-style model section. That was going to be my next task. The plan was to take the model from the homepage (http://openmx.psyc.virginia.edu/) and put it in the Guide.

S4 objects and S4 classes are two related terms that are used to describe the dominant implementation of object-oriented programming in R. I am not much of an R historian, but the name "S4 objects" is somehow related to "S3 objects" which was an earlier implementation of object-oriented programming in R. One of the more concise descriptions I've seen of S4 is called "S4 Classes in 15 pages, more or less." Sadly, I have not found a good stand-alone introduction to S4 objects. The presentations online tend to assume the reader is familiar with some sort of object-oriented implementation.

tbates's picture
Offline
Joined: 07/31/2009
The R philosophy is "make

The R philosophy is "make standard things easy"
Ross Ihaka

I agree the learning curve needs to be flatter for standard things.

Part of that could be fixed for sure by slowing down and re-ordering. Beginning a tutorial on an SEM program with an example of matrix algebra and inner and outer products doesn't seem helpful to me.

Why not start with the idea of the variance of one column of data, then a correlation, with path diagrams at each point, and full downloadable and runnable cut and paste usable scripts, each of which answers a real question - the value of the desired parameter and its significance or confidence interval.

And side-by-side comparisons to getting the same thing done in R using var() and cor() to help people make connections...

With our new instrument, let people play songs they already know how to hum, and which they find useful.

Also, endeavor not to call things names like "q3" when "Matrix A.B" would be more informative. Characters are no longer valuable - we can spend them freely.

I'd add too that this assumes a ton of knowledge about matrices (like what an inner and outer product are) without defining them, giving any references, or mentioning why, if ever, someone doing an SEM would need to know about them.

Perhaps solution is to have an OpenMX tutorial based around getting SEM done, and then, somewhere else, a tutorial on using OpenMx for doing Matrix algebra and/or learning OpenMx if you happen to already know Mx.

wrevelle's picture
Offline
Joined: 08/04/2009
I disagree with Tim about the

I disagree with Tim about the speed. One goes to OpenMx to solve sem type problems. To have the tutorial start off showing how to do matrix multiplication (which is basic R function) seems off putting. One has to work through multiple examples doing trivial stuff before you get to why you are using OpenMx.

Tim's final suggestion for a straight sem example right off the top with a separate section on how to do matrix algrebra makes better sense.

I would like a simple model for a congeneric problem. Then I would like to solve it. Then do it for two groups. Then .... ?

Having path diagrams for what one is doing is a great suggestion.

Jim Steiger's suggestion to show the commands and then show the output is very good (at least from an R perspective). I want to make sure that what I am doing is producing what it should produce.

Bill

Steve's picture
Offline
Joined: 07/30/2009
Sounds to me like Bill and

Sounds to me like Bill and Tim are in agreement about slowing things down.

I also agree with Jim Steiger's suggestion about always including output. It helps people to see what should be produced.

tbates's picture
Offline
Joined: 07/31/2009
Hi Bill, did mean you agree

Hi Bill,
did mean you agree (rather than disagree?) with me about speed? I think we are on the same page...