An introduction to the mxRun() socket interface begins with the checkpoint instructions  in the user guide.
Use the argument "useSocket=TRUE" in the invocation of mxRun(). The relevant mxOption() arguments are "Socket Server", "Socket Port", "Socket Units", and "Socket Count" (see
?mxOption). You will probably be setting "Socket Server" to the value "localhost".
In the socket interface, the optimizer takes the role of the connection client, and you must setup the connection server. Your GUI will act as the server, either in a non-blocking manner with some sort of polling, or another strategy is to use a separate thread in your GUI to manage the connection. You can fake this behavior in R with the command
con1 <- socketConnection(port = XYZ, server = TRUE). Note this is a blocking call.