What is JOpt?

JOpt is an open source Java wrapper that provides objects like Variable, Constraint, and Term and lets you express your linear or mixed integer programs in a natural manner, while remaining agnostic to the details of the solver backend. JOpt is not a solver. Rather, it requires a solver such as CPlex or the free LPSolve to operate. In the future, we aim to provide backend support for at least one free solver, such as hooks into the GNU Linear Programming Kit, etc. This means that any program written with JOpt will work regardless of the solver actually licensed or downloaded by the end user.

JOpt also provides a round-robin load balancer. If you solve multiple concurrent LP/MIPs, and have access to multiple machines with a solver, JOpt can take advantage of your setup to get a linear speed increase.

Why JOpt?

• You do linear or mixed integer programming, but want to think in terms of variables and constraints.

• You want to automatically distribute and load balance your problems to one or more solver machines.

What is New?

Since the last release:

- infeasibility analysis (when using CPLEX); see MIPInfeasibleException
- you can propose initial values to a MIP, which may speed up MIP solution (when using CPLEX); see IMIP.*SolveParam functions

Instructions

JOpt ships as one package (Jar File) that contains both a server-side component (which runs the actual solver) and a client-side component (which you integrate with your program). Jopt actually runs in one of two modes:

• Unified Mode: The client and server sit on the same machine, along with CPLEX or Other Solver. This mode is very appropriate if you are running a few LARGE MIPs/LPs, as it avoids serialization overhead. This mode requires you to know where CPLEX and JAVA are installed on your machine.

• Client/Server Mode: The client sits anywhere, and the server sits on the same machine (or on many machines) with CPLEX or Other Solver. This mode is very appropriate if you are running many small MIPs/LPs in parallel as it takes advantage of working on many parts of your sub-problem at once. The server-side component can sit on one or many powerful backend server machines, while the client can be anywhere. There is a serialization penalty as your MIP gets moved across the wire to the server machine before execution. If your MIPs/LPs are small in formulation size (number of constraints, variables) but long in solve time, this mode is most appropriate. This mode requires you to USE AN EXISTING JOPT SERVER or SET UP YOUR OWN JOPT SERVER, and also to install JAVA and APACHE ANT Is on your machine (Instructions below).

Note: The instructions below were written for CPLEX 8.1.

JOpt now requires CPLEX 10.0 or 10.1.

The paths below are for 8.1 -- please use v10 paths instead

Also note that paths below are for specific operating systems: each CPLEX install path is different depending on the OS/architecture

(old) 8.1 Install Instructions

I. Setting up JOpt to run in unified mode (Solver, Jopt-Server, Jopt-Client all on one machine)
This mode is very appropriate if you are running a few LARGE MIPs/LPs, as it avoids serialization overhead. If, on the other hand, you are running many small MIPs/LPs in parallel, it makes more sense to run in client/server mode. The tradeoff point depends on the size of your MIP/LP in terms of serialization cost, and the number of MIPs you run in parallel. (If you don't run any MIPs/LPs in parallel, it always makes sense to run in unified mode.)

In the zip file, there is an ExampleSimple.java -- look at how it can call SolverClient() with no parameters.

CPLEX USERS

Special Instructions for those at Harvard:

If you want to run CPLEX and your program on a windows machine, you should read the instructions below to get the gist of how to set up the environment on UNIX. You then should get the Windows distribution of CPLEX from /usr/ilog/dist/windows on econcs.eecs.harvard.edu, and follow these instructions (thanks to Haoqi Zhang!)

I finally got Cplex set up to run locally on my windows machine. What I had to do was: 1. Install concert13 with all the options 2. Install cplex with all the options 3. Install ilm (ilog license manager) with all the options 4. copy over /deas/arch/i386_linux24/vendor/depot/ilog/ilm/access.ilm into my ilog/ilm directory In eclipse, in the run, run... menu, I set the vm arguments to be -Djava.library.path=c:/ILOG/cplex81/bin/msvc7 (which came from installing cplex with the extra visual studio related files)

setenv ILOG_LICENSE_FILE /deas/arch/i386_linux24/vendor/depot/ilog/ilm/access.ilm
This environment variable is required by CPLEX to find the license that Harvard has purchased from ILOG Inc.

setenv PATH /deas/vendor/java/j2sdk/bin/:${PATH}
It turns out that the default 'java' on the deas blades for many users in not Sun Java, but is the Gnu Java. Gnu Java is not supported.

setenv CLASSPATH .:jopt.jar:/deas/arch/i386_linux24/vendor/depot/ilog/cplex81/lib/cplex.jar:
(As written, this line assumes that jopt.jar is in the current directory.) This line does three things: 1. Tells Java to look in your current directory (.) for your programs. 2. Tells Java to look in the current directory for jopt.jar, and 3. tells Java where to find the CPLEX specific jar (provided by ILog.)

At this point, when you type java -version you should see something like:
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0)
Java HotSpot(TM) Client VM (build 1.5.0, mixed mode)

Ok. All ready! Now you run

java -Djava.library.path=/deas/arch/i386_linux24/vendor/depot/ilog/cplex81/bin/i86_linux2_glibc2.3_gcc3.2 -Xmx1024m YourProgramHere

which tells Java 3 things: 1. Where to find CPLEX library files (the cplex.jar is just a wrapper to the CPLEX C library) 2. -Xmx1024m tells Java to use up to 1 Gig of memory 3. YourProgramHere is your program!

If all went well, you might experiment with ExampleSimple.java (remembering to change the call to SolverClient to be the version with no parameters, since you are running in unified mode.)

You should now skip to the “Logging” section down below.

OTHER SOLVER (Non-CPLEX) USERS

To be written.

II. Setting up JOpt to run in Client/Server Mode (Solver, Jopt-Server on different machines than Jopt-Client)

CPLEX USERS

PART 1. Setting up the Server for running in Client/Server Mode (CPLEX 8.0 or higher users)
This mode is very appropriate if you are running many small MIPs/LPs in parallel as it takes advantage of working on many parts of your sub-problem at once. If, on the other hand, you are running a few LARGE MIPs/LPs, it makes more sense to run in unifed mode (instructions above). The tradeoff point depends on the size of your MIP/LP in terms of serialization cost, and the number of MIPs you run in parallel. (If you don't run any MIPs/LPs in parallel, it always makes sense to run in unified mode.)

Special Instructions for those at Harvard:

We have set up CPLEX Server running on a few machines. This could change at any time. But you can start and stop the server via a web interface. See jeffsh for information. You still may want to set up your own server, if you don't want to share.

Verify that you have the same JDK version running on both the client and the servers. This may not be required, but is probably a good idea. JOpt has been tested and should be compatible with JDK 1.4 and 1.5.

Install Apache Ant 1.6.2 or higher.
If you are doing Java programming and you don't use Ant, then you are in for a treat. Ant is the de facto standard buildfile / makefile replacement and will make your life easier. See http://ant.apache.org/ You can check that you have a recent version installed in your path by typing:

% ant -version
Apache Ant version 1.6.2 compiled on September 28 2004

Install CPLEX.

Special Instructions for those at Harvard:
You may or may not find CPLEX on the shared server machines. If you don't, you need to get the CPLEX distribution onto your machine of choice.
- Presuming you have an eecs account, CPLEX can be found on econcs.eecs.harvard.edu at /usr/ilog/dist/linux/cplex81_1.linux.tar.gz. (If you do not have an eecs account, contact jeffsh.)
- After exploding the tar file, you need to copy the /usr/ilog/ilm/access.ilm on econcs.eecs.harvard.edu to your own directory.
- If you don't install CPLEX into /usr/ilog/ilm, then you also need to set your ILOG_LICENSE_FILE environment variable to point to the access.ilm file on your machine. (See step 4 of http://www.eecs.harvard.edu/~jeffsh/_cplexdoc/getstart/html/installation3.html#152115)

If using bash, you would do this by adding these commands to ~/.bash_profile:

if [ -f ~/.bashrc ]; then
source ~/.bashrc
fi

and this command to your ~/.bashrc:

export ILOG_LICENSE_FILE=/enter/path/to/your/license/file/here/access.ilm

- You should be able to run (from the cplexdir/ilm directory):

% ./ilmcheck /enter/path/to/your/license/file/here/access.ilm
Checking license file "access.ilm"
License file "access.ilm" is correct. the License file is correct.

The JOpt server install is actually just a mkdir jopt and then unzip the jopt zip file into this directory. Jopt comes pre-compiled and is in lib/jopt.jar. (The zip file also contains the logging config file, and example, the JOpt source code, this documentation, and the redistribution license file.)

Above the directory where you installed JOpt, create a symbolic link to your cplex installation directory. (eg: ln -s /usr/ilog/cplex81 cplexdir). Unless you change build.xml to teach it otherwise, JOpt running in server-mode assumes that the directory structure looks something like:

.\cplexdir (you made this symbolic link above)
.\cplexdir\bin\i86_linux2_glibc2.3_gcc3.2 (ships with CPLEX 8.1)
.\jopt (you mkdir'd this above)
.\jopt\jopt.jar (you unzip'd this above)

You are now done with the server side installation. To test, you should run:

ant runcplex

and you should see:

Buildfile: build.xml

runcplex:

[java] MAIN SolverServer: Binding server to port: 2000

If you don't want to bind to port 2000, change the line in build.xml:

<property name="port" value="2000"/>

That's it! Now read the section “Setting up the Client” Below

PART 2. Setting up the Client for running in Client/Server Mode (CPLEX 8.0 or higher users)

If you followed the instructions for setting up the server, you are basically there. The zip file comes packaged with two example files in the root directory, ExampleSimple.java and ExampleSimpleWrapper.java. To test them, in bash, you would type exactly:

% javac -d . -classpath lib/jopt.jar ExampleSimple.java
% java -classpath .:lib/jopt.jar edu.harvard.econcs.jopt.example.ExampleSimple

(Note that it is important that your classpath include both the JOpt directory (. in the above example) and the jopt.jar file) and you should see:

Usage: java edu.harvard.econcs.jopt.example.ExampleSimple ServerHostName ServerPortNumber

For instance, if you are running your server on drdoom.eecs.harvard.edu on port 2000, you would type:

% java -cp .:lib/jopt.jar edu.harvard.econcs.jopt.example.ExampleSimple drdoom 2000

and you should see:

Vars:
y
x
Constraints:
Constraint {[1.0*y] == 2.0}
Constraint {[1.0*x, -2.0*y] <= 5.0}
Objective Function: Max[ 2.0*x + 2.0*y ]
Constraint {[1.0*y] == 2.0}
objective: 22.0
x is 9.0
y is 2.0

OTHER SOLVER (Non-CPLEX) USERS

To be written.

III. Logging

The file edu/harvard/econcs/util/log.config controls the level of logging (screen output) that you will see while running JOpt, in both server-mode and client-mode. The default settings provide very little screen feedback. Increasing the JOpt debugging output, while helpful for debugging, will cause your program to run slowly.

If you are using CPLEX, note that CPLEX does it's own logging in addition to JOpt's screen output. This can be controlled using Jopt's MIP.setSolveParam(MIP_DISPLAY) and related control variables.

JOpt logging just uses Java 1.4 logging, but you can piggy back your own logging code in your own programming that can use what we've already written. In your file MyClass.java, add the line:

private static Log log = new Log(MyClass.class);

Then, after importing import edu.harvard.econcs.util.Log, you can make calls like:

log.debug(“This is a debug message”);

In a complicated program, picking the right level of message printing will be useful to you, since you can control what level of message gets printed by adding a line to the bottom of file log.config in local directory edu/harvard/econcs/util/ For instance, if MyClass is in package edu.harvard.work, you would add the line:

edu.harvard.work.MyClass = DEBUG

if you wanted to see all messages that were DEBUG severity or worse. Note that this overrides the default value that is listed in .level in that same log.config file.

Possible log levels (from most severe to least severe) are:

ERROR

WARNING

MAIN

INFO

TRACE

DEBUG

TRIVIAL

As a final performance note, if the debug output is in a speed-critical part of your code, it is worthwhile to write your code in this fashion:

if (log.isDebugEnabled())

log.debug("Skipping term: " + term);

IV. FAQ

Q. Help!
A. Read this FAQ, then go to https://list.eecs.harvard.edu/mailman/listinfo/jopt-users

LOGGING

Q: I get an error like:

Can't read log config stream

Jun 23, 2005 11:48:24 AM edu.harvard.econcs.util.Log info

INFO: Contacting Server for remote solver: econcs.eecs.harvard.edu:2001

A: You need to include Jopt directory in the classpath (so it can find log.config, which lives in the edu.harvard.econcs.util directory)

SPEED

Q. Let's say that I'm writing a single large MIP that is 128MB or so. Do I really need to serialize this over the wire, and re-serialize it before CPLEX solves it?

A. No! (This is what the “Unified Mode” is all about. See above instrunctions.) In summary, here is how to embed CPLEX calls into your client program. In effect, you eliminate the need for a server, and your client also runs the server code, which calls CPLEX. Rather than calling new SolverClient(host, port), just call new SolverClient() in your code. But, in order to do this, you also need to:

1. Help CPLEX by setting your ILOG_LICENSE_FILE enviornment variable.

2. include cplex.jar (ships with cplex) [and jopt.jar, of course] in your classpath.

2. provide Java with a pointer to the cplex .so file (Linux) or dll file (windows)

For example, if running csh on a machine where CPLEX is installed in /usr/ilog and jopt.jar is in your current directory:
setenv CLASSPATH .:jopt.jar:/usr/ilog/cplex81/lib/cplex.jar
followed by:
java -Djava.library.path=/usr/ilog/cplex81/bin/i86_linux2_glibc2.3_gcc3.2 -Xmx1024m YourClassFileHere

Q. Why is JOpt running slowly when I call solve? (My machine is pegged at 100%.)

A. Do you have debugging on? Debugging eats up a lot of local processor on a solve call for large MIPs, as it spits out the MIP it is trying to solve on the console screen. Look at log.config in your local directory edu/harvard/econcs/util/ and look for the line that starts “.level=”. This should be MAIN. Moreover, if you have a debugging function that is very expensive, you should wrap it with a call to log.isXXXEnabled. For instance, a call to log.trace(EXPENSIVE ARGUMENTS) should be wrapped with log.isTraceEnabled()

A. Are you running in client-server mode? Is it serializing a huge MIP? If so, consider unified mode.

A. Look at your process overhead -- is it the solver or JOpt that is the bottleneck? If it is the solver, not much we can do.

Q. Running in client-server mode, why is JOpt running slowly when I call solve? (My machine is not pegged.)

A. Does the server-side have debugging on? Debugging eats up a lot of server processor on a solve call for large MIPs, as it spits out all sorts of progress information to the console screen. Look at log.config in your remote directory edu/harvard/econcs/util/ and look for the line that starts “.level=”. This should be MAIN. If you did not set up the server-side component, ask the person who did to check this.

A. Look at your process overhead -- is it the solver or JOpt that is the bottleneck? If it is the solver, not much we can do.

Q. Could it be serialization time / transfer time that is making my program run slowly?

A. Yes, it is possible, especially if your client and server have a poor communications link, or if the MIP is 10s of megabytes in size or larger. But if you haven't already turned off debugging (see previous questions), try that first. Communicating a MIP problem with 100's of constraints and variables can usually be measured in ms, or a few seconds.

LPs/MIPs are transferred from your program to the server over Java RMI. This arrangement makes efficient use out of these server machines and doesn't let any one user hog a solver license. Our experience has been that for SMALL/MEDIUM problems, the Java RMI time is small compared to the solve time, and if you are using co-racked blades, the RMI serialization / transfer / deserialization time for both the client to server and server back to CPLEX is actually dominated by the time it takes to load CPLEX with the problem! For LARGE problems, Java RMI serialization sucks and you should run in unified mode.

MEMORY

Q. What memory things should I know about?

A. In your own launching of Java and in the build.xml file on the server side, you may need to specify a -Xmx parameter to avoid a java.lang.OutOfMemoryError for large problems. Run java -X for more information about this error. The default JOpt Server settings is 1GB.

SPECIAL FEATURES OF JOPT

Q. Special Features: What server parameters can I control?

A. In theory, this depends on whether you are using CPLEX, lpSolve, or some other backend solver. In practice, right now you can only edit parameters with CPLEX, and these parameters are a subset of what we thought was useful from the Cplex documentation. (See the Appendix in the CPLEX documentation. A copy for 8.1 for use in Harvard is at eecs.harvard.edu/~jeffsh/_cplexdoc

Q. Special Features: Seeding values: How do I know that seeding values is working?

A. When using CPLEX, you should see:

[java] MIP start values provide initial solution.

but you know that the starting values weren't accepted if you see:

[java] Warning: MIP start values ignored because of non-optimal

[java] subproblem optimization.

UNIFIED MODE

Q. When I run the server, I see the following:

java.lang.UnsatisfiedLinkError: no cplexXX in java.library.path
java.library.path must point to the directory containing the CPLEX shared library
try invoking java with java -Djava.library.path=...
java.lang.UnsatisfiedLinkError: CPXopenCPLEX
at ilog.cplex.Cplex.CPXopenCPLEX(Native Method)
at ilog.cplex.CplexI.init(CplexI.java:3601)
at ilog.cplex.CplexI.<init>(CplexI.java:287)
at ilog.cplex.IloCplex.<init>(IloCplex.java:8556)
at
edu.harvard.econcs.jopt.solver.impl.CPlexMIPSolver.solve(CPlexMIPSolver.java:40)
at (etc)

A. Either you haven't -Djava.library.path'd correctly, or this is likely a CPLEX installation issue. Make sure that you have installed more than just the static CPLEX library during CPLEX installation. Then, make sure you are running Java with something like: -Djava.library.path=C:\ILOG\cplex81\bin\msvc6

CLIENT/SERVER MODE

Q. When running the test program, it seems to hang when making a client connection.
A. If your server is multi-homed, this can occur. Sun's view of the world for RMI (used by JOpt) is that every multi-homed host has a 'most public' IP address. This is somewhat bogus in real life. One workaround is to edit build.xml to have a -Djava.rmi.server.hostname=REAL-IP-ADDRESS. We may have a cleaner workaround in the future.

Q. In client-server mode, why does my client get a:

ERROR SolverClient: Can't contact server

java.rmi.ConnectException: Connection refused to host: MACHINE NAME HERE; nested exception is:

java.net.ConnectException: Connection refused: connect

at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:574)

at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:185)

at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)

A. This means that the server, MACHINE NAME, isn't running, or isn't listening on the port that you expect. Are you sure the server is running? Are you sure you are contacting it on the right port?

Q. Client/Server: I just starting using JOpt again, and I know the server is running -- in fact, it has been running for a long while without being used. When the client calls solve now, it seems to hang. What is going on?

A. Restart the server. Not sure why this is, but guessing it's a RMI registry timeout issue.

Q. I am seeing weird behavior when running the client in Eclipse.

A. We think we fixed these, but Eclipse seems to use a different internal compiler that whatever you have installed, which screws you up if you don't have SerialUID since it can generate different SerialUID.

Q. Why do I get a:

ERROR SolverClient: Can't contact server

java.rmi.UnmarshalException: error unmarshalling return; nested exception is:

java.lang.ClassNotFoundException: edu.harvard.econcs.jopt.solver.server.SolverServer_Stub (no security manager: RMI class loader disabled)

A. You should never see the “no security manager” message anymore. This message used to mean that your JOpt client jar is out of date with the server jar. But we think we fixed that.

COMPILING JOPT

Q. When compiling JOpt from scratch, and I run RMIC, I get a message that RMIC can't be run.

A. This can occur if you didn't install the Sun JDK but only installed the Sun JRE. (For instance, on Windows: Is rmic.exe on your machine? Is it where java.exe lives? Are you sure? Do you have multiple java.exe's but only a few rmic.exes, indicating that you have installed several copies of the JDK/JRE in the past but that you are using the wrong one?)

Q. When compiling JOpt from scratch, and I run RMIC, I get a message like:

java.lang.UnsupportedClassVersionError: sun/rmi/rmic/Main (Unsupported major.minor version 49.0)

A. Make sure your rmic.exe and java.exe are from the same version. This can occur if you compile with 1.5 and rmic with 1.4, for instance.