Using Matlab on the HPC Cluster

Matlab is installed on the HPC cluster, but in order to use it properly, your code needs to be written to call the MPI functions which take advantage of the cluster.  It should also be noted that each node that you run Matlab on will use a Matlab license, as well as a toolbox license for any toolbox functions which your code uses.  (See below for how to check for licenses)

Once you have an account on the cluster, you will need to make some changes to your user settings:

• Edit your .cshrc file add the following lines to the end of this file:
  setenv MATLAB_SHELL /bin/sh
  setenv MLM_LICENSE_FILE 27005@lic.science.oregonstate.edu
  *You will need to logout and back in for this change to take effect
• Create a new file, called startup.m, in the directory which contains your matlab code.  Add the following to this file:
  addpath /usr/local/apps64/MPI
  *You will also want to create addpath lines for any other directories containing functions your code uses, other than the regular Matlab toolboxes

Customize the Matlab MPI Settings:

• Make a local copy of the MatMPI_Comm_settings.m file:
  cp /usr/local/apps64/MPI/MatMPI_Comm_settings.m ~userid/<your source directory>
• edit this file, changing the following sections:
  machine_db_settings.email - Set this to your e-mail address
  machine_db_settings.jobid - Leave this as the default, or give your jobs a unique name
  machine_db_settings.qsub - Leave this as the default, or set it to one of the other queues on the cluster

Modify your Matlab code to understand MPI:

Use the following code as a guide to help you modify your Matlab program:

% Initialize MPI environment
MPI_Init;
comm = MPI_COMM_WORLD;
comm_size = MPI_Comm_size(comm);
my_rank = MPI_Comm_rank(comm);

disp(['my_rank: ',num2str(my_rank)]);
disp(['comm size: ',num2str(comm_size)]);
leader = 0;

% Load data file. This is used if data is created before simulations are to be run.
load('SimData.mat');

% Size of array
[n Nsims]=size(YsimOS); % YsimOS is the name of the data matrix read in the above load()
                        % statement. Nsims just needs to be the number of simulations
                        % to run.

% Break the job into pieces
chunksize = floor(Nsims / comm_size)
startchunk = chunksize * (my_rank) + 1
endchunk = startchunk + chunksize - 1
if (Nsims - endchunk > 0 & Nsims - endchunk < chunksize)
     endchunk = Nsims
end

% To prevent file locking problems, output to separate files
filename = strcat('SimResults.',num2str(my_rank));

for Sim=startchunk:endchunk
     disp(['Simulation ' num2str(Sim) ' of ' num2str(Nsims)])
            ...
            % Do some real work here
            ...
     fid=fopen(filename,'a'); % Open the output file to write results for current iteration.
     fprintf(fid,'%d\t',Sim);
     fprintf(fid,'\n');
     fclose(fid);
end

% Cleanup MPI environment
MPI_Finalize;
disp('SUCCESS');
if (my_rank ~= MatMPI_Host_rank(comm))
     exit;
end

 Create a Matlab Launcher Program:

* This needs to be put in the directory with your matlab code file(s) - *.m files
Use the following sample RUN.m file

% Delete left over MPI directory
MatMPI_Delete_all;
pause(2.0);

% Number of nodes to use on the cluster
num_nodes = 10;

% Name of matlab file to run on the cluster
% Change this line to be the name of your .m file
m_name = 'TestSim';

% Define machine names to run on cluster
machines = cell(1,num_nodes);

for node=1:num_nodes
     machines{node} = strcat('node',num2str(node));
end

% call the MPI commands to run the code on the cluster
eval ( MPI_Run(m_name, num_nodes, machines ) );

Submit Your MPI Matlab Job to the Cluster: 

matlab -nojvm -nosplash < RUN.m

* To use a specific version of Matlab, give the full path to the Matlab executable
** Before you can submit any jobs to the queue, make sure you have setup your SGE/MPI environment first by sourcing the appropriate settings.csh file for mpich v1 or settings-mpich2.csh for v2.

Checking for Matlab Licenses:

When your code runs on each node, a matlab license, and a license for any toolbox you use will be checked out.  In order to prevent your job on that node from failing completely, add code like the following to wait until a license becomes available:

avail=license('checkout','MATLAB'); 
while avail == 0
     disp('License not available, waiting...');
     pause(10.0);
     avail=license('checkout','MATLAB');
end
disp('Got the needed license'); 

This code checks for a Matlab license, but you will need to check for licenses for each of the toolboxes your code uses as well.  The names of the toolboxes to test for are:

Troubleshooting:

"qsub command not found"
     This is caused by not having the MATLAB_SHELL environment variable set.  Update your .cshrc file (see above).
or could be caused by not having the SGE/MPI environment setup up.

"Undefined function or variable 'MatMPI_Delete_all'"
     The Matlab MPI functions are not in the path.  Create/edit a startup.m file in your current directory and add the addpath lines from above.