Getting Started¶

Directory Structure¶

The following files and sub-directories will be found:

COPYRIGHT - Please get familiar with the Copyright before using this code!
README - Explains how to obtain, install and update the code.
doc/ - Provides additional documentation, including this user guide.
examples/ - Provides a careful set of examples that are known to work.
f90/ - The code base, makefiles and configuration scripts.
matlab/ - Auxiliary Matlab scripts for file conversion, and others.

Compiling ModEM¶

The build system for ModEM uses a ModEM configure script and the fmkmf.pl Perl script:

configure script (./f90/config/configure) - Calls fmkmf.pl with various arguments
fmkmf.pl - Called by Configure scripts and automatically generates a ModEM makefile.

Once you have run configure, you’ll be able to run make on the generated Makefile.

In the f90 directory there are some prototype Makefiles which can be used to compile ModEM without having to run the configure script; however, these Makefiles only create the Matrix-Free version of ModEM and you may want to compile the other two version SP/SP2. To do this you will need to run the configure script, which is explained below.

ModEM Dependencies¶

ModEM requires two libraries, the BLAS Library and the LAPACK library and optionally the MPI library.

Often BLAS and LAPACK are already available on your system and are linked automatically, but occasionally you might need to install or link them yourself. For instructions on building ModEM dependencies see Installing ModEM Dependencies.

Configure Scripts¶

There are a number of configure scripts inside ./f90/CONFIG in the form of Configure.***, Configure.SP.***, Configure.3D_MT.**, etc. These configure files are considered deprecated, and the configure script inside f90/CONFIG should be used instead; however, the deprecated configure scripts are left for backwards compatibility.

Creating Makefiles from configure¶

Note

By default, configure, with no optional arguments will generate an MPI Makefile with the SP2 forward solver.

To create a Makefile, run the configure script found in the f90 with arguments that match your system and the version of ModEM you want to use.

Currently, ModEM has three different forwards ‘flavors’:

MF - Matrix Free - Original/classic ModEM
SP - Hao’s Sparse Matrix
SP2 - (DEFAULT) - Hao’s Second Sparse Matrix - Fastest and has Hao’s
fine-grained capability

$ cd f90/
$ # By default ./CONFIG/configure creates a SP2 Makefile with MPI
$ ./CONFIG/configure Makefile gfortran

The Configure scripts has the following usage:

$ Usage: ./CONFIG/Configure [-h] [-u] [-g release|debug] [-m mpi|serial] [-l solver] [-f] makefile-name compiler
    -h Display full help message
    -u Display this usage message
    -g <release | debug> Specify whether to build with debug symbols or not
        release - Default - Create Makefile with full optimizations (i.e. -O3)
        debug   - Create Makefile with debug symbols (i.e. -g) and no optimizations
    -m <mpi* | serial> Specify MPI or serial
        mpi - Default - Builds with MPI
        serial - Builds Serially
    -l <MF | SP | SP2> Specify solver
        MF - Builds with MF
        SP - Builds with SP
        SP2 - Default - Builds with SP2
    -f Build with Fined-Grained -DFG on SP2
        Only able to build with SP2
    -s Build spherical version of ModEM

Running configure will create a Makefile that you specify in the first argument. In the above case the Makefile will be named Makefile. We can then call make on that Makefile

$ make

Examples Using Configure¶

A few examples using Configure

$ ./CONFIG/configure Makefile gfortran
$ # Generates a Makefile with GFortran options using MPI

$ ./CONFIG/configure -d debug Makefile ifort
$ # Generates an MPI Makefile with ifort debugging options

$ ./CONFIG/configure -m serial -l MF Makefile gfortran
$ # Generates a serial Makefile with GFortran with the MF solver

$ FC=ftn LDFLAGS=-L/home/username/install/lib ./CONFIG/configure Makefile gfortran
$ # Generates an SP2, MPI capabile makefile, adding LDFLAGS to the link step and
$ # specifies the compiler to be `ftn`.

$ FFLAGS='-mSSE2' ./CONFIG/configure Makefile ifort
$ # Create a SP2, MPI capabile makefile, with ifort/intel compiler options and add
$ # the '-mSSE2' to all compiling steps.

$ LDFLAGS="-L/path/to/install/lib/" ./CONFIG/configure Makefile gfortran
$ # Generates a Makefile with GFortran options using MPI and passes
$ # the location of /path/to/install/lib to the linker step

Configuring on Cray Systems¶

If you are running on a Cray system, you will need to use Cray’s Fortran compiler wrapper: ftn. You can easily specify the name of the compiler by setting the FC environment variable:

$ FC=ftn ./CONFIG/configure Makefile gfortran
$ # Or for intel:
$ FC=ftn ./CONFIG/configure Makefile ifort

Ensure BLAS and LAPACK are linked¶

If you get errors that you are not able to find -lblas or -llapack, but have them installed, you can use the LDFLAGS environment variable to tell the linker where to find them. You can specify this when you run configure:

$ LDFLAGS="-L/path/to/install/lib/" ./CONFIG/configure Makefile gfortran

If you have already created your makefile, you can also add the -L specification to the LIBS_PATH of the Macro-Def section of your Makefile:

# ------------------Macro-Defs---------------------
include Makefile.local
OBJDIR = ./objs/3D_MT/csemBuild
F90 = mpifort
FFLAGS = -cpp -DMPI -DCSEM_Dipole1D -O3 -ffree-line-length-none -dI -fallow-argument-mismatch
MPIFLAGS = -cpp -DMPI -DCSEM_Dipole1D
MODULE = -J $(OBJDIR)
LIBS_PATH = -L/path/to/install/lib
LIBS = -lblas -llapack

Compiling¶

Once you have generated your own makefile, or decided to use one of the defaults, you can compile ModEM by renaming that makefile to Makefile and then run make

$ cp Makefile.3D.MF.gnu Makefile
$ make clean # Not necessary from a fresh clone
$ make

Compiling with MPI¶

If you have MPI installed, and already have a serial makefile generated you can easily compile the MPI version of ModEM by altering the makefile. This is sometimes easier then creating a new makefile from the configuration scripts.

To compile an MPI version, change the F90 variable in your makefile to be an MPI compiler (e.g. mpifort) and add -DMPI to the MPIFLAGGS variable:

F90 = mpifort
MPIFLAGS = -DMPI ... omitting other flags

However, you can also generate an MPI makefile by running the configuration scripts and passing it MPI as the type.

Important

When running ModEM with MPI you must use at least 2 tasks and a max of (2 x nTransmitters) + 1 tasks (except for the SP2 version compiled with -DFG).

Running ModEM: The basics¶

Running the compiled program with no command line arguments will print a brief summary of usage

Output information to files, and progress report to screen (default).
Usage: Mod3DMT -[job] [args]

[READ_WRITE]
    -R rFile_Model rFile_Data [wFile_Model wFile_Data]
    Reads your input files and checks them for validity;
    optionally also writes them out
[FORWARD]
    -F rFile_Model rFile_Data wFile_Data [wFile_EMsoln rFile_fwdCtrl]
    Calculates the predicted data and saves the EM solution
[COMPUTE_J]
    -J rFile_Model rFile_Data wFile_Sens [rFile_fwdCtrl]
    Calculates and saves the full J(acobian)
[MULT_BY_J]
    -M rFile_Model rFile_dModel rFile_Data wFile_Data [rFile_fwdCtrl]
    Multiplies a model by J to create a data vector
[MULT_BY_J_T]
    -T rFile_Model rFile_Data wFile_dModel [rFile_fwdCtrl]
    Multiplies a data vector by J^T to create a model
[INVERSE]
    -I NLCG rFile_Model rFile_Data [lambda eps]
    Here, lambda = the initial damping parameter for inversion
        eps = misfit tolerance for the forward solver
    OR
    -I NLCG rFile_Model rFile_Data [rFile_invCtrl rFile_fwdCtrl]
    Optionally, may also supply
        the model covariance configuration file [rFile_Cov]
        the starting model parameter perturbation [rFile_dModel]
    Runs an inverse search to yield an inverse model at every iteration
[TEST_COV]
    -C rFile_Model wFile_Model [rFile_Cov]
    Applies the model covariance to produce a smooth model output
[TEST_ADJ]
    -A J rFile_Model rFile_dModel rFile_Data [wFile_Model wFile_Data]
    Tests the equality d^T J m = m^T J^T d for any model and data.
    Optionally, outputs J m and J^T d.
[TEST_SENS]
    -S rFile_Model rFile_dModel rFile_Data wFile_Data [wFile_Sens]
    Multiplies by the full Jacobian, row by row, to get d = J m.
    Compare to the output of [MULT_BY_J] to test [COMPUTE_J]

Optional final argument -v [debug|full|regular|compact|result|none]
indicates the desired level of output to screen and to files.

The -F [FORWARD] and -I [INVERSE] options will suffice for simple applications. For some of the more advanced applications discussed below the -C [TEST_COV] option is also needed. The -R [READ_WRITE] option can be useful for testing whether your input files are in the correct format. The -v option controls the level of program output; the default is “regular”; if you want to reduce outputs (“none” is not recommended!) you can experiment with other output levels; “full ” and “debug” would be primarily useful for developers. All of the other options (-J, -M, etc.) are primarily useful for development, testing, and some specialized applications that are beyond the scope of this basic documentation.

Sub-directory examples/ contains some sample input files with instructions for simple cases. These should be sufficient to test your installation, and also will provide a template for your own initial use. The doc/ sub-directory contains some further information on the ModEM system, beyond the summary of usage provided here.

All of the various options have a required set of arguments, which define input and output file names (rFile, and wFile in the usage summary, respectively). There are also optional arguments, mostly also specifying additional input or output file names. In all cases the order of arguments must be exactly as indicated in the “Usage” block, so to give an optional argument at the end of the list, all previous optional arguments must also be provided. Thus, for example, with the -F option, if you want to specify the input rFile_fwdCtrl file, you must also specify the output wFile_EMsoln file, even if you do not specifically want the full set of output EM fields, which this option will result in. Before providing a more detailed discussion of the various options, we consider file formats.

Using Docker to Compile and Run ModEM¶

For convenience, a Dockerfile has been provided where you can compile and run ModEM inside. To run the ModEM docker container, first ensure you have docker installed and running (Either have the docker daemon docker running, or Docker Desktop Running.

The Dockerfile builds an Ubuntu instance with GFortran, MPI, and all the other tools needed to compile and run ModEM.

Building¶

After you install Docker, you will need to build the docker container. Inside the ModEM directory (where the Dockerfile resides), run the following:

$ docker build . -t modem:latest --build-arg ncpus=2

You can specify the number of CPUs to use in the ncpus variable. This will set the number of CPUs to use when building MPICH.

Important

For personal machines, it’s best to set ncpus no greater than the max number of cores on your machines.

For shared login nodes you may want to use no more than 2-4 CPUs in order to not be detrimental to other users.

After running the above command Docker will create an image named modem:latest that you can then use to run; it will have all necessary libraries and compilers to compile and run ModEM.

Note

It can take anywhere from 3-15 minutes to build the ModEM docker image, depending on how fast your CPU is and how many CPUs you specify in ncpus.

Running and building ModEM in Docker¶

By default, the dockerfile automatically checks out the main branch of ModEM and places it in the root user’s home directory: /root/ModEM-main. This copy of ModEM has been provided for the convenience of new users.

First, let us start a container using the image that we built above:

$ docker volume create modem
$ docker run -it -v modem:/root/ModEM-main modem:latest

Normally, when running a docker container, the data created inside the container will be deleted when the container terminates. To prevent this, we can use a volume.

First, we need to create a named volume using docker volume create modem, then in our run command we attach that volume to /root/ModEM-main. This will save any data that we created inside /root/ModEM-main, but nowhere else.

After you terminate that container, and wish to start it again, you will need to specify your run command with the same arguments as above to reuse the data inside the volume, see the important note below.

The docker run command will run the image modem:latest and create a container for us to use The -it argument will tell Docker to start an interactive tty session and will drop you inside a bash shell inside the running container.

Now, we can move into ~/ModEM-main to configure, build and run ModEM:

$ cd ~/ModEM-main
$ cd f90/
$ # Compile with MPI, SP2 solver:
$ ./CONFIG/configure Makefile gfortran
$ # or compile serial, SP2 solver
$ ./CONFIG/configure -m serial Makefile gfortran
$ make
$ # Run ModEM serially or with MPI:
$ mpiexec -n 2 ./Mod3DMT ...
$ # Or if you compiled serial:
$ ./Mod3DMT ...

You can terminate this docker session by logging out by typing exit or you can use Ctrl-D.

Important

Again, if you want to re-run docker and reuse the files/folders/data you created before you will need to specify the volume that you used before

$ docker run -it -v modem:/root/ModEM-main modem:latest

Warning

Any data you create outside of the ModEM-main directory will not be saved. If you want to save data you will need to specify additional volumes or mounts. See Specifying mounts points.

For more information on persistent data inside a container see:

https://docs.docker.com/get-started/docker-concepts/running-containers/persisting-container-data/

Specifying mounts points¶

Note

For more information on Docker mounts see: https://docs.docker.com/engine/storage/bind-mounts/.

While it is helpful to have the ModEM-Main/ provided, it is also helpful to work with local version of ModEM you have on your host system. To do that we can use mounts, which will allow us to mirror a directory on our host system into the container.

We can add a mount by passing in a --mount option during our run command:

$ docker run --mount=type=bind,source=/absolute/path/to/ModEM,target=/root/ModEM -it modem:latest

The above command will run the container in the same way as above, but it will mount /absolute/path/to/ModEM/ from your host system to /root/ModEM to the container. Any changes you make in this folder, in either the container or the host machine will be present on the other machine.

It might be helpful to specify additional mount points, for instance it might be helpful to mount a work directory, such as the ModEM-Examples repository:

$ docker run --mount=type=bind,source=/abosolute/path/to/ModEM,target=/root/ModEM \
             --mount=type=bind,source=/home/users/uname/ModEM-Examples,target=/root/ModEM-Examples \
             -it modem

Important

Again, each time you run your modem container, you will need to specify volume and mount points.

Table of Contents

Previous topic

Next topic

This Page

Getting Started¶

Directory Structure¶

Compiling ModEM¶

ModEM Dependencies¶

Configure Scripts¶

Creating Makefiles from configure¶

Examples Using Configure¶

Configuring on Cray Systems¶

Ensure BLAS and LAPACK are linked¶

Compiling¶

Compiling with MPI¶

Running ModEM: The basics¶

Using Docker to Compile and Run ModEM¶

Building¶

Running and building ModEM in Docker¶

Specifying mounts points¶