Amanzi Third Party Library (TPL) Build System#

Introduction#

This project builds the required and optional TPL software for the Amanzi code project. The build system is designed to configure and build TPLs to the requirements documented here https://software.lanl.gov/ascem/trac/wiki/Amanzi/Building/SoftwareRequirements.

Users are required to provide the following to build:

  • A recent and stable GCC or Intel compiler. We have successfully built with GNU versions 4.x to 9.x and Intel versions 11 through 17.

  • Mercurial version 1.8 or higher.

  • CMake version 3.11.4 or higher.

  • MPI installation with compiler wrappers. (mpicc, mpicxx, etc.)

  • BLAS/LAPACK built and tuned to the target architecture. See the Trilinos Configuration section for more information.

  • A location to install all the TPLs. The entire software stack will be installed in directories rooted to this directory. Since the download–patch-build-install cycle for each TPL is tied to a single target, the user must have read and write permission for this location.

  • The build directory will need approximately 2.1 Gb of space and the install directory tree requires 300 Mb.

On most UNIX-like OS (including Mac OSX) systems, the entire TPL software stack can be built using the bootstrap.sh shell script found in the config directory. The script searches for CMake and will download and build CMake 2.8.8, if it does not locate the cmake command. The script also searches for a working mpi installation and will download and build OpenMPI 1.4.4 if it does not locate mpi. At this time, the script does not build compilers, LAPACK or BLAS. See the compilers section on how to obtain a supported compiler and LAPACK for LAPACK and BLAS suggestions. See the MPI section on how to obtain and install other mpi implementations.

The bootstrap.sh script will create two build directories, one for the TPLs, TPL_BUILD, and one for amanzi, amanzi-build, in the directory the script is called from. Users can control the install location of the TPLs and Amanzi through command line arguments. The command

<amanzi source root>/bootstrap.sh --help

will print all the available options and default values.

If a TPL fails to download, configure, build or install, check the log files stored under a directory labeled:

<TPL name lowercase>/<TPL name lowercase>-timestamps.

Each step step is recorded with STDOUT and STDERR outputs stored in separate files. Any bug report related to the TPLs should include these log files.

Obtaining A Supported Compiler#

At this time, Amanzi can be built with recent versions of GCC, 4.x through 9.x, and Intel versions 11.* and 17.*. Since GCC is widely used by the development team and it is free, we recommend using one of the supported GCC compilers.

GCC On Mac OSX#

The default GCC install on Mac OSX does not include a Fortran compiler. Mac users should use the MacPorts software http://www.macports.org/ to install GCC. Once MacPorts is installed,

port install gcc

will install GCC in the /opt/local directory. Be aware that Apple requires these compiler names to be gcc-mp-x.x, g++-mp-x.x and gfortran-mp-x.x, not the standard names gcc, g++ and gfortran.

GCC On Ubuntu#

Use the apt-get utility,

apt-get install gcc

Building GCC From Source#

Please contact your local IT support before attempting to build GCC from source. The binaries may be available in a non-standard location.

Building GCC requires several precision libraries that must be installed before building GCC. See http://gcc.gnu.org/install/prerequisites.html for more information. Once these packages are installed, GCC must be configured with C, C++ and Fortran enabled. Documentation of the available configure options is here http://gcc.gnu.org/install/configure.html. The quickest way to determine the best configure options is to find a similar machine with a matching OS that has the target GCC already installed, and look at the verbose version output. The command

gcc -v

will print out the configure options used to build that GCC compiler. One important note to remember: DO NOT BUILD GCC IN THE SOURCE DIRECTORY. Create a separate build directory. The GCC build time is long. On most systems it will need more than 2 hours to complete.

Obtaining MPI#

Amanzi has been built against both MPICH2 and OpenMPI. Both are easy to build and install, but on clusters, users should contact their local support before building either one. Many vendors provide tuned and optimized MPI libraries that should be used to maximize performance. Otherwise, install either MPICH2 or OpenMPI.

MPICH2#

We recommend that users install a stable version of MPICH2. MPICH even minor versions (Ex: 1.4.1 minor version is 4) are stable. Odd minor versions are beta releases.

Steps to build and install MPICH2 are

  1. Download MPICH2

  2. Unpack the tar file and create a separate build directory. In this build directory, configure MPICH2 with:

    configure --prefix=<install prefix> \
          --enable-f77 \
          --enable-fc \
          --enable-cxx \
          --enable-romio \
          --enable-shared \
          --enable-static \
          CC=<full path to C compiler>
          CXX=<full path to C++ compiler>
          FC=<full path to Fortran compiler>
          F77=<full path to Fortran compiler that support F77>

  3. Build, test and install. MPICH does not support parallel builds.:

    make all; make test; make install

  4. Set environment variables, PATH, LD_LIBRARY_PATH on UNIX/Linux system:

    setenv PATH <install prefix>/bin:${PATH}
setenv LD_LIBRARY_PATH <install prefix>/lib:${LD_LIBRARY_PATH}

    or for bash shells:

    export PATH=<install prefix>/bin:${PATH}
export LD_LIBRARY_PATH=<install prefix>/lib:${LD_LIBRARY_PATH}

    For Mac OSX, set PATH and DYLD_LIBRARY_PATH environment variables:

    setenv PATH <install prefix>/bin:${PATH}
setenv DYLD_LIBRARY_PATH <install prefix>/lib:${DYLD_LIBRARY_PATH}

    or for bash shells:

    export PATH=<install prefix>/bin:${PATH}
export DYLD_LIBRARY_PATH=<install prefix>/lib:${DYLD_LIBRARY_PATH}

  5. Set environment variable MPI_ROOT. The bootstrap.sh script searches for compiler wrappers in this directory when this is set.:

    setenv MPI_ROOT <install prefix> or export MPI_ROOT=<install prefix>

OpenMPI#

Steps to build and install OpenMPI are

  1. Download OpenMPI http://www.open-mpi.org

  2. Unpack the tar file in a directory and create a separate build directory. In this build directory configure OpenMPI with

    configure \
          --prefix=<install prefix> \
          --enable-mpi-f77 \
          --enable-mpi-f90 \
          --enable-mpi-cxx \
          --enable-shared \
          --enable-static \
          CC=<full path to C compiler>
          CXX=<full path to C++ compiler>
          FC=<full path to Fortran compiler>
          F77=<full path to Fortran compiler that support F77>,

    where the compilers are defined explicitly on the command line.

  3. Build, test and install. OpenMPI does support parallel builds.:

    make all; make test; make install

  4. Set environment variables, PATH, LD_LIBRARY_PATH on UNIX/Linux system:

    setenv PATH <install prefix>/bin:${PATH}
setenv LD_LIBRARY_PATH <install prefix>/lib:${LD_LIBRARY_PATH}

    or for bash shells:

    export PATH=<install prefix>/bin:${PATH}
export LD_LIBRARY_PATH=<install prefix>/lib:${LD_LIBRARY_PATH}

    For Mac OSX, set PATH and DYLD_LIBRARY_PATH environment variables:

    setenv PATH <install prefix>/bin:${PATH}
setenv DYLD_LIBRARY_PATH <install prefix>/lib:${DYLD_LIBRARY_PATH}

    or for bash shells:

    export PATH=<install prefix>/bin:${PATH}
export DYLD_LIBRARY_PATH=<install prefix>/lib:${DYLD_LIBRARY_PATH}

  5. Set environment variable MPI_ROOT. The bootstrap.sh script searches for compiler wrappers in this directory when this is set.

    setenv MPI_ROOT <install prefix> or export MPI_ROOT=<install prefix>

Obtaining LAPACK/BLAS#

Most systems have LAPACK and BLAS installed in system directories. Building these libraries from source and optimizing for the target machine is difficult. We recommend that users attempt to build Amanzi before building LAPACK and BLAS from source. Trilnos, the package that requires both, detects the libraries in their build system and in many cases finds them automatically. If the Trilinos build system fails to find the libraries or if the user wants to build against non-standard LAPACK/BLAS distributions, please read the Trilinos configuration section.

If the user must build LAPACK and BLAS from source, we recommend ATLAS, a sofware package that automatically builds tuned BLAS libraries and a small subset of LAPACK. By default ATLAS does not build a full LAPACK distribution, but Amanzi requires a full LAPACK library. Read the ATLAS documentation on how to build ATLAS with a full LAPACK installation. ATLAS downloads are available at http://math-atlas.sourceforge.net.

Building TPLs and Amanzi Using Bootstrap.sh#

The script bootstrap.sh is located in the config directory. See bootstrap.sh --help for a full list of options and default settings. The script will create two subdirectories in the currect directory. The first is TPL_BUILD used for downloading and builing the TPLs. The second is amanzi-build used for building Amanzi. The TPLs and Amanzi will be installed in the directory specified by the option --prefix.

See the Quickstart Using Bootstrap section for instructions building using Bootstrap.sh.

Building TPLs From The Command Line using CMake#

We recommend that users run the provided bootstrap.sh script in the config directory, to build the TPLs. The instructions in this section are for advanced users and developers.

The steps to build entire suite of TPL software from the command line are

  1. Configure with cmake:

    cmake  \
       -D CMAKE_C_COMPILER=/full/path/mpicc \
       -D CMAKE_CXX_COMPILER=/full/path/mpicxx \
       -D CMAKE_Fortran_COMPILER=/full/path/mpif90 \
       -D TPL_INSTALL_PREFIX=/full/path/to/tpls
       /full/path/to/amanzi/config/SuperBuild

  2. Build with make (in parallel if desired):

    make -j n

    where n is the number of make jobs.

  3. Install with make:

    make install

The final step, make install, copies a CMake configuration file,amanzi-tpl-config.cmake, to the install location under share/cmake that is useful when building Amanzi. Passing this configuration to cmake using the -C option defines all the TPL locations needed to build amanzi.

For example

cmake -C /full/path/to/amanzi-tpl-config.cmake \
  [Additional CMake opts] /full/path/amanzi

will build and link amanzi linked against the TPLs built with this project, reducing the need for complicated build scripts.

Known Issues#

NetCDF fails to configure on Mac OSX#

This problem occurs when a user has HDF5 installed through MacPorts and uses the MacPorts OpenMPI. This is an issue with NetCDF’s configure script which does not allow the user to control the HDF5 prefix installation. Users can either install OpenMPI in another directory and use the wrappers in that directory (RECOMMENDED) or set ENABLE_NetCDF4:BOOL=FALSE while configuring SuperBuild from the command line.

Library parallelio (ASCEM-IO) missing archive index (MacOSX)#

This bug has only been reported on Macs. Users will experience this bug when linking the HDF5 test binaries in the Amanzi test suite. Despite a kludge in the shell script to address this when building ASCEM-IO, this problem persists. The work-around is to run ranlib on the installed libparallelio.a library.

Trilinos Configuration#

Amanzi relies on Trilinos for basic distributed data structures, input file handling and non-linear solvers. It is critical, for performance reasons, that the Trilinos build point to tuned BLAS and LAPACK libraries. On most systems, the BLAS and LAPACK libraries are found in standard system libraries (/usr/lib or /usr/lib64). However on large scale clusters these libraries are typically generic implementations and will not provide optimal performance. Such machines have tuned BLAS/LAPACK libraries. Please consult the support documentation for these machines to find the recommended libraries.

To guide the search for BLAS/LAPACK libraries, add -D ENABLE_BLA_Search:BOOL=TRUE to your cmake options. The CMake mechanism (FindBLAS and FindLAPACK) searches through a list of known vendor supplied BLAS/LAPACK libraries. Unless, BLA_VENDOR is set, all possible vendors are searched, and the search ceases once a valid BLAS/LAPACK library is found. For the search to be successful, the library must be located in a standard installation location or defined in the user’s environment variables LD_LIBRARY_PATH or DYLD_LIBRARY_PATH.

Supported BLAS/LAPACK Vendors:

Vendor

Description

ATLAS

Automatically Tuned Linear Algebra Software

PhiPACK

PHiPAC (Portable High Performance ANSI C)

CXML

HP Libraries

DXML

??

SunPerf

Sun Performance Libraries

SCSL

SGI/Cray Performance Libraries

SGIMATH

??

IBMESSL

??

ACML

AMD Core Math Library

ACML_MP

AMD Core Math Library multi-threaded

Apple

Mac OSX Accelerate Framework

NAS

??

Intel10_32

Intel MKL with 32bit int

Intel10_64lp

Intel MKL with 64bit int

Intel

Older versions of MKL

Generic

Generic BLAS

If the search fails during the TPL configuration or if the Trilinos configuration fails to locate these libraries, the user can override the search by explicitly defining the libraries with BLAS_LIBRARIES and LAPACK_LIBRARIES at configuration time. Multiple library names should be separated with a semi-colon. In some cases, Intel’s MKL libraries, require additional link flags when building Amanzi. Use the CMAKE_EXE_LINKER_FLAGS configuration variable to define these additional link flags when building Amanzi. Consult the documentation for the particular library for more information.

Trilinos Build Configuration Files#

SuperBuild contains Trilinos CMake configuration files for specific platforms and compilers. The files are located under the tpl-build/cmake directory. The file name should be defined while configuring tpl-build with the option -D Trilinos_Build_Config_File:STRING=<file>. This configuration file will be the last CMake configuration variable passed to Trilinos, thus all the definitions in this file will override previous configuration definitions.

Hopper (NERSC):
Compiler:

Intel

Config File:

trilinos-build-libsci-intel.cmake

Description:

Defines the LibSci libraries as BLAS/LAPACK libraries.

Hopper (NERSC):
Compiler:

GNU

Config File:

trilinos-build-libsci-gnu.cmake

Description:

Defines the LibSci libraries as BLAS/LAPACK libraries.