.. _TPLs: ============================================= Amanzi Third Party Library (TPL) Build System ============================================= Introduction ------------ This project builds the required and optional TPL software for the Amanzi code project. The build system is designed to configure and build TPLs to the requirements documented here https://software.lanl.gov/ascem/trac/wiki/Amanzi/Building/SoftwareRequirements. Users are required to provide the following to build: * A recent and stable GCC or Intel compiler. We have successfully built with GNU versions 4.x to 9.x and Intel versions 11 through 17. * Mercurial version 1.8 or higher. * CMake version 3.11.4 or higher. * MPI installation with compiler wrappers. (mpicc, mpicxx, etc.) * BLAS/LAPACK built and tuned to the target architecture. See the Trilinos Configuration section for more information. * A location to install all the TPLs. The entire software stack will be installed in directories rooted to this directory. Since the download--patch-build-install cycle for each TPL is tied to a single target, the user must have read and write permission for this location. * The build directory will need approximately 2.1 Gb of space and the install directory tree requires 300 Mb. On most UNIX-like OS (including Mac OSX) systems, the entire TPL software stack can be built using the ``bootstrap.sh`` shell script found in the config directory. The script searches for CMake and will download and build CMake 2.8.8, if it does not locate the cmake command. The script also searches for a working mpi installation and will download and build OpenMPI 1.4.4 if it does not locate mpi. At this time, the script does not build compilers, ``LAPACK`` or ``BLAS``. See the compilers_ section on how to obtain a supported compiler and LAPACK_ for ``LAPACK`` and ``BLAS`` suggestions. See the MPI_ section on how to obtain and install other mpi implementations. The ``bootstrap.sh`` script will create two build directories, one for the TPLs, TPL_BUILD, and one for amanzi, amanzi-build, in the directory the script is called from. Users can control the install location of the TPLs and Amanzi through command line arguments. The command :: /bootstrap.sh --help will print all the available options and default values. If a TPL fails to download, configure, build or install, check the log files stored under a directory labeled:: /-timestamps. Each step step is recorded with ``STDOUT`` and ``STDERR`` outputs stored in separate files. Any bug report related to the TPLs should include these log files. .. _compilers: Obtaining A Supported Compiler ++++++++++++++++++++++++++++++ At this time, Amanzi can be built with recent versions of GCC, 4.x through 9.x, and Intel versions 11.* and 17.*. Since GCC is widely used by the development team and it is free, we recommend using one of the supported GCC compilers. GCC On Mac OSX ************** The default GCC install on Mac OSX does not include a Fortran compiler. Mac users should use the MacPorts software http://www.macports.org/ to install GCC. Once MacPorts is installed, :: port install gcc will install GCC in the /opt/local directory. Be aware that Apple requires these compiler names to be gcc-mp-x.x, g++-mp-x.x and gfortran-mp-x.x, not the standard names gcc, g++ and gfortran. GCC On Ubuntu ************* Use the apt-get utility, :: apt-get install gcc Building GCC From Source ************************ Please contact your local IT support before attempting to build GCC from source. The binaries may be available in a non-standard location. Building GCC requires several precision libraries that must be installed before building GCC. See http://gcc.gnu.org/install/prerequisites.html for more information. Once these packages are installed, GCC must be configured with C, C++ and Fortran enabled. Documentation of the available configure options is here http://gcc.gnu.org/install/configure.html. The quickest way to determine the best configure options is to find a similar machine with a matching OS that has the target GCC already installed, and look at the verbose version output. The command :: gcc -v will print out the configure options used to build that GCC compiler. One important note to remember: DO NOT BUILD GCC IN THE SOURCE DIRECTORY. Create a separate build directory. The GCC build time is long. On most systems it will need more than 2 hours to complete. .. _MPI: Obtaining MPI +++++++++++++ Amanzi has been built against both MPICH2 and OpenMPI. Both are easy to build and install, but on clusters, users should contact their local support before building either one. Many vendors provide tuned and optimized MPI libraries that should be used to maximize performance. Otherwise, install either MPICH2 or OpenMPI. MPICH2 ****** We recommend that users install a stable version of MPICH2. MPICH even minor versions (Ex: 1.4.1 minor version is 4) are stable. Odd minor versions are beta releases. Steps to build and install MPICH2 are 1. Download `MPICH2 `_ 2. Unpack the tar file and create a separate build directory. In this build directory, configure MPICH2 with:: configure --prefix= \ --enable-f77 \ --enable-fc \ --enable-cxx \ --enable-romio \ --enable-shared \ --enable-static \ CC= CXX= FC= F77= 3. Build, test and install. MPICH does not support parallel builds.:: make all; make test; make install 4. Set environment variables, PATH, LD_LIBRARY_PATH on UNIX/Linux system:: setenv PATH /bin:${PATH} setenv LD_LIBRARY_PATH /lib:${LD_LIBRARY_PATH} or for ``bash`` shells:: export PATH=/bin:${PATH} export LD_LIBRARY_PATH=/lib:${LD_LIBRARY_PATH} For Mac OSX, set PATH and DYLD_LIBRARY_PATH environment variables:: setenv PATH /bin:${PATH} setenv DYLD_LIBRARY_PATH /lib:${DYLD_LIBRARY_PATH} or for ``bash`` shells:: export PATH=/bin:${PATH} export DYLD_LIBRARY_PATH=/lib:${DYLD_LIBRARY_PATH} 5. Set environment variable MPI_ROOT. The ``bootstrap.sh`` script searches for compiler wrappers in this directory when this is set.:: setenv MPI_ROOT or export MPI_ROOT= OpenMPI ******* Steps to build and install OpenMPI are 1. Download OpenMPI http://www.open-mpi.org 2. Unpack the tar file in a directory and create a separate build directory. In this build directory configure OpenMPI with :: configure \ --prefix= \ --enable-mpi-f77 \ --enable-mpi-f90 \ --enable-mpi-cxx \ --enable-shared \ --enable-static \ CC= CXX= FC= F77=, where the compilers are defined explicitly on the command line. 3. Build, test and install. OpenMPI does support parallel builds.:: make all; make test; make install 4. Set environment variables, PATH, LD_LIBRARY_PATH on UNIX/Linux system:: setenv PATH /bin:${PATH} setenv LD_LIBRARY_PATH /lib:${LD_LIBRARY_PATH} or for ``bash`` shells:: export PATH=/bin:${PATH} export LD_LIBRARY_PATH=/lib:${LD_LIBRARY_PATH} For Mac OSX, set PATH and DYLD_LIBRARY_PATH environment variables:: setenv PATH /bin:${PATH} setenv DYLD_LIBRARY_PATH /lib:${DYLD_LIBRARY_PATH} or for ``bash`` shells:: export PATH=/bin:${PATH} export DYLD_LIBRARY_PATH=/lib:${DYLD_LIBRARY_PATH} 5. Set environment variable MPI_ROOT. The bootstrap.sh script searches for compiler wrappers in this directory when this is set. :: setenv MPI_ROOT or export MPI_ROOT= .. _LAPACK: Obtaining LAPACK/BLAS +++++++++++++++++++++ Most systems have ``LAPACK`` and ``BLAS`` installed in system directories. Building these libraries from source and optimizing for the target machine is difficult. We recommend that users attempt to build Amanzi before building ``LAPACK`` and ``BLAS`` from source. Trilnos, the package that requires both, detects the libraries in their build system and in many cases finds them automatically. If the Trilinos build system fails to find the libraries or if the user wants to build against non-standard ``LAPACK``/``BLAS`` distributions, please read the Trilinos_ configuration section. If the user must build ``LAPACK`` and ``BLAS`` from source, we recommend ``ATLAS``, a sofware package that automatically builds tuned ``BLAS`` libraries and a small subset of ``LAPACK``. By default ``ATLAS`` does not build a full ``LAPACK`` distribution, but Amanzi requires a full ``LAPACK`` library. Read the ``ATLAS`` documentation on how to build ``ATLAS`` with a full ``LAPACK`` installation. ``ATLAS`` downloads are available at http://math-atlas.sourceforge.net. Building TPLs and Amanzi Using ``Bootstrap.sh`` ---------------------------------------------- The script ``bootstrap.sh`` is located in the ``config`` directory. See ``bootstrap.sh --help`` for a full list of options and default settings. The script will create two subdirectories in the currect directory. The first is TPL_BUILD used for downloading and builing the TPLs. The second is amanzi-build used for building Amanzi. The TPLs and Amanzi will be installed in the directory specified by the option ``--prefix``. See the :ref:`bootstrap` section for instructions building using ``Bootstrap.sh``. Building TPLs From The Command Line using CMake ----------------------------------------------- We recommend that users run the provided ``bootstrap.sh`` script in the ``config`` directory, to build the TPLs. The instructions in this section are for advanced users and developers. The steps to build entire suite of TPL software from the command line are 1. Configure with cmake:: cmake \ -D CMAKE_C_COMPILER=/full/path/mpicc \ -D CMAKE_CXX_COMPILER=/full/path/mpicxx \ -D CMAKE_Fortran_COMPILER=/full/path/mpif90 \ -D TPL_INSTALL_PREFIX=/full/path/to/tpls /full/path/to/amanzi/config/SuperBuild 2. Build with make (in parallel if desired):: make -j n where n is the number of make jobs. 3. Install with make:: make install The final step, make install, copies a CMake configuration file,amanzi-tpl-config.cmake, to the install location under share/cmake that is useful when building Amanzi. Passing this configuration to cmake using the -C option defines all the TPL locations needed to build amanzi. For example :: cmake -C /full/path/to/amanzi-tpl-config.cmake \ [Additional CMake opts] /full/path/amanzi will build and link amanzi linked against the TPLs built with this project, reducing the need for complicated build scripts. Known Issues ------------ NetCDF fails to configure on Mac OSX ++++++++++++++++++++++++++++++++++++ This problem occurs when a user has HDF5 installed through MacPorts and uses the MacPorts OpenMPI. This is an issue with NetCDF's configure script which does not allow the user to control the HDF5 prefix installation. Users can either install OpenMPI in another directory and use the wrappers in that directory (RECOMMENDED) or set ENABLE_NetCDF4:BOOL=FALSE while configuring SuperBuild from the command line. Library parallelio (ASCEM-IO) missing archive index (MacOSX) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ This bug has only been reported on Macs. Users will experience this bug when linking the HDF5 test binaries in the Amanzi test suite. Despite a kludge in the shell script to address this when building ASCEM-IO, this problem persists. The work-around is to run ranlib on the installed libparallelio.a library. .. _Trilinos: Trilinos Configuration ---------------------- Amanzi relies on Trilinos for basic distributed data structures, input file handling and non-linear solvers. It is critical, for performance reasons, that the Trilinos build point to tuned BLAS and LAPACK libraries. On most systems, the BLAS and LAPACK libraries are found in standard system libraries (/usr/lib or /usr/lib64). However on large scale clusters these libraries are typically generic implementations and will not provide optimal performance. Such machines have tuned BLAS/LAPACK libraries. Please consult the support documentation for these machines to find the recommended libraries. To guide the search for BLAS/LAPACK libraries, add ``-D ENABLE_BLA_Search:BOOL=TRUE`` to your cmake options. The CMake mechanism (FindBLAS and FindLAPACK) searches through a list of known vendor supplied BLAS/LAPACK libraries. Unless, ``BLA_VENDOR`` is set, all possible vendors are searched, and the search ceases once a valid BLAS/LAPACK library is found. For the search to be successful, the library must be located in a standard installation location or defined in the user's environment variables ``LD_LIBRARY_PATH`` or ``DYLD_LIBRARY_PATH``. Supported BLAS/LAPACK Vendors: ============= ============================================ Vendor Description ------------- -------------------------------------------- ATLAS Automatically Tuned Linear Algebra Software PhiPACK PHiPAC (Portable High Performance ANSI C) CXML HP Libraries DXML ?? SunPerf Sun Performance Libraries SCSL SGI/Cray Performance Libraries SGIMATH ?? IBMESSL ?? ACML AMD Core Math Library ACML_MP AMD Core Math Library multi-threaded Apple Mac OSX Accelerate Framework NAS ?? Intel10_32 Intel MKL with 32bit int Intel10_64lp Intel MKL with 64bit int Intel Older versions of MKL Generic Generic BLAS ============= ============================================ If the search fails during the TPL configuration or if the Trilinos configuration fails to locate these libraries, the user can override the search by explicitly defining the libraries with ``BLAS_LIBRARIES`` and ``LAPACK_LIBRARIES`` at configuration time. Multiple library names should be separated with a semi-colon. In some cases, Intel's MKL libraries, require additional link flags when building Amanzi. Use the ``CMAKE_EXE_LINKER_FLAGS`` configuration variable to define these additional link flags when building Amanzi. Consult the documentation for the particular library for more information. Trilinos Build Configuration Files ++++++++++++++++++++++++++++++++++ SuperBuild contains Trilinos CMake configuration files for specific platforms and compilers. The files are located under the tpl-build/cmake directory. The file name should be defined while configuring tpl-build with the option ``-D Trilinos_Build_Config_File:STRING=``. This configuration file will be the last CMake configuration variable passed to Trilinos, thus all the definitions in this file will override previous configuration definitions. :Hopper (NERSC): :Compiler: Intel :Config File: trilinos-build-libsci-intel.cmake :Description: Defines the LibSci libraries as BLAS/LAPACK libraries. :Hopper (NERSC): :Compiler: GNU :Config File: trilinos-build-libsci-gnu.cmake :Description: Defines the LibSci libraries as BLAS/LAPACK libraries.