"Fossies" - the Fresh Open Source Software Archive

Member "openmpi-3.1.6/README" (18 Mar 2020, 92892 Bytes) of package /linux/misc/openmpi-3.1.6.tar.bz2:

As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. See also the last Fossies "Diffs" side-by-side code changes report for "README": 4.0.0_vs_4.0.1.

    1 Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
    2                         University Research and Technology
    3                         Corporation.  All rights reserved.
    4 Copyright (c) 2004-2007 The University of Tennessee and The University
    5                         of Tennessee Research Foundation.  All rights
    6                         reserved.
    7 Copyright (c) 2004-2008 High Performance Computing Center Stuttgart,
    8                         University of Stuttgart.  All rights reserved.
    9 Copyright (c) 2004-2007 The Regents of the University of California.
   10                         All rights reserved.
   11 Copyright (c) 2006-2018 Cisco Systems, Inc.  All rights reserved.
   12 Copyright (c) 2006-2011 Mellanox Technologies. All rights reserved.
   13 Copyright (c) 2006-2012 Oracle and/or its affiliates.  All rights reserved.
   14 Copyright (c) 2007      Myricom, Inc.  All rights reserved.
   15 Copyright (c) 2008-2017 IBM Corporation.  All rights reserved.
   16 Copyright (c) 2010      Oak Ridge National Labs.  All rights reserved.
   17 Copyright (c) 2011      University of Houston. All rights reserved.
   18 Copyright (c) 2013-2017 Intel, Inc. All rights reserved.
   19 Copyright (c) 2015      NVIDIA Corporation.  All rights reserved.
   20 Copyright (c) 2017-2018 Los Alamos National Security, LLC.  All rights
   21                         reserved.
   22 Copyright (c) 2017      Research Organization for Information Science
   23                         and Technology (RIST). All rights reserved.
   27 Additional copyrights may follow
   29 $HEADER$
   31 ===========================================================================
   33 When submitting questions and problems, be sure to include as much
   34 extra information as possible.  This web page details all the
   35 information that we request in order to provide assistance:
   37      http://www.open-mpi.org/community/help/
   39 The best way to report bugs, send comments, or ask questions is to
   40 sign up on the user's and/or developer's mailing list (for user-level
   41 and developer-level questions; when in doubt, send to the user's
   42 list):
   44         users@lists.open-mpi.org
   45         devel@lists.open-mpi.org
   47 Because of spam, only subscribers are allowed to post to these lists
   48 (ensure that you subscribe with and post from exactly the same e-mail
   49 address -- joe@example.com is considered different than
   50 joe@mycomputer.example.com!).  Visit these pages to subscribe to the
   51 lists:
   53      http://lists.open-mpi.org/mailman/listinfo/users
   54      http://lists.open-mpi.org/mailman/listinfo/devel
   56 Thanks for your time.
   58 ===========================================================================
   60 Much, much more information is also available in the Open MPI FAQ:
   62     https://www.open-mpi.org/faq/
   64 ===========================================================================
   66 The following abbreviated list of release notes applies to this code
   67 base as of this writing (January 2020):
   69 General notes
   70 -------------
   72 - Open MPI now includes two public software layers: MPI and OpenSHMEM.
   73   Throughout this document, references to Open MPI implicitly include
   74   both of these layers. When distinction between these two layers is
   75   necessary, we will reference them as the "MPI" and "OpenSHMEM"
   76   layers respectively.
   78 - OpenSHMEM is a collaborative effort between academia, industry, and
   79   the U.S. Government to create a specification for a standardized API
   80   for parallel programming in the Partitioned Global Address Space
   81   (PGAS).  For more information about the OpenSHMEM project, including
   82   access to the current OpenSHMEM specification, please visit:
   84      http://openshmem.org/
   86   This OpenSHMEM implementation will only work in Linux environments
   87   with a restricted set of supported networks.
   89 - Open MPI includes support for a wide variety of supplemental
   90   hardware and software package.  When configuring Open MPI, you may
   91   need to supply additional flags to the "configure" script in order
   92   to tell Open MPI where the header files, libraries, and any other
   93   required files are located.  As such, running "configure" by itself
   94   may not include support for all the devices (etc.) that you expect,
   95   especially if their support headers / libraries are installed in
   96   non-standard locations.  Network interconnects are an easy example
   97   to discuss -- Libfabric and OpenFabrics networks, for example, both
   98   have supplemental headers and libraries that must be found before
   99   Open MPI can build support for them.  You must specify where these
  100   files are with the appropriate options to configure.  See the
  101   listing of configure command-line switches, below, for more details.
  103 - The majority of Open MPI's documentation is here in this file, the
  104   included man pages, and on the web site FAQ
  105   (https://www.open-mpi.org/).
  107 - Note that Open MPI documentation uses the word "component"
  108   frequently; the word "plugin" is probably more familiar to most
  109   users.  As such, end users can probably completely substitute the
  110   word "plugin" wherever you see "component" in our documentation.
  111   For what it's worth, we use the word "component" for historical
  112   reasons, mainly because it is part of our acronyms and internal API
  113   function calls.
  115 - The run-time systems that are currently supported are:
  116   - rsh / ssh
  117   - PBS Pro, Torque
  118   - Platform LSF (v7.0.2 and later)
  119   - SLURM
  120   - Cray XE, XC, and XK
  121   - Oracle Grid Engine (OGE) 6.1, 6.2 and open source Grid Engine
  123 - Systems that have been tested are:
  124   - Linux (various flavors/distros), 64 bit (x86), with gcc, Absoft,
  125     Intel, and Portland (*)
  126   - macOS (10.12), 64 bit (x85_64) with XCode compilers
  128   (*) Be sure to read the Compiler Notes, below.
  130 - Other systems have been lightly (but not fully tested):
  131   - Linux (various flavors/distros), 32 bit, with gcc
  132   - Cygwin 32 & 64 bit with gcc
  133   - ARMv6, ARMv7, ARMv8 (aarch64)
  134   - Other 64 bit platforms (e.g., Linux on PPC64)
  135   - Oracle Solaris 10 and 11, 32 and 64 bit (SPARC, i386, x86_64),
  136     with Oracle Solaris Studio 12.5
  137   - OpenBSD.  Requires configure options --enable-mca-no-build=patcher
  138     and --disable-slopen with this release.
  139   - Problems have been reported when building Open MPI on FreeBSD 11.1
  140     using the clang-4.0 system compiler. A workaround is to build
  141     Open MPI using the GNU compiler.
  143 Platform Notes
  144 --------------
  146 - N/A
  149 Compiler Notes
  150 --------------
  152 - Open MPI requires a C99-capable compiler to build.
  154 - Mixing compilers from different vendors when building Open MPI
  155   (e.g., using the C/C++ compiler from one vendor and the Fortran
  156   compiler from a different vendor) has been successfully employed by
  157   some Open MPI users (discussed on the Open MPI user's mailing list),
  158   but such configurations are not tested and not documented.  For
  159   example, such configurations may require additional compiler /
  160   linker flags to make Open MPI build properly.
  162 - In general, the latest versions of compilers of a given vendor's
  163   series have the least bugs.  We have seen cases where Vendor XYZ's
  164   compiler version A.B fails to compile Open MPI, but version A.C
  165   (where C>B) works just fine.  If you run into a compile failure, you
  166   might want to double check that you have the latest bug fixes and
  167   patches for your compiler.
  169 - Users have reported issues with older versions of the Fortran PGI
  170   compiler suite when using Open MPI's (non-default) --enable-debug
  171   configure option.  Per the above advice of using the most recent
  172   version of a compiler series, the Open MPI team recommends using the
  173   latest version of the PGI suite, and/or not using the --enable-debug
  174   configure option.  If it helps, here's what we have found with some
  175   (not comprehensive) testing of various versions of the PGI compiler
  176   suite:
  178     pgi-8 : NO known good version with --enable-debug
  179     pgi-9 : 9.0-4 known GOOD
  180     pgi-10: 10.0-0 known GOOD
  181     pgi-11: NO known good version with --enable-debug
  182     pgi-12: 12.10 known BAD with -m32, but known GOOD without -m32
  183             (and 12.8 and 12.9 both known BAD with --enable-debug)
  184     pgi-13: 13.9 known BAD with -m32, 13.10 known GOOD without -m32
  185     pgi-15: 15.10 known BAD with -m32
  187 - Similarly, there is a known Fortran PGI compiler issue with long
  188   source directory path names that was resolved in 9.0-4 (9.0-3 is
  189   known to be broken in this regard).
  191 - Open MPI does not support the PGI compiler suite on OS X or MacOS.
  192   See issues below for more details:
  193   https://github.com/open-mpi/ompi/issues/2604
  194   https://github.com/open-mpi/ompi/issues/2605
  196 - OpenSHMEM Fortran bindings do not support the `no underscore` Fortran
  197   symbol convention. IBM's xlf compilers build in that mode by default.
  198   As such, IBM's xlf compilers cannot build/link the OpenSHMEM Fortran
  199   bindings by default. A workaround is to pass FC="xlf -qextname" at
  200   configure time to force a trailing underscore. See the issue below
  201   for more details:
  202   https://github.com/open-mpi/ompi/issues/3612
  204 - MPI applications that use the mpi_f08 module on PowerPC platforms
  205   (tested ppc64le) will likely experience runtime failures if:
  206    - they are using a GNU linker (ld) version after v2.25.1 and before v2.28,
  207      -and-
  208    - they compiled with PGI (tested 17.5) or XL (tested v15.1.5) compilers.
  209   This was noticed on Ubuntu 16.04 which uses the 2.26.1 version of ld by
  210   default. However, this issue impacts any OS using a version of ld noted
  211   above. This GNU linker regression will be fixed in version 2.28.
  212   Below is a link to the GNU bug on this issue:
  213     https://sourceware.org/bugzilla/show_bug.cgi?id=21306
  214   The XL compiler will include a fix for this issue in a future release.
  216 - On NetBSD-6 (at least AMD64 and i386), and possibly on OpenBSD,
  217   libtool misidentifies properties of f95/g95, leading to obscure
  218   compile-time failures if used to build Open MPI.  You can work
  219   around this issue by ensuring that libtool will not use f95/g95
  220   (e.g., by specifying FC=<some_other_compiler>, or otherwise ensuring
  221   a different Fortran compiler will be found earlier in the path than
  222   f95/g95), or by disabling the Fortran MPI bindings with
  223   --disable-mpi-fortran.
  225 - On OpenBSD/i386, if you configure with
  226   --enable-mca-no-build=patcher, you will also need to add
  227   --disable-dlopen.  Otherwise, odd crashes can occur
  228   nondeterministically.
  230 - Absoft 11.5.2 plus a service pack from September 2012 (which Absoft
  231   says is available upon request), or a version later than 11.5.2
  232   (e.g., 11.5.3), is required to compile the Fortran mpi_f08
  233   module.
  235 - Open MPI does not support the Sparc v8 CPU target.  However,
  236   as of Solaris Studio 12.1,  and later compilers, one should not
  237   specify -xarch=v8plus or -xarch=v9.  The use of the options
  238   -m32 and -m64 for producing 32 and 64 bit targets, respectively,
  239   are now preferred by the Solaris Studio compilers.  GCC may
  240   require either "-m32" or "-mcpu=v9 -m32", depending on GCC version.
  242 - It has been noticed that if one uses CXX=sunCC, in which sunCC
  243   is a link in the Solaris Studio compiler release, that the OMPI
  244   build system has issue with sunCC and does not build libmpi_cxx.so.
  245   Therefore  the make install fails.  So we suggest that one should
  246   use CXX=CC, which works, instead of CXX=sunCC.
  248 - If one tries to build OMPI on Ubuntu with Solaris Studio using the C++
  249   compiler and the -m32 option, you might see a warning:
  251     CC: Warning: failed to detect system linker version, falling back to
  252     custom linker usage
  254   And the build will fail.  One can overcome this error by either
  255   setting LD_LIBRARY_PATH to the location of the 32 bit libraries (most
  256   likely /lib32), or giving LDFLAGS="-L/lib32 -R/lib32" to the configure
  257   command.  Officially, Solaris Studio is not supported on Ubuntu Linux
  258   distributions, so additional problems might be incurred.
  260 - Open MPI does not support the gccfss compiler (GCC For SPARC
  261   Systems; a now-defunct compiler project from Sun).
  263 - At least some versions of the Intel 8.1 compiler seg fault while
  264   compiling certain Open MPI source code files.  As such, it is not
  265   supported.
  267 - The Intel 9.0 v20051201 compiler on IA64 platforms seems to have a
  268   problem with optimizing the ptmalloc2 memory manager component (the
  269   generated code will segv).  As such, the ptmalloc2 component will
  270   automatically disable itself if it detects that it is on this
  271   platform/compiler combination.  The only effect that this should
  272   have is that the MCA parameter mpi_leave_pinned will be inoperative.
  274 - It has been reported that the Intel 9.1 and 10.0 compilers fail to
  275   compile Open MPI on IA64 platforms.  As of 12 Sep 2012, there is
  276   very little (if any) testing performed on IA64 platforms (with any
  277   compiler).  Support is "best effort" for these platforms, but it is
  278   doubtful that any effort will be expended to fix the Intel 9.1 /
  279   10.0 compiler issuers on this platform.
  281 - Early versions of the Intel 12.1 Linux compiler suite on x86_64 seem
  282   to have a bug that prevents Open MPI from working.  Symptoms
  283   including immediate segv of the wrapper compilers (e.g., mpicc) and
  284   MPI applications.  As of 1 Feb 2012, if you upgrade to the latest
  285   version of the Intel 12.1 Linux compiler suite, the problem will go
  286   away.
  288 - Early versions of the Portland Group 6.0 compiler have problems
  289   creating the C++ MPI bindings as a shared library (e.g., v6.0-1).
  290   Tests with later versions show that this has been fixed (e.g.,
  291   v6.0-5).
  293 - The Portland Group compilers prior to version 7.0 require the
  294   "-Msignextend" compiler flag to extend the sign bit when converting
  295   from a shorter to longer integer.  This is is different than other
  296   compilers (such as GNU).  When compiling Open MPI with the Portland
  297   compiler suite, the following flags should be passed to Open MPI's
  298   configure script:
  300   shell$ ./configure CFLAGS=-Msignextend CXXFLAGS=-Msignextend \
  301          --with-wrapper-cflags=-Msignextend \
  302          --with-wrapper-cxxflags=-Msignextend ...
  304   This will both compile Open MPI with the proper compile flags and
  305   also automatically add "-Msignextend" when the C and C++ MPI wrapper
  306   compilers are used to compile user MPI applications.
  308 - It has been reported that Pathscale 5.0.5 and 6.0.527 compilers
  309   give an internal compiler error when trying to Open MPI.
  311 - Using the MPI C++ bindings with older versions of the Pathscale
  312   compiler on some platforms is an old issue that seems to be a
  313   problem when Pathscale uses a back-end GCC 3.x compiler. Here's a
  314   proposed solution from the Pathscale support team (from July 2010):
  316       The proposed work-around is to install gcc-4.x on the system and
  317       use the pathCC -gnu4 option. Newer versions of the compiler (4.x
  318       and beyond) should have this fixed, but we'll have to test to
  319       confirm it's actually fixed and working correctly.
  321   We don't anticipate that this will be much of a problem for Open MPI
  322   users these days (our informal testing shows that not many users are
  323   still using GCC 3.x).  Contact Pathscale support if you continue to
  324   have problems with Open MPI's C++ bindings.
  326   Note the MPI C++ bindings have been deprecated by the MPI Forum and
  327   may not be supported in future releases.
  329 - As of July 2017, the Pathscale compiler suite apparently has no
  330   further commercial support, and it does not look like there will be
  331   further releases.  Any issues discovered regarding building /
  332   running Open MPI with the Pathscale compiler suite therefore may not
  333   be able to be resolved.
  335 - Using the Absoft compiler to build the MPI Fortran bindings on Suse
  336   9.3 is known to fail due to a Libtool compatibility issue.
  338 - MPI Fortran API support has been completely overhauled since the
  339   Open MPI v1.5/v1.6 series.
  341   ********************************************************************
  342   ********************************************************************
  343   *** There is now only a single Fortran MPI wrapper compiler and a
  344   *** single Fortran OpenSHMEM wrapper compiler: mpifort and oshfort,
  345   *** respectively.  mpif77 and mpif90 still exist, but they are
  346   *** symbolic links to mpifort.
  347   ********************************************************************
  348   *** Similarly, Open MPI's configure script only recognizes the FC
  349   *** and FCFLAGS environment variables (to specify the Fortran
  350   *** compiler and compiler flags, respectively).  The F77 and FFLAGS
  351   *** environment variables are IGNORED.
  352   ********************************************************************
  353   ********************************************************************
  355   As a direct result, it is STRONGLY recommended that you specify a
  356   Fortran compiler that uses file suffixes to determine Fortran code
  357   layout (e.g., free form vs. fixed).  For example, with some versions
  358   of the IBM XLF compiler, it is preferable to use FC=xlf instead of
  359   FC=xlf90, because xlf will automatically determine the difference
  360   between free form and fixed Fortran source code.
  362   However, many Fortran compilers allow specifying additional
  363   command-line arguments to indicate which Fortran dialect to use.
  364   For example, if FC=xlf90, you may need to use "mpifort --qfixed ..."
  365   to compile fixed format Fortran source files.
  367   You can use either ompi_info or oshmem_info to see with which Fortran
  368   compiler Open MPI was configured and compiled.
  370   There are up to three sets of Fortran MPI bindings that may be
  371   provided depending on your Fortran compiler):
  373   - mpif.h: This is the first MPI Fortran interface that was defined
  374     in MPI-1.  It is a file that is included in Fortran source code.
  375     Open MPI's mpif.h does not declare any MPI subroutines; they are
  376     all implicit.
  378   - mpi module: The mpi module file was added in MPI-2.  It provides
  379     strong compile-time parameter type checking for MPI subroutines.
  381   - mpi_f08 module: The mpi_f08 module was added in MPI-3.  It
  382     provides many advantages over the mpif.h file and mpi module.  For
  383     example, MPI handles have distinct types (vs. all being integers).
  384     See the MPI-3 document for more details.
  386     *** The mpi_f08 module is STRONGLY is recommended for all new MPI
  387         Fortran subroutines and applications.  Note that the mpi_f08
  388         module can be used in conjunction with the other two Fortran
  389         MPI bindings in the same application (only one binding can be
  390         used per subroutine/function, however).  Full interoperability
  391         between mpif.h/mpi module and mpi_f08 module MPI handle types
  392         is provided, allowing mpi_f08 to be used in new subroutines in
  393         legacy MPI applications.
  395   Per the OpenSHMEM specification, there is only one Fortran OpenSHMEM
  396   binding provided:
  398   - shmem.fh: All Fortran OpenSHMEM programs **should** include
  399     'shmem.fh', and Fortran OpenSHMEM programs that use constants
  400     defined by OpenSHMEM **MUST** include 'shmem.fh'.
  402   The following notes apply to the above-listed Fortran bindings:
  404   - All Fortran compilers support the mpif.h/shmem.fh-based bindings,
  405     with one exception: the MPI_SIZEOF interfaces will only be present
  406     when Open MPI is built with a Fortran compiler that support the
  407     INTERFACE keyword and ISO_FORTRAN_ENV.  Most notably, this
  408     excludes the GNU Fortran compiler suite before version 4.9.
  410   - The level of support provided by the mpi module is based on your
  411     Fortran compiler.
  413     If Open MPI is built with a non-GNU Fortran compiler, or if Open
  414     MPI is built with the GNU Fortran compiler >= v4.9, all MPI
  415     subroutines will be prototyped in the mpi module.  All calls to
  416     MPI subroutines will therefore have their parameter types checked
  417     at compile time.
  419     If Open MPI is built with an old gfortran (i.e., < v4.9), a
  420     limited "mpi" module will be built.  Due to the limitations of
  421     these compilers, and per guidance from the MPI-3 specification,
  422     all MPI subroutines with "choice" buffers are specifically *not*
  423     included in the "mpi" module, and their parameters will not be
  424     checked at compile time.  Specifically, all MPI subroutines with
  425     no "choice" buffers are prototyped and will receive strong
  426     parameter type checking at run-time (e.g., MPI_INIT,
  427     MPI_COMM_RANK, etc.).
  429     Similar to the mpif.h interface, MPI_SIZEOF is only supported on
  430     Fortran compilers that support INTERFACE and ISO_FORTRAN_ENV.
  432   - The mpi_f08 module has been tested with the Intel Fortran compiler
  433     and gfortran >= 4.9.  Other modern Fortran compilers likely also
  434     work.
  436     Many older Fortran compilers do not provide enough modern Fortran
  437     features to support the mpi_f08 module.  For example, gfortran <
  438     v4.9 does provide enough support for the mpi_f08 module.
  440   You can examine the output of the following command to see all
  441   the Fortran features that are/are not enabled in your Open MPI
  442   installation:
  444   shell$ ompi_info | grep -i fort
  447 General Run-Time Support Notes
  448 ------------------------------
  450 - The Open MPI installation must be in your PATH on all nodes (and
  451   potentially LD_LIBRARY_PATH (or DYLD_LIBRARY_PATH), if libmpi/libshmem
  452   is a shared library), unless using the --prefix or
  453   --enable-mpirun-prefix-by-default functionality (see below).
  455 - Open MPI's run-time behavior can be customized via MPI Component
  456   Architecture (MCA) parameters (see below for more information on how
  457   to get/set MCA parameter values).  Some MCA parameters can be set in
  458   a way that renders Open MPI inoperable (see notes about MCA
  459   parameters later in this file).  In particular, some parameters have
  460   required options that must be included.
  462   - If specified, the "btl" parameter must include the "self"
  463     component, or Open MPI will not be able to deliver messages to the
  464     same rank as the sender.  For example: "mpirun --mca btl tcp,self
  465     ..."
  466   - If specified, the "btl_tcp_if_exclude" parameter must include the
  467     loopback device ("lo" on many Linux platforms), or Open MPI will
  468     not be able to route MPI messages using the TCP BTL.  For example:
  469     "mpirun --mca btl_tcp_if_exclude lo,eth1 ..."
  471 - Running on nodes with different endian and/or different datatype
  472   sizes within a single parallel job is supported in this release.
  473   However, Open MPI does not resize data when datatypes differ in size
  474   (for example, sending a 4 byte MPI_DOUBLE and receiving an 8 byte
  475   MPI_DOUBLE will fail).
  478 MPI Functionality and Features
  479 ------------------------------
  481 - All MPI-3 functionality is supported.
  483 - Rank reordering support is available using the TreeMatch library. It
  484   is activated for the graph and dist_graph topologies.
  486 - When using MPI deprecated functions, some compilers will emit
  487   warnings.  For example:
  489   shell$ cat deprecated_example.c
  490   #include <mpi.h>
  491   void foo(void) {
  492       MPI_Datatype type;
  493       MPI_Type_struct(1, NULL, NULL, NULL, &type);
  494   }
  495   shell$ mpicc -c deprecated_example.c
  496   deprecated_example.c: In function 'foo':
  497   deprecated_example.c:4: warning: 'MPI_Type_struct' is deprecated (declared at /opt/openmpi/include/mpi.h:1522)
  498   shell$
  500 - MPI_THREAD_MULTIPLE is supported with some exceptions.  Note that
  501   Open MPI must be configured with --enable-mpi-thread-multiple to get
  502   this level of thread safety support.
  504   The following PMLs support MPI_THREAD_MULTIPLE:
  505   - cm (see list (1) of supported MTLs, below)
  506   - ob1 (see list (2) of supported BTLs, below)
  507   - ucx
  508   - yalla
  510   (1) The cm PML and the following MTLs support MPI_THREAD_MULTIPLE:
  511       - MXM
  512       - ofi (Libfabric)
  513       - portals4
  515   (2) The ob1 PML and the following BTLs support MPI_THREAD_MULTIPLE:
  516       - openib (see exception below)
  517       - self
  518       - sm
  519       - smcuda
  520       - tcp
  521       - ugni
  522       - usnic
  523       - vader (shared memory)
  525   The openib BTL's RDMACM based connection setup mechanism is also not
  526   thread safe.  The default UDCM method should be used for
  527   applications requiring MPI_THREAD_MULTIPLE support.
  529   Currently, MPI File operations are not thread safe even if MPI is
  530   initialized for MPI_THREAD_MULTIPLE support.
  532 - MPI_REAL16 and MPI_COMPLEX32 are only supported on platforms where a
  533   portable C datatype can be found that matches the Fortran type
  534   REAL*16, both in size and bit representation.
  536 - The "libompitrace" library is bundled in Open MPI and is installed
  537   by default (it can be disabled via the --disable-libompitrace
  538   flag).  This library provides a simplistic tracing of select MPI
  539   function calls via the MPI profiling interface.  Linking it in to
  540   your application via (e.g., via -lompitrace) will automatically
  541   output to stderr when some MPI functions are invoked:
  543   shell$ cd examples/
  544   shell$ mpicc hello_c.c -o hello_c -lompitrace
  545   shell$ mpirun -np 1 hello_c
  546   MPI_INIT: argc 1
  547   Hello, world, I am 0 of 1
  549   MPI_FINALIZE[0]
  550   shell$
  552   Keep in mind that the output from the trace library is going to
  553   stderr, so it may output in a slightly different order than the
  554   stdout from your application.
  556   This library is being offered as a "proof of concept" / convenience
  557   from Open MPI.  If there is interest, it is trivially easy to extend
  558   it to printf for other MPI functions.  Pull requests on github.com
  559   would be greatly appreciated.
  561 OpenSHMEM Functionality and Features
  562 ------------------------------------
  564 - All OpenSHMEM-1.3 functionality is supported.
  567 MPI Collectives
  568 ---------------
  570 - The "fca" coll component: the Mellanox Fabric Collective Accelerator
  571   (FCA) is a solution for offloading collective operations from the
  572   MPI process onto Mellanox QDR InfiniBand switch CPUs and HCAs.
  574 - The "cuda" coll component provides CUDA-aware support for the
  575   reduction type collectives with GPU buffers. This component is only
  576   compiled into the library when the library has been configured with
  577   CUDA-aware support.  It intercepts calls to the reduction
  578   collectives, copies the data to staging buffers if GPU buffers, then
  579   calls underlying collectives to do the work.
  581 OpenSHMEM Collectives
  582 ---------------------
  584 - The "fca" scoll component: the Mellanox Fabric Collective
  585   Accelerator (FCA) is a solution for offloading collective operations
  586   from the MPI process onto Mellanox QDR InfiniBand switch CPUs and
  587   HCAs.
  589 - The "basic" scoll component: Reference implementation of all
  590   OpenSHMEM collective operations.
  593 Network Support
  594 ---------------
  596 - There are four main MPI network models available: "ob1", "cm",
  597   "yalla", and "ucx".  "ob1" uses BTL ("Byte Transfer Layer")
  598   components for each supported network.  "cm" uses MTL ("Matching
  599   Tranport Layer") components for each supported network.  "yalla"
  600   uses the Mellanox MXM transport.  "ucx" uses the OpenUCX transport.
  602   - "ob1" supports a variety of networks that can be used in
  603     combination with each other:
  605     - OpenFabrics: InfiniBand, iWARP, and RoCE
  606     - Loopback (send-to-self)
  607     - Shared memory
  608     - TCP
  609     - Intel Phi SCIF
  610     - SMCUDA
  611     - Cisco usNIC
  612     - uGNI (Cray Gemini, Aries)
  613     - vader (XPMEM, Linux CMA, Linux KNEM, and copy-in/copy-out shared
  614       memory)
  616   - "cm" supports a smaller number of networks (and they cannot be
  617     used together), but may provide better overall MPI performance:
  619     - Intel Omni-Path PSM2
  620     - Intel True Scale PSM (QLogic InfiniPath)
  621     - OpenFabrics Interfaces ("libfabric" tag matching)
  622     - Portals 4
  624     Open MPI will, by default, choose to use "cm" when one of the
  625     above transports can be used, unless OpenUCX or MXM support is
  626     detected, in which case the "ucx" or "yalla" PML will be used
  627     by default.  Otherwise, "ob1" will be used and the corresponding
  628     BTLs will be selected. Users can force the use of ob1 or cm if
  629     desired by setting the "pml" MCA parameter at run-time:
  631       shell$ mpirun --mca pml ob1 ...
  632       or
  633       shell$ mpirun --mca pml cm ...
  635 - Similarly, there are two OpenSHMEM network models available: "ucx",
  636   and "ikrit":
  637   - "ucx" interfaces directly with UCX;
  638   - "ikrit" interfaces directly with Mellanox MXM.
  640 - UCX is the Unified Communication X (UCX) communication library
  641   (http://www.openucx.org/).
  642   This is an open-source project developed in collaboration between
  643   industry, laboratories, and academia to create an open-source
  644   production grade communication framework for data centric and
  645   high-performance applications.
  646   UCX currently supports:
  647   - OFA Verbs;
  648   - Cray's uGNI;
  649   - NVIDIA CUDA drivers.
  651 - MXM is the Mellanox Messaging Accelerator library utilizing a full
  652   range of IB transports to provide the following messaging services
  653   to the upper level MPI/OpenSHMEM libraries:
  655   - Usage of all available IB transports
  656   - Native RDMA support
  657   - Progress thread
  658   - Shared memory communication
  659   - Hardware-assisted reliability
  661 - The usnic BTL is support for Cisco's usNIC device ("userspace NIC")
  662   on Cisco UCS servers with the Virtualized Interface Card (VIC).
  663   Although the usNIC is accessed via the OpenFabrics Libfabric API
  664   stack, this BTL is specific to Cisco usNIC devices.
  666 - uGNI is a Cray library for communicating over the Gemini and Aries
  667   interconnects.
  669 - The OpenFabrics Enterprise Distribution (OFED) software package v1.0
  670   will not work properly with Open MPI v1.2 (and later) due to how its
  671   Mellanox InfiniBand plugin driver is created.  The problem is fixed
  672   OFED v1.1 (and later).
  674 - Better memory management support is available for OFED-based
  675   transports using the "ummunotify" Linux kernel module.  OFED memory
  676   managers are necessary for better bandwidth when re-using the same
  677   buffers for large messages (e.g., benchmarks and some applications).
  679   Unfortunately, the ummunotify module was not accepted by the Linux
  680   kernel community (and is still not distributed by OFED).  But it
  681   still remains the best memory management solution for MPI
  682   applications that used the OFED network transports.  If Open MPI is
  683   able to find the <linux/ummunotify.h> header file, it will build
  684   support for ummunotify and include it by default.  If MPI processes
  685   then find the ummunotify kernel module loaded and active, then their
  686   memory managers (which have been shown to be problematic in some
  687   cases) will be disabled and ummunotify will be used.  Otherwise, the
  688   same memory managers from prior versions of Open MPI will be used.
  689   The ummunotify Linux kernel module can be downloaded from:
  691     http://lwn.net/Articles/343351/
  693 - The use of fork() with OpenFabrics-based networks (i.e., the openib
  694   BTL) is only partially supported, and only on Linux kernels >=
  695   v2.6.15 with libibverbs v1.1 or later (first released as part of
  696   OFED v1.2), per restrictions imposed by the OFED network stack.
  698 - Linux "knem" support is used when the "vader" or "sm" (shared
  699   memory) BTLs are compiled with knem support (see the --with-knem
  700   configure option) and the knem Linux module is loaded in the running
  701   kernel.  If the knem Linux kernel module is not loaded, the knem
  702   support is (by default) silently deactivated during Open MPI jobs.
  704   See http://runtime.bordeaux.inria.fr/knem/ for details on Knem.
  706 - Linux Cross-Memory Attach (CMA) or XPMEM is used by the vader
  707   shared-memory BTL when the CMA/XPMEM libraries are installedm,
  708   respectively.  Linux CMA and XPMEM are similar (but different)
  709   mechanisms for Open MPI to utilize single-copy semantics for shared
  710   memory.
  712 Open MPI Extensions
  713 -------------------
  715 - An MPI "extensions" framework is included in Open MPI, but is not
  716   enabled by default.  See the "Open MPI API Extensions" section below
  717   for more information on compiling and using MPI extensions.
  719 - The following extensions are included in this version of Open MPI:
  721   - affinity: Provides the OMPI_Affinity_str() routine on retrieving
  722     a string that contains what resources a process is bound to.  See
  723     its man page for more details.
  724   - cr: Provides routines to access to checkpoint restart routines.
  725     See ompi/mpiext/cr/mpiext_cr_c.h for a listing of available
  726     functions.
  727   - cuda: When the library is compiled with CUDA-aware support, it
  728     provides two things.  First, a macro
  729     MPIX_CUDA_AWARE_SUPPORT. Secondly, the function
  730     MPIX_Query_cuda_support that can be used to query for support.
  731   - example: A non-functional extension; its only purpose is to
  732     provide an example for how to create other extensions.
  734 ===========================================================================
  736 Building Open MPI
  737 -----------------
  739 If you have checked out a DEVELOPER'S COPY of Open MPI (i.e., you
  740 cloned from Git), you really need to read the HACKING file before
  741 attempting to build Open MPI. Really.
  743 If you have downloaded a tarball, then things are much simpler.
  744 Open MPI uses a traditional configure script paired with "make" to
  745 build.  Typical installs can be of the pattern:
  747 shell$ ./configure [...options...]
  748 shell$ make [-j N] all install
  749     (use an integer value of N for parallel builds)
  751 There are many available configure options (see "./configure --help"
  752 for a full list); a summary of the more commonly used ones is included
  753 below.
  755 Note that for many of Open MPI's --with-<foo> options, Open MPI will,
  756 by default, search for header files and/or libraries for <foo>.  If
  757 the relevant files are found, Open MPI will built support for <foo>;
  758 if they are not found, Open MPI will skip building support for <foo>.
  759 However, if you specify --with-<foo> on the configure command line and
  760 Open MPI is unable to find relevant support for <foo>, configure will
  761 assume that it was unable to provide a feature that was specifically
  762 requested and will abort so that a human can resolve out the issue.
  764 Additionally, if a search directory is specified in the form
  765 --with-<foo>=<dir>, Open MPI will:
  767 1. Search for <foo>'s header files in <dir>/include.
  768 2. Search for <foo>'s library files:
  769    2a. If --with-<foo>-libdir=<libdir> was specified, search in
  770        <libdir>.
  771    2b. Otherwise, search in <dir>/lib, and if they are not found
  772        there, search again in <dir>/lib64.
  773 3. If both the relevant header files and libraries are found:
  774    3a. Open MPI will build support for <foo>.
  775    3b. If the root path where the <foo> libraries are found is neither
  776        "/usr" nor "/usr/local", Open MPI will compile itself with
  777        RPATH flags pointing to the directory where <foo>'s libraries
  778        are located.  Open MPI does not RPATH /usr/lib[64] and
  779        /usr/local/lib[64] because many systems already search these
  780        directories for run-time libraries by default; adding RPATH for
  781        them could have unintended consequences for the search path
  782        ordering.
  786 --prefix=<directory>
  787   Install Open MPI into the base directory named <directory>.  Hence,
  788   Open MPI will place its executables in <directory>/bin, its header
  789   files in <directory>/include, its libraries in <directory>/lib, etc.
  791 --disable-shared
  792   By default, Open MPI and OpenSHMEM build shared libraries, and all
  793   components are built as dynamic shared objects (DSOs). This switch
  794   disables this default; it is really only useful when used with
  795   --enable-static.  Specifically, this option does *not* imply
  796   --enable-static; enabling static libraries and disabling shared
  797   libraries are two independent options.
  799 --enable-static
  800   Build MPI and OpenSHMEM as static libraries, and statically link in
  801   all components.  Note that this option does *not* imply
  802   --disable-shared; enabling static libraries and disabling shared
  803   libraries are two independent options.
  805   Be sure to read the description of --without-memory-manager, below;
  806   it may have some effect on --enable-static.
  808 --disable-wrapper-rpath
  809   By default, the wrapper compilers (e.g., mpicc) will enable "rpath"
  810   support in generated executables on systems that support it.  That
  811   is, they will include a file reference to the location of Open MPI's
  812   libraries in the application executable itself.  This means that
  813   the user does not have to set LD_LIBRARY_PATH to find Open MPI's
  814   libraries (e.g., if they are installed in a location that the
  815   run-time linker does not search by default).
  817   On systems that utilize the GNU ld linker, recent enough versions
  818   will actually utilize "runpath" functionality, not "rpath".  There
  819   is an important difference between the two:
  821   "rpath": the location of the Open MPI libraries is hard-coded into
  822       the MPI/OpenSHMEM application and cannot be overridden at
  823       run-time.
  824   "runpath": the location of the Open MPI libraries is hard-coded into
  825       the MPI/OpenSHMEM application, but can be overridden at run-time
  826       by setting the LD_LIBRARY_PATH environment variable.
  828   For example, consider that you install Open MPI vA.B.0 and
  829   compile/link your MPI/OpenSHMEM application against it.  Later, you
  830   install Open MPI vA.B.1 to a different installation prefix (e.g.,
  831   /opt/openmpi/A.B.1 vs. /opt/openmpi/A.B.0), and you leave the old
  832   installation intact.
  834   In the rpath case, your MPI application will always use the
  835   libraries from your A.B.0 installation.  In the runpath case, you
  836   can set the LD_LIBRARY_PATH environment variable to point to the
  837   A.B.1 installation, and then your MPI application will use those
  838   libraries.
  840   Note that in both cases, however, if you remove the original A.B.0
  841   installation and set LD_LIBRARY_PATH to point to the A.B.1
  842   installation, your application will use the A.B.1 libraries.
  844   This rpath/runpath behavior can be disabled via
  845   --disable-wrapper-rpath.
  847   If you would like to keep the rpath option, but not enable runpath
  848   a different configure option is avalabile
  849   --disable-wrapper-runpath.
  851 --enable-dlopen
  852   Build all of Open MPI's components as standalone Dynamic Shared
  853   Objects (DSO's) that are loaded at run-time (this is the default).
  854   The opposite of this option, --disable-dlopen, causes two things:
  856   1. All of Open MPI's components will be built as part of Open MPI's
  857      normal libraries (e.g., libmpi).
  858   2. Open MPI will not attempt to open any DSO's at run-time.
  860   Note that this option does *not* imply that OMPI's libraries will be
  861   built as static objects (e.g., libmpi.a).  It only specifies the
  862   location of OMPI's components: standalone DSOs or folded into the
  863   Open MPI libraries.  You can control whether Open MPI's libraries
  864   are build as static or dynamic via --enable|disable-static and
  865   --enable|disable-shared.
  867 --disable-show-load-errors-by-default
  868   Set the default value of the mca_base_component_show_load_errors MCA
  869   variable: the --enable form of this option sets the MCA variable to
  870   true, the --disable form sets the MCA variable to false.  The MCA
  871   mca_base_component_show_load_errors variable can still be overridden
  872   at run time via the usual MCA-variable-setting mechanisms; this
  873   configure option simply sets the default value.
  875   The --disable form of this option is intended for Open MPI packagers
  876   who tend to enable support for many different types of networks and
  877   systems in their packages.  For example, consider a packager who
  878   includes support for both the FOO and BAR networks in their Open MPI
  879   package, both of which require support libraries (libFOO.so and
  880   libBAR.so).  If an end user only has BAR hardware, they likely only
  881   have libBAR.so available on their systems -- not libFOO.so.
  882   Disabling load errors by default will prevent the user from seeing
  883   potentially confusing warnings about the FOO components failing to
  884   load because libFOO.so is not available on their systems.
  886   Conversely, system administrators tend to build an Open MPI that is
  887   targeted at their specific environment, and contains few (if any)
  888   components that are not needed.  In such cases, they might want
  889   their users to be warned that the FOO network components failed to
  890   load (e.g., if libFOO.so was mistakenly unavailable), because Open
  891   MPI may otherwise silently failover to a slower network path for MPI
  892   traffic.
  894 --with-platform=FILE
  895   Load configure options for the build from FILE.  Options on the
  896   command line that are not in FILE are also used.  Options on the
  897   command line and in FILE are replaced by what is in FILE.
  899 --with-libmpi-name=STRING
  900   Replace libmpi.* and libmpi_FOO.* (where FOO is one of the fortran
  901   supporting libraries installed in lib) with libSTRING.* and
  902   libSTRING_FOO.*. This is provided as a convenience mechanism for
  903   third-party packagers of Open MPI that might want to rename these
  904   libraries for their own purposes. This option is *not* intended for
  905   typical users of Open MPI.
  907 --enable-mca-no-build=LIST
  908   Comma-separated list of <type>-<component> pairs that will not be
  909   built. For example, "--enable-mca-no-build=btl-portals,oob-ud" will
  910   disable building the portals BTL and the ud OOB component.
  914 --with-fca=<directory>
  915   Specify the directory where the Mellanox FCA library and
  916   header files are located.
  918   FCA is the support library for Mellanox switches and HCAs.
  920 --with-hcoll=<directory>
  921   Specify the directory where the Mellanox hcoll library and header
  922   files are located.  This option is generally only necessary if the
  923   hcoll headers and libraries are not in default compiler/linker
  924   search paths.
  926   hcoll is the support library for MPI collective operation offload on
  927   Mellanox ConnectX-3 HCAs (and later).
  929 --with-knem=<directory>
  930   Specify the directory where the knem libraries and header files are
  931   located.  This option is generally only necessary if the knem headers
  932   and libraries are not in default compiler/linker search paths.
  934   knem is a Linux kernel module that allows direct process-to-process
  935   memory copies (optionally using hardware offload), potentially
  936   increasing bandwidth for large messages sent between messages on the
  937   same server.  See http://runtime.bordeaux.inria.fr/knem/ for
  938   details.
  940 --with-libfabric=<directory>
  941   Specify the directory where the OpenFabrics Interfaces libfabric
  942   library and header files are located.  This option is generally only
  943   necessary if the libfabric headers and libraries are not in default
  944   compiler/linker search paths.
  946   Libfabric is the support library for OpenFabrics Interfaces-based
  947   network adapters, such as Cisco usNIC, Intel True Scale PSM, Cray
  948   uGNI, etc.
  950 --with-libfabric-libdir=<directory>
  951   Look in directory for the libfabric libraries.  By default, Open MPI
  952   will look in <libfabric directory>/lib and <libfabric
  953   directory>/lib64, which covers most cases.  This option is only
  954   needed for special configurations.
  956 --with-mxm=<directory>
  957   Specify the directory where the Mellanox MXM library and header
  958   files are located.  This option is generally only necessary if the
  959   MXM headers and libraries are not in default compiler/linker search
  960   paths.
  962   MXM is the support library for Mellanox Network adapters.
  964 --with-mxm-libdir=<directory>
  965   Look in directory for the MXM libraries.  By default, Open MPI will
  966   look in <mxm directory>/lib and <mxm directory>/lib64, which covers
  967   most cases.  This option is only needed for special configurations.
  969 --with-portals4=<directory>
  970   Specify the directory where the Portals4 libraries and header files
  971   are located.  This option is generally only necessary if the Portals4
  972   headers and libraries are not in default compiler/linker search
  973   paths.
  975   Portals is a low-level network API for high-performance networking
  976   on high-performance computing systems developed by Sandia National
  977   Laboratories, Intel Corporation, and the University of New Mexico.
  978   The Portals 4 Reference Implementation is a complete implementation
  979   of Portals 4, with transport over InfiniBand verbs and UDP.
  981 --with-portals4-libdir=<directory>
  982   Location of libraries to link with for Portals4 support.
  984 --with-portals4-max-md-size=SIZE
  985 --with-portals4-max-va-size=SIZE
  986   Set configuration values for Portals 4
  988 --with-psm=<directory>
  989   Specify the directory where the QLogic InfiniPath / Intel True Scale
  990   PSM library and header files are located.  This option is generally
  991   only necessary if the PSM headers and libraries are not in default
  992   compiler/linker search paths.
  994   PSM is the support library for QLogic InfiniPath and Intel TrueScale
  995   network adapters.
  997 --with-psm-libdir=<directory>
  998   Look in directory for the PSM libraries.  By default, Open MPI will
  999   look in <psm directory>/lib and <psm directory>/lib64, which covers
 1000   most cases.  This option is only needed for special configurations.
 1002 --with-psm2=<directory>
 1003   Specify the directory where the Intel Omni-Path PSM2 library and
 1004   header files are located.  This option is generally only necessary
 1005   if the PSM2 headers and libraries are not in default compiler/linker
 1006   search paths.
 1008   PSM is the support library for Intel Omni-Path network adapters.
 1010 --with-psm2-libdir=<directory>
 1011   Look in directory for the PSM2 libraries.  By default, Open MPI will
 1012   look in <psm2 directory>/lib and <psm2 directory>/lib64, which
 1013   covers most cases.  This option is only needed for special
 1014   configurations.
 1016 --with-scif=<dir>
 1017   Look in directory for Intel SCIF support libraries
 1019 --with-verbs=<directory>
 1020   Specify the directory where the verbs (also known as OpenFabrics
 1021   verbs, or Linux verbs, and previously known as OpenIB) libraries and
 1022   header files are located.  This option is generally only necessary
 1023   if the verbs headers and libraries are not in default
 1024   compiler/linker search paths.
 1026   The Verbs library usually implies operating system bypass networks,
 1027   such as InfiniBand, usNIC, iWARP, and RoCE (aka "IBoIP").
 1029 --with-verbs-libdir=<directory>
 1030   Look in directory for the verbs libraries.  By default, Open MPI
 1031   will look in <verbs_directory>/lib and <verbs_ directory>/lib64,
 1032   which covers most cases.  This option is only needed for special
 1033   configurations.
 1035 --with-verbs-usnic
 1036   Note that this option is no longer necessary in recent Linux distro
 1037   versions.  If your Linux distro uses the "rdma-core" package (instead
 1038   of a standalone "libibverbs" package), not only do you not need this
 1039   option, you shouldn't use it, either.  More below.
 1041   This option will activate support in Open MPI for disabling a
 1042   dire-sounding warning message from libibverbs that Cisco usNIC
 1043   devices are not supported (because Cisco usNIC devices are supported
 1044   through libfabric, not libibverbs).  This libibverbs warning can
 1045   also be suppressed by installing the "no op" libusnic_verbs plugin
 1046   for libibverbs (see https://github.com/cisco/libusnic_verbs, or
 1047   download binaries from cisco.com).
 1049   This option is disabled by default for two reasons:
 1051   1. It causes libopen-pal.so to depend on libibverbs.so, which is
 1052      undesirable to many downstream packagers.
 1053   2. As mentioned above, recent versions of the libibverbs library
 1054      (included in the "rdma-core" package) do not have the bug that
 1055      will emit dire-sounding warnings about usnic devices.  Indeed,
 1056      the --with-verbs-usnic option will enable code in Open MPI that
 1057      is actually incompatible with rdma-core (i.e., cause Open MPI to
 1058      fail to compile).
 1060    If you enable --with-verbs-usnic and your system uses the rdma-core
 1061    package, configure will safely abort with a helpful message telling
 1062    you that you should not use --with-verbs-usnic.
 1064 --with-usnic
 1065   Abort configure if Cisco usNIC support cannot be built.
 1069 --enable-mpirun-prefix-by-default
 1070   This option forces the "mpirun" command to always behave as if
 1071   "--prefix $prefix" was present on the command line (where $prefix is
 1072   the value given to the --prefix option to configure).  This prevents
 1073   most rsh/ssh-based users from needing to modify their shell startup
 1074   files to set the PATH and/or LD_LIBRARY_PATH for Open MPI on remote
 1075   nodes.  Note, however, that such users may still desire to set PATH
 1076   -- perhaps even in their shell startup files -- so that executables
 1077   such as mpicc and mpirun can be found without needing to type long
 1078   path names.  --enable-orterun-prefix-by-default is a synonym for
 1079   this option.
 1081 --enable-orte-static-ports
 1082    Enable orte static ports for tcp oob (default: enabled).
 1084 --with-alps
 1085   Force the building of for the Cray Alps run-time environment.  If
 1086   Alps support cannot be found, configure will abort.
 1088 --with-lsf=<directory>
 1089   Specify the directory where the LSF libraries and header files are
 1090   located.  This option is generally only necessary if the LSF headers
 1091   and libraries are not in default compiler/linker search paths.
 1093   LSF is a resource manager system, frequently used as a batch
 1094   scheduler in HPC systems.
 1096   NOTE: If you are using LSF version 7.0.5, you will need to add
 1097         "LIBS=-ldl" to the configure command line.  For example:
 1099             ./configure LIBS=-ldl --with-lsf ...
 1101         This workaround should *only* be needed for LSF 7.0.5.
 1103 --with-lsf-libdir=<directory>
 1104   Look in directory for the LSF libraries.  By default, Open MPI will
 1105   look in <lsf directory>/lib and <lsf directory>/lib64, which covers
 1106   most cases.  This option is only needed for special configurations.
 1108 --with-pmi
 1109   Build PMI support (by default on non-Cray XE/XC systems, it is not
 1110   built).  On Cray XE/XC systems, the location of pmi is detected
 1111   automatically as part of the configure process.  For non-Cray
 1112   systems, if the pmi2.h header is found in addition to pmi.h, then
 1113   support for PMI2 will be built.
 1115 --with-slurm
 1116   Force the building of SLURM scheduler support.
 1118 --with-sge
 1119   Specify to build support for the Oracle Grid Engine (OGE) resource
 1120   manager and/or the Open Grid Engine.  OGE support is disabled by
 1121   default; this option must be specified to build OMPI's OGE support.
 1123   The Oracle Grid Engine (OGE) and open Grid Engine packages are
 1124   resource manager systems, frequently used as a batch scheduler in
 1125   HPC systems.
 1127 --with-tm=<directory>
 1128   Specify the directory where the TM libraries and header files are
 1129   located.  This option is generally only necessary if the TM headers
 1130   and libraries are not in default compiler/linker search paths.
 1132   TM is the support library for the Torque and PBS Pro resource
 1133   manager systems, both of which are frequently used as a batch
 1134   scheduler in HPC systems.
 1138 --with-blcr=<directory>
 1139   Specify the directory where the Berkeley Labs Checkpoint / Restart
 1140   (BLCR) libraries and header files are located.  This option is
 1141   generally only necessary if the BLCR headers and libraries are not
 1142   in default compiler/linker search paths.
 1144   This option is only meaningful if the --with-ft option is also used
 1145   to active Open MPI's fault tolerance behavior.
 1147 --with-blcr-libdir=<directory>
 1148   Look in directory for the BLCR libraries.  By default, Open MPI will
 1149   look in <blcr directory>/lib and <blcr directory>/lib64, which
 1150   covers most cases.  This option is only needed for special
 1151   configurations.
 1153 --with-dmtcp=<directory>
 1154   Specify the directory where the Distributed MultiThreaded
 1155   Checkpointing (DMTCP) libraries and header files are located.  This
 1156   option is generally only necessary if the DMTCP headers and
 1157   libraries are not in default compiler/linker search paths.
 1159   This option is only meaningful if the --with-ft option is also used
 1160   to active Open MPI's fault tolerance behavior.
 1162 --with-dmtcp-libdir=<directory>
 1163   Look in directory for the DMTCP libraries.  By default, Open MPI
 1164   will look in <dmtcp directory>/lib and <dmtcp directory>/lib64,
 1165   which covers most cases.  This option is only needed for special
 1166   configurations.
 1168 --with-libevent(=value)
 1169   This option specifies where to find the libevent support headers and
 1170   library.  The following VALUEs are permitted:
 1172     internal:    Use Open MPI's internal copy of libevent.
 1173     external:    Use an external libevent installation (rely on default
 1174                  compiler and linker paths to find it)
 1175     <no value>:  Same as "internal".
 1176     <directory>: Specify the location of a specific libevent
 1177                  installation to use
 1179   By default (or if --with-libevent is specified with no VALUE), Open
 1180   MPI will build and use the copy of libevent that it has in its
 1181   source tree.  However, if the VALUE is "external", Open MPI will
 1182   look for the relevant libevent header file and library in default
 1183   compiler / linker locations.  Or, VALUE can be a directory tree
 1184   where the libevent header file and library can be found.  This
 1185   option allows operating systems to include Open MPI and use their
 1186   default libevent installation instead of Open MPI's bundled libevent.
 1188   libevent is a support library that provides event-based processing,
 1189   timers, and signal handlers.  Open MPI requires libevent to build;
 1190   passing --without-libevent will cause configure to abort.
 1192 --with-libevent-libdir=<directory>
 1193   Look in directory for the libevent libraries.  This option is only
 1194   usable when building Open MPI against an external libevent
 1195   installation.  Just like other --with-FOO-libdir configure options,
 1196   this option is only needed for special configurations.
 1198 --with-hwloc(=value)
 1199   Build hwloc support (default: enabled).  This option specifies where
 1200   to find the hwloc support headers and library.  The following values
 1201   are permitted:
 1203     internal:    Use Open MPI's internal copy of hwloc.
 1204     external:    Use an external hwloc installation (rely on default
 1205                  compiler and linker paths to find it)
 1206     <no value>:  Same as "internal".
 1207     <directory>: Specify the location of a specific hwloc
 1208                  installation to use
 1210   By default (or if --with-hwloc is specified with no VALUE), Open MPI
 1211   will build and use the copy of hwloc that it has in its source tree.
 1212   However, if the VALUE is "external", Open MPI will look for the
 1213   relevant hwloc header files and library in default compiler / linker
 1214   locations.  Or, VALUE can be a directory tree where the hwloc header
 1215   file and library can be found.  This option allows operating systems
 1216   to include Open MPI and use their default hwloc installation instead
 1217   of Open MPI's bundled hwloc.
 1219   hwloc is a support library that provides processor and memory
 1220   affinity information for NUMA platforms.
 1222 --with-hwloc-libdir=<directory>
 1223   Look in directory for the hwloc libraries.  This option is only
 1224   usable when building Open MPI against an external hwloc
 1225   installation.  Just like other --with-FOO-libdir configure options,
 1226   this option is only needed for special configurations.
 1228 --disable-hwloc-pci
 1229   Disable building hwloc's PCI device-sensing capabilities.  On some
 1230   platforms (e.g., SusE 10 SP1, x86-64), the libpci support library is
 1231   broken.  Open MPI's configure script should usually detect when
 1232   libpci is not usable due to such brokenness and turn off PCI
 1233   support, but there may be cases when configure mistakenly enables
 1234   PCI support in the presence of a broken libpci.  These cases may
 1235   result in "make" failing with warnings about relocation symbols in
 1236   libpci.  The --disable-hwloc-pci switch can be used to force Open
 1237   MPI to not build hwloc's PCI device-sensing capabilities in these
 1238   cases.
 1240   Similarly, if Open MPI incorrectly decides that libpci is broken,
 1241   you can force Open MPI to build hwloc's PCI device-sensing
 1242   capabilities by using --enable-hwloc-pci.
 1244   hwloc can discover PCI devices and locality, which can be useful for
 1245   Open MPI in assigning message passing resources to MPI processes.
 1247 --with-libltdl=<directory>
 1248   Specify the directory where the GNU Libtool libltdl libraries and
 1249   header files are located.  This option is generally only necessary
 1250   if the libltdl headers and libraries are not in default
 1251   compiler/linker search paths.
 1253   Note that this option is ignored if --disable-dlopen is specified.
 1255 --disable-libompitrace
 1256   Disable building the simple "libompitrace" library (see note above
 1257   about libompitrace)
 1259 --with-valgrind(=<directory>)
 1260   Directory where the valgrind software is installed.  If Open MPI
 1261   finds Valgrind's header files, it will include additional support
 1262   for Valgrind's memory-checking debugger.
 1264   Specifically, it will eliminate a lot of false positives from
 1265   running Valgrind on MPI applications.  There is a minor performance
 1266   penalty for enabling this option.
 1270 --with-mpi-param-check(=value)
 1271   Whether or not to check MPI function parameters for errors at
 1272   runtime.  The following values are permitted:
 1274     always:  MPI function parameters are always checked for errors
 1275     never:   MPI function parameters are never checked for errors
 1276     runtime: Whether MPI function parameters are checked depends on
 1277              the value of the MCA parameter mpi_param_check (default:
 1278              yes).
 1279     yes:     Synonym for "always" (same as --with-mpi-param-check).
 1280     no:      Synonym for "none" (same as --without-mpi-param-check).
 1282   If --with-mpi-param is not specified, "runtime" is the default.
 1284 --disable-mpi-thread-multiple
 1285   Disable the MPI thread level MPI_THREAD_MULTIPLE (it is enabled by
 1286   default).
 1288 --enable-mpi-cxx
 1289   Enable building the C++ MPI bindings (default: disabled).
 1291   The MPI C++ bindings were deprecated in MPI-2.2, and removed from
 1292   the MPI standard in MPI-3.0.
 1294 --enable-mpi-java
 1295   Enable building of an EXPERIMENTAL Java MPI interface (disabled by
 1296   default).  You may also need to specify --with-jdk-dir,
 1297   --with-jdk-bindir, and/or --with-jdk-headers.  See README.JAVA.txt
 1298   for details.
 1300   Note that this Java interface is INCOMPLETE (meaning: it does not
 1301   support all MPI functionality) and LIKELY TO CHANGE.  The Open MPI
 1302   developers would very much like to hear your feedback about this
 1303   interface.  See README.JAVA.txt for more details.
 1305 --enable-mpi-fortran(=value)
 1306   By default, Open MPI will attempt to build all 3 Fortran bindings:
 1307   mpif.h, the "mpi" module, and the "mpi_f08" module.  The following
 1308   values are permitted:
 1310     all:        Synonym for "yes".
 1311     yes:        Attempt to build all 3 Fortran bindings; skip
 1312                 any binding that cannot be built (same as
 1313                 --enable-mpi-fortran).
 1314     mpifh:      Build mpif.h support.
 1315     usempi:     Build mpif.h and "mpi" module support.
 1316     usempif08:  Build mpif.h, "mpi" module, and "mpi_f08"
 1317                 module support.
 1318     none:       Synonym for "no".
 1319     no:         Do not build any MPI Fortran support (same as
 1320                 --disable-mpi-fortran).  This is mutually exclusive
 1321                 with building the OpenSHMEM Fortran interface.
 1323 --enable-mpi-ext(=<list>)
 1324   Enable Open MPI's non-portable API extensions.  If no <list> is
 1325   specified, all of the extensions are enabled.
 1327   See "Open MPI API Extensions", below, for more details.
 1329 --disable-mpi-io
 1330   Disable built-in support for MPI-2 I/O, likely because an
 1331   externally-provided MPI I/O package will be used. Default is to use
 1332   the internal framework system that uses the ompio component and a
 1333   specially modified version of ROMIO that fits inside the romio
 1334   component
 1336 --disable-io-romio
 1337   Disable the ROMIO MPI-IO component
 1339 --with-io-romio-flags=flags
 1340   Pass flags to the ROMIO distribution configuration script.  This
 1341   option is usually only necessary to pass
 1342   parallel-filesystem-specific preprocessor/compiler/linker flags back
 1343   to the ROMIO system.
 1345 --disable-io-ompio
 1346   Disable the ompio MPI-IO component
 1348 --enable-sparse-groups
 1349   Enable the usage of sparse groups. This would save memory
 1350   significantly especially if you are creating large
 1351   communicators. (Disabled by default)
 1355 --disable-oshmem
 1356   Disable building the OpenSHMEM implementation (by default, it is
 1357   enabled).
 1359 --disable-oshmem-fortran
 1360   Disable building only the Fortran OpenSHMEM bindings. Please see
 1361   the "Compiler Notes" section herein which contains further
 1362   details on known issues with various Fortran compilers.
 1366 --without-memory-manager
 1367   Disable building Open MPI's memory manager.  Open MPI's memory
 1368   manager is usually built on Linux based platforms, and is generally
 1369   only used for optimizations with some OpenFabrics-based networks (it
 1370   is not *necessary* for OpenFabrics networks, but some performance
 1371   loss may be observed without it).
 1373   However, it may be necessary to disable the memory manager in order
 1374   to build Open MPI statically.
 1376 --with-ft=TYPE
 1377   Specify the type of fault tolerance to enable.  Options: LAM
 1378   (LAM/MPI-like), cr (Checkpoint/Restart).  Fault tolerance support is
 1379   disabled unless this option is specified.
 1381 --enable-peruse
 1382   Enable the PERUSE MPI data analysis interface.
 1384 --enable-heterogeneous
 1385   Enable support for running on heterogeneous clusters (e.g., machines
 1386   with different endian representations).  Heterogeneous support is
 1387   disabled by default because it imposes a minor performance penalty.
 1391 --with-wrapper-cflags=<cflags>
 1392 --with-wrapper-cxxflags=<cxxflags>
 1393 --with-wrapper-fflags=<fflags>
 1394 --with-wrapper-fcflags=<fcflags>
 1395 --with-wrapper-ldflags=<ldflags>
 1396 --with-wrapper-libs=<libs>
 1397   Add the specified flags to the default flags that used are in Open
 1398   MPI's "wrapper" compilers (e.g., mpicc -- see below for more
 1399   information about Open MPI's wrapper compilers).  By default, Open
 1400   MPI's wrapper compilers use the same compilers used to build Open
 1401   MPI and specify a minimum set of additional flags that are necessary
 1402   to compile/link MPI applications.  These configure options give
 1403   system administrators the ability to embed additional flags in
 1404   OMPI's wrapper compilers (which is a local policy decision).  The
 1405   meanings of the different flags are:
 1407   <cflags>:   Flags passed by the mpicc wrapper to the C compiler
 1408   <cxxflags>: Flags passed by the mpic++ wrapper to the C++ compiler
 1409   <fcflags>:  Flags passed by the mpifort wrapper to the Fortran compiler
 1410   <ldflags>:  Flags passed by all the wrappers to the linker
 1411   <libs>:     Flags passed by all the wrappers to the linker
 1413   There are other ways to configure Open MPI's wrapper compiler
 1414   behavior; see the Open MPI FAQ for more information.
 1416 There are many other options available -- see "./configure --help".
 1418 Changing the compilers that Open MPI uses to build itself uses the
 1419 standard Autoconf mechanism of setting special environment variables
 1420 either before invoking configure or on the configure command line.
 1421 The following environment variables are recognized by configure:
 1423 CC          - C compiler to use
 1424 CFLAGS      - Compile flags to pass to the C compiler
 1425 CPPFLAGS    - Preprocessor flags to pass to the C compiler
 1427 CXX         - C++ compiler to use
 1428 CXXFLAGS    - Compile flags to pass to the C++ compiler
 1429 CXXCPPFLAGS - Preprocessor flags to pass to the C++ compiler
 1431 FC          - Fortran compiler to use
 1432 FCFLAGS     - Compile flags to pass to the Fortran compiler
 1434 LDFLAGS     - Linker flags to pass to all compilers
 1435 LIBS        - Libraries to pass to all compilers (it is rarely
 1436               necessary for users to need to specify additional LIBS)
 1438 PKG_CONFIG  - Path to the pkg-config utility
 1440 For example:
 1442   shell$ ./configure CC=mycc CXX=myc++ FC=myfortran ...
 1444 *** NOTE: We generally suggest using the above command line form for
 1445     setting different compilers (vs. setting environment variables and
 1446     then invoking "./configure").  The above form will save all
 1447     variables and values in the config.log file, which makes
 1448     post-mortem analysis easier if problems occur.
 1450 Note that if you intend to compile Open MPI with a "make" other than
 1451 the default one in your PATH, then you must either set the $MAKE
 1452 environment variable before invoking Open MPI's configure script, or
 1453 pass "MAKE=your_make_prog" to configure.  For example:
 1455   shell$ ./configure MAKE=/path/to/my/make ...
 1457 This could be the case, for instance, if you have a shell alias for
 1458 "make", or you always type "gmake" out of habit.  Failure to tell
 1459 configure which non-default "make" you will use to compile Open MPI
 1460 can result in undefined behavior (meaning: don't do that).
 1462 Note that you may also want to ensure that the value of
 1463 LD_LIBRARY_PATH is set appropriately (or not at all) for your build
 1464 (or whatever environment variable is relevant for your operating
 1465 system).  For example, some users have been tripped up by setting to
 1466 use a non-default Fortran compiler via FC, but then failing to set
 1467 LD_LIBRARY_PATH to include the directory containing that non-default
 1468 Fortran compiler's support libraries.  This causes Open MPI's
 1469 configure script to fail when it tries to compile / link / run simple
 1470 Fortran programs.
 1472 It is required that the compilers specified be compile and link
 1473 compatible, meaning that object files created by one compiler must be
 1474 able to be linked with object files from the other compilers and
 1475 produce correctly functioning executables.
 1477 Open MPI supports all the "make" targets that are provided by GNU
 1478 Automake, such as:
 1480 all       - build the entire Open MPI package
 1481 install   - install Open MPI
 1482 uninstall - remove all traces of Open MPI from the $prefix
 1483 clean     - clean out the build tree
 1485 Once Open MPI has been built and installed, it is safe to run "make
 1486 clean" and/or remove the entire build tree.
 1488 VPATH and parallel builds are fully supported.
 1490 Generally speaking, the only thing that users need to do to use Open
 1491 MPI is ensure that <prefix>/bin is in their PATH and <prefix>/lib is
 1492 in their LD_LIBRARY_PATH.  Users may need to ensure to set the PATH
 1493 and LD_LIBRARY_PATH in their shell setup files (e.g., .bashrc, .cshrc)
 1494 so that non-interactive rsh/ssh-based logins will be able to find the
 1495 Open MPI executables.
 1497 ===========================================================================
 1499 Open MPI Version Numbers and Binary Compatibility
 1500 -------------------------------------------------
 1502 Open MPI has two sets of version numbers that are likely of interest
 1503 to end users / system administrator:
 1505   * Software version number
 1506   * Shared library version numbers
 1508 Both are predicated on Open MPI's definition of "backwards
 1509 compatibility."
 1511 NOTE: The version numbering conventions were changed with the release
 1512       of v1.10.0.  Most notably, Open MPI no longer uses an "odd/even"
 1513       release schedule to indicate feature development vs. stable
 1514       releases.  See the README in releases prior to v1.10.0 for more
 1515       information (e.g.,
 1516       https://github.com/open-mpi/ompi/blob/v1.8/README#L1392-L1475).
 1518 Backwards Compatibility
 1519 -----------------------
 1521 Open MPI version Y is backwards compatible with Open MPI version X
 1522 (where Y>X) if users can:
 1524   * Compile an MPI/OpenSHMEM application with version X, mpirun/oshrun
 1525     it with version Y, and get the same user-observable behavior.
 1526   * Invoke ompi_info with the same CLI options in versions X and Y and
 1527     get the same user-observable behavior.
 1529 Note that this definition encompasses several things:
 1531   * Application Binary Interface (ABI)
 1532   * MPI / OpenSHMEM run time system
 1533   * mpirun / oshrun command line options
 1534   * MCA parameter names / values / meanings
 1536 However, this definition only applies when the same version of Open
 1537 MPI is used with all instances of the runtime and MPI / OpenSHMEM
 1538 processes in a single MPI job.  If the versions are not exactly the
 1539 same everywhere, Open MPI is not guaranteed to work properly in any
 1540 scenario.
 1542 Backwards compatibility tends to work best when user applications are
 1543 dynamically linked to one version of the Open MPI / OSHMEM libraries,
 1544 and can be updated at run time to link to a new version of the Open
 1545 MPI / OSHMEM libraries.
 1547 For example, if an MPI / OSHMEM application links statically against
 1548 the libraries from Open MPI vX, then attempting to launch that
 1549 application with mpirun / oshrun from Open MPI vY is not guaranteed to
 1550 work (because it is mixing vX and vY of Open MPI in a single job).
 1552 Similarly, if using a container technology that internally bundles all
 1553 the libraries from Open MPI vX, attempting to launch that container
 1554 with mpirun / oshrun from Open MPI vY is not guaranteed to work.
 1556 Software Version Number
 1557 -----------------------
 1559 Official Open MPI releases use the common "A.B.C" version identifier
 1560 format.  Each of the three numbers has a specific meaning:
 1562   * Major: The major number is the first integer in the version string
 1563     Changes in the major number typically indicate a significant
 1564     change in the code base and/or end-user functionality, and also
 1565     indicate a break from backwards compatibility.  Specifically: Open
 1566     MPI releases with different major version numbers are not
 1567     backwards compatibale with each other.
 1569     CAVEAT: This rule does not extend to versions prior to v1.10.0.
 1570             Specifically: v1.10.x is not guaranteed to be backwards
 1571             compatible with other v1.x releases.
 1573   * Minor: The minor number is the second integer in the version
 1574     string.  Changes in the minor number indicate a user-observable
 1575     change in the code base and/or end-user functionality.  Backwards
 1576     compatibility will still be preserved with prior releases that
 1577     have the same major version number (e.g., v2.5.3 is backwards
 1578     compatible with v2.3.1).
 1580   * Release: The release number is the third integer in the version
 1581     string.  Changes in the release number typically indicate a bug
 1582     fix in the code base and/or end-user functionality.  For example,
 1583     if there is a release that only contains bug fixes and no other
 1584     user-observable changes or new features, only the third integer
 1585     will be increased (e.g., from v4.3.0 to v4.3.1).
 1587 The "A.B.C" version number may optionally be followed by a Quantifier:
 1589   * Quantifier: Open MPI version numbers sometimes have an arbitrary
 1590     string affixed to the end of the version number. Common strings
 1591     include:
 1593     o aX: Indicates an alpha release. X is an integer indicating the
 1594       number of the alpha release (e.g., v1.10.3a5 indicates the 5th
 1595       alpha release of version 1.10.3).
 1596     o bX: Indicates a beta release. X is an integer indicating the
 1597       number of the beta release (e.g., v1.10.3b3 indicates the 3rd
 1598       beta release of version 1.10.3).
 1599     o rcX: Indicates a release candidate. X is an integer indicating
 1600       the number of the release candidate (e.g., v1.10.3rc4 indicates
 1601       the 4th release candidate of version 1.10.3).
 1603 Nightly development snapshot tarballs use a different version number
 1604 scheme; they contain three distinct values:
 1606    * The git branch name from which the tarball was created.
 1607    * The date/timestamp, in YYYYMMDDHHMM format.
 1608    * The hash of the git commit from which the tarball was created.
 1610 For example, a snapshot tarball filename of
 1611 "openmpi-v2.x-201703070235-e4798fb.tar.gz" indicates that this tarball
 1612 was created from the v2.x branch, on March 7, 2017, at 2:35am GMT,
 1613 from git hash e4798fb.
 1615 Shared Library Version Number
 1616 -----------------------------
 1618 The GNU Libtool official documentation details how the versioning
 1619 scheme works.  The quick version is that the shared library versions
 1620 are a triple of integers: (current,revision,age), or "c:r:a".  This
 1621 triple is not related to the Open MPI software version number.  There
 1622 are six simple rules for updating the values (taken almost verbatim
 1623 from the Libtool docs):
 1625  1. Start with version information of "0:0:0" for each shared library.
 1627  2. Update the version information only immediately before a public
 1628     release of your software. More frequent updates are unnecessary,
 1629     and only guarantee that the current interface number gets larger
 1630     faster.
 1632  3. If the library source code has changed at all since the last
 1633     update, then increment revision ("c:r:a" becomes "c:r+1:a").
 1635  4. If any interfaces have been added, removed, or changed since the
 1636     last update, increment current, and set revision to 0.
 1638  5. If any interfaces have been added since the last public release,
 1639     then increment age.
 1641  6. If any interfaces have been removed since the last public release,
 1642     then set age to 0.
 1644 Here's how we apply those rules specifically to Open MPI:
 1646  1. The above rules do not apply to MCA components (a.k.a. "plugins");
 1647     MCA component .so versions stay unspecified.
 1649  2. The above rules apply exactly as written to the following
 1650     libraries starting with Open MPI version v1.5 (prior to v1.5,
 1651     libopen-pal and libopen-rte were still at 0:0:0 for reasons
 1652     discussed in bug ticket #2092
 1653     https://svn.open-mpi.org/trac/ompi/ticket/2092):
 1655     * libopen-rte
 1656     * libopen-pal
 1657     * libmca_common_*
 1659  3. The following libraries use a slightly modified version of the
 1660     above rules: rules 4, 5, and 6 only apply to the official MPI and
 1661     OpenSHMEM interfaces (functions, global variables).  The rationale
 1662     for this decision is that the vast majority of our users only care
 1663     about the official/public MPI/OpenSHMEM interfaces; we therefore
 1664     want the .so version number to reflect only changes to the
 1665     official MPI/OpenSHMEM APIs.  Put simply: non-MPI/OpenSHMEM API /
 1666     internal changes to the MPI-application-facing libraries are
 1667     irrelevant to pure MPI/OpenSHMEM applications.
 1669     * libmpi
 1670     * libmpi_mpifh
 1671     * libmpi_usempi_tkr
 1672     * libmpi_usempi_ignore_tkr
 1673     * libmpi_usempif08
 1674     * libmpi_cxx
 1675     * libmpi_java
 1676     * liboshmem
 1678 ===========================================================================
 1680 Checking Your Open MPI Installation
 1681 -----------------------------------
 1683 The "ompi_info" command can be used to check the status of your Open
 1684 MPI installation (located in <prefix>/bin/ompi_info).  Running it with
 1685 no arguments provides a summary of information about your Open MPI
 1686 installation.
 1688 Note that the ompi_info command is extremely helpful in determining
 1689 which components are installed as well as listing all the run-time
 1690 settable parameters that are available in each component (as well as
 1691 their default values).
 1693 The following options may be helpful:
 1695 --all       Show a *lot* of information about your Open MPI
 1696             installation.
 1697 --parsable  Display all the information in an easily
 1698             grep/cut/awk/sed-able format.
 1699 --param <framework> <component>
 1700             A <framework> of "all" and a <component> of "all" will
 1701             show all parameters to all components.  Otherwise, the
 1702             parameters of all the components in a specific framework,
 1703             or just the parameters of a specific component can be
 1704             displayed by using an appropriate <framework> and/or
 1705             <component> name.
 1706 --level <level>
 1707             By default, ompi_info only shows "Level 1" MCA parameters
 1708             -- parameters that can affect whether MPI processes can
 1709             run successfully or not (e.g., determining which network
 1710             interfaces to use).  The --level option will display all
 1711             MCA parameters from level 1 to <level> (the max <level>
 1712             value is 9).  Use "ompi_info --param <framework>
 1713             <component> --level 9" to see *all* MCA parameters for a
 1714             given component.  See "The Modular Component Architecture
 1715             (MCA)" section, below, for a fuller explanation.
 1717 Changing the values of these parameters is explained in the "The
 1718 Modular Component Architecture (MCA)" section, below.
 1720 When verifying a new Open MPI installation, we recommend running six
 1721 tests:
 1723 1. Use "mpirun" to launch a non-MPI program (e.g., hostname or uptime)
 1724    across multiple nodes.
 1726 2. Use "mpirun" to launch a trivial MPI program that does no MPI
 1727    communication (e.g., the hello_c program in the examples/ directory
 1728    in the Open MPI distribution).
 1730 3. Use "mpirun" to launch a trivial MPI program that sends and
 1731    receives a few MPI messages (e.g., the ring_c program in the
 1732    examples/ directory in the Open MPI distribution).
 1734 4. Use "oshrun" to launch a non-OpenSHMEM program across multiple
 1735    nodes.
 1737 5. Use "oshrun" to launch a trivial MPI program that does no OpenSHMEM
 1738    communication (e.g., hello_shmem.c program in the examples/
 1739    directory in the Open MPI distribution.)
 1741 6. Use "oshrun" to launch a trivial OpenSHMEM program that puts and
 1742    gets a few messages. (e.g., the ring_shmem.c in the examples/
 1743    directory in the Open MPI distribution.)
 1745 If you can run all six of these tests successfully, that is a good
 1746 indication that Open MPI built and installed properly.
 1748 ===========================================================================
 1750 Open MPI API Extensions
 1751 -----------------------
 1753 Open MPI contains a framework for extending the MPI API that is
 1754 available to applications.  Each extension is usually a standalone set
 1755 of functionality that is distinct from other extensions (similar to
 1756 how Open MPI's plugins are usually unrelated to each other).  These
 1757 extensions provide new functions and/or constants that are available
 1758 to MPI applications.
 1760 WARNING: These extensions are neither standard nor portable to other
 1761 MPI implementations!
 1763 Compiling the extensions
 1764 ------------------------
 1766 Open MPI extensions are all enabled by default; they can be disabled
 1767 via the --disable-mpi-ext command line switch.
 1769 Since extensions are meant to be used by advanced users only, this
 1770 file does not document which extensions are available or what they
 1771 do.  Look in the ompi/mpiext/ directory to see the extensions; each
 1772 subdirectory of that directory contains an extension.  Each has a
 1773 README file that describes what it does.
 1775 Using the extensions
 1776 --------------------
 1778 To reinforce the fact that these extensions are non-standard, you must
 1779 include a separate header file after <mpi.h> to obtain the function
 1780 prototypes, constant declarations, etc.  For example:
 1782 -----
 1783 #include <mpi.h>
 1784 #if defined(OPEN_MPI) && OPEN_MPI
 1785 #include <mpi-ext.h>
 1786 #endif
 1788 int main() {
 1789     MPI_Init(NULL, NULL);
 1791 #if defined(OPEN_MPI) && OPEN_MPI
 1792     {
 1793         char ompi_bound[OMPI_AFFINITY_STRING_MAX];
 1794         char current_binding[OMPI_AFFINITY_STRING_MAX];
 1795         char exists[OMPI_AFFINITY_STRING_MAX];
 1796         OMPI_Affinity_str(OMPI_AFFINITY_LAYOUT_FMT, ompi_bound,
 1797                           current_bindings, exists);
 1798     }
 1799 #endif
 1800     MPI_Finalize();
 1801     return 0;
 1802 }
 1803 -----
 1805 Notice that the Open MPI-specific code is surrounded by the #if
 1806 statement to ensure that it is only ever compiled by Open MPI.
 1808 The Open MPI wrapper compilers (mpicc and friends) should
 1809 automatically insert all relevant compiler and linker flags necessary
 1810 to use the extensions.  No special flags or steps should be necessary
 1811 compared to "normal" MPI applications.
 1813 ===========================================================================
 1815 Compiling Open MPI Applications
 1816 -------------------------------
 1818 Open MPI provides "wrapper" compilers that should be used for
 1819 compiling MPI and OpenSHMEM applications:
 1821 C:          mpicc, oshcc
 1822 C++:        mpiCC, oshCC (or mpic++ if your filesystem is case-insensitive)
 1823 Fortran:    mpifort, oshfort
 1825 For example:
 1827   shell$ mpicc hello_world_mpi.c -o hello_world_mpi -g
 1828   shell$
 1830 For OpenSHMEM applications:
 1832   shell$ oshcc hello_shmem.c -o hello_shmem -g
 1833   shell$
 1835 All the wrapper compilers do is add a variety of compiler and linker
 1836 flags to the command line and then invoke a back-end compiler.  To be
 1837 specific: the wrapper compilers do not parse source code at all; they
 1838 are solely command-line manipulators, and have nothing to do with the
 1839 actual compilation or linking of programs.  The end result is an MPI
 1840 executable that is properly linked to all the relevant libraries.
 1842 Customizing the behavior of the wrapper compilers is possible (e.g.,
 1843 changing the compiler [not recommended] or specifying additional
 1844 compiler/linker flags); see the Open MPI FAQ for more information.
 1846 Alternatively, Open MPI also installs pkg-config(1) configuration
 1847 files under $libdir/pkgconfig.  If pkg-config is configured to find
 1848 these files, then compiling / linking Open MPI programs can be
 1849 performed like this:
 1851   shell$ gcc hello_world_mpi.c -o hello_world_mpi -g \
 1852               `pkg-config ompi-c --cflags --libs`
 1853   shell$
 1855 Open MPI supplies multiple pkg-config(1) configuration files; one for
 1856 each different wrapper compiler (language):
 1858 ------------------------------------------------------------------------
 1859 ompi       Synonym for "ompi-c"; Open MPI applications using the C
 1860            MPI bindings
 1861 ompi-c     Open MPI applications using the C MPI bindings
 1862 ompi-cxx   Open MPI applications using the C or C++ MPI bindings
 1863 ompi-fort  Open MPI applications using the Fortran MPI bindings
 1864 ------------------------------------------------------------------------
 1866 The following pkg-config(1) configuration files *may* be installed,
 1867 depending on which command line options were specified to Open MPI's
 1868 configure script.  They are not necessary for MPI applications, but
 1869 may be used by applications that use Open MPI's lower layer support
 1870 libraries.
 1872 orte:       Open MPI Run-Time Environment applications
 1873 opal:       Open Portable Access Layer applications
 1875 ===========================================================================
 1877 Running Open MPI Applications
 1878 -----------------------------
 1880 Open MPI supports both mpirun and mpiexec (they are exactly
 1881 equivalent) to launch MPI applications.  For example:
 1883   shell$ mpirun -np 2 hello_world_mpi
 1884   or
 1885   shell$ mpiexec -np 1 hello_world_mpi : -np 1 hello_world_mpi
 1887 are equivalent.  Some of mpiexec's switches (such as -host and -arch)
 1888 are not yet functional, although they will not error if you try to use
 1889 them.
 1891 The rsh launcher (which defaults to using ssh) accepts a -hostfile
 1892 parameter (the option "-machinefile" is equivalent); you can specify a
 1893 -hostfile parameter indicating an standard mpirun-style hostfile (one
 1894 hostname per line):
 1896   shell$ mpirun -hostfile my_hostfile -np 2 hello_world_mpi
 1898 If you intend to run more than one process on a node, the hostfile can
 1899 use the "slots" attribute.  If "slots" is not specified, a count of 1
 1900 is assumed.  For example, using the following hostfile:
 1902 ---------------------------------------------------------------------------
 1903 node1.example.com
 1904 node2.example.com
 1905 node3.example.com slots=2
 1906 node4.example.com slots=4
 1907 ---------------------------------------------------------------------------
 1909   shell$ mpirun -hostfile my_hostfile -np 8 hello_world_mpi
 1911 will launch MPI_COMM_WORLD rank 0 on node1, rank 1 on node2, ranks 2
 1912 and 3 on node3, and ranks 4 through 7 on node4.
 1914 Other starters, such as the resource manager / batch scheduling
 1915 environments, do not require hostfiles (and will ignore the hostfile
 1916 if it is supplied).  They will also launch as many processes as slots
 1917 have been allocated by the scheduler if no "-np" argument has been
 1918 provided.  For example, running a SLURM job with 8 processors:
 1920   shell$ salloc -n 8 mpirun a.out
 1922 The above command will reserve 8 processors and run 1 copy of mpirun,
 1923 which will, in turn, launch 8 copies of a.out in a single
 1924 MPI_COMM_WORLD on the processors that were allocated by SLURM.
 1926 Note that the values of component parameters can be changed on the
 1927 mpirun / mpiexec command line.  This is explained in the section
 1928 below, "The Modular Component Architecture (MCA)".
 1930 Open MPI supports oshrun to launch OpenSHMEM applications. For
 1931 example:
 1933    shell$ oshrun -np 2 hello_world_oshmem
 1935 OpenSHMEM applications may also be launched directly by resource
 1936 managers such as SLURM. For example, when OMPI is configured
 1937 --with-pmi and --with-slurm, one may launch OpenSHMEM applications via
 1938 srun:
 1940    shell$ srun -N 2 hello_world_oshmem
 1942 ===========================================================================
 1944 The Modular Component Architecture (MCA)
 1946 The MCA is the backbone of Open MPI -- most services and functionality
 1947 are implemented through MCA components.  Here is a list of all the
 1948 component frameworks in Open MPI:
 1950 ---------------------------------------------------------------------------
 1952 MPI component frameworks:
 1953 -------------------------
 1955 bml       - BTL management layer
 1956 coll      - MPI collective algorithms
 1957 fbtl      - file byte transfer layer: abstraction for individual
 1958             read/write operations for OMPIO
 1959 fcoll     - collective read and write operations for MPI I/O
 1960 fs        - file system functions for MPI I/O
 1961 io        - MPI I/O
 1962 mtl       - Matching transport layer, used for MPI point-to-point
 1963             messages on some types of networks
 1964 op        - Back end computations for intrinsic MPI_Op operators
 1965 osc       - MPI one-sided communications
 1966 pml       - MPI point-to-point management layer
 1967 rte       - Run-time environment operations
 1968 sharedfp  - shared file pointer operations for MPI I/O
 1969 topo      - MPI topology routines
 1970 vprotocol - Protocols for the "v" PML
 1972 OpenSHMEM component frameworks:
 1973 -------------------------
 1975 atomic    - OpenSHMEM atomic operations
 1976 memheap   - OpenSHMEM memory allocators that support the
 1977             PGAS memory model
 1978 scoll     - OpenSHMEM collective operations
 1979 spml      - OpenSHMEM "pml-like" layer: supports one-sided,
 1980             point-to-point operations
 1981 sshmem    - OpenSHMEM shared memory backing facility
 1984 Back-end run-time environment (RTE) component frameworks:
 1985 ---------------------------------------------------------
 1987 dfs       - Distributed file system
 1988 errmgr    - RTE error manager
 1989 ess       - RTE environment-specific services
 1990 filem     - Remote file management
 1991 grpcomm   - RTE group communications
 1992 iof       - I/O forwarding
 1993 notifier  - System-level notification support
 1994 odls      - OpenRTE daemon local launch subsystem
 1995 oob       - Out of band messaging
 1996 plm       - Process lifecycle management
 1997 ras       - Resource allocation system
 1998 rmaps     - Resource mapping system
 1999 rml       - RTE message layer
 2000 routed    - Routing table for the RML
 2001 rtc       - Run-time control framework
 2002 schizo    - OpenRTE personality framework
 2003 state     - RTE state machine
 2005 Miscellaneous frameworks:
 2006 -------------------------
 2008 allocator   - Memory allocator
 2009 backtrace   - Debugging call stack backtrace support
 2010 btl         - Point-to-point Byte Transfer Layer
 2011 dl          - Dynamic loading library interface
 2012 event       - Event library (libevent) versioning support
 2013 hwloc       - Hardware locality (hwloc) versioning support
 2014 if          - OS IP interface support
 2015 installdirs - Installation directory relocation services
 2016 memchecker  - Run-time memory checking
 2017 memcpy      - Memory copy support
 2018 memory      - Memory management hooks
 2019 mpool       - Memory pooling
 2020 patcher     - Symbol patcher hooks
 2021 pmix        - Process management interface (exascale)
 2022 pstat       - Process status
 2023 rcache      - Memory registration cache
 2024 sec         - Security framework
 2025 shmem       - Shared memory support (NOT related to OpenSHMEM)
 2026 timer       - High-resolution timers
 2028 ---------------------------------------------------------------------------
 2030 Each framework typically has one or more components that are used at
 2031 run-time.  For example, the btl framework is used by the MPI layer to
 2032 send bytes across different types underlying networks.  The tcp btl,
 2033 for example, sends messages across TCP-based networks; the openib btl
 2034 sends messages across OpenFabrics-based networks.
 2036 Each component typically has some tunable parameters that can be
 2037 changed at run-time.  Use the ompi_info command to check a component
 2038 to see what its tunable parameters are.  For example:
 2040   shell$ ompi_info --param btl tcp
 2042 shows some of the parameters (and default values) for the tcp btl
 2043 component (use --level to show *all* the parameters; see below).
 2045 Note that ompi_info only shows a small number a component's MCA
 2046 parameters by default.  Each MCA parameter has a "level" value from 1
 2047 to 9, corresponding to the MPI-3 MPI_T tool interface levels.  In Open
 2048 MPI, we have interpreted these nine levels as three groups of three:
 2050  1. End user / basic
 2051  2. End user / detailed
 2052  3. End user / all
 2054  4. Application tuner / basic
 2055  5. Application tuner / detailed
 2056  6. Application tuner / all
 2058  7. MPI/OpenSHMEM developer / basic
 2059  8. MPI/OpenSHMEM developer / detailed
 2060  9. MPI/OpenSHMEM developer / all
 2062 Here's how the three sub-groups are defined:
 2064  1. End user: Generally, these are parameters that are required for
 2065     correctness, meaning that someone may need to set these just to
 2066     get their MPI/OpenSHMEM application to run correctly.
 2067  2. Application tuner: Generally, these are parameters that can be
 2068     used to tweak MPI application performance.
 2069  3. MPI/OpenSHMEM developer: Parameters that either don't fit in the
 2070     other two, or are specifically intended for debugging /
 2071     development of Open MPI itself.
 2073 Each sub-group is broken down into three classifications:
 2075  1. Basic: For parameters that everyone in this category will want to
 2076     see.
 2077  2. Detailed: Parameters that are useful, but you probably won't need
 2078     to change them often.
 2079  3. All: All other parameters -- probably including some fairly
 2080     esoteric parameters.
 2082 To see *all* available parameters for a given component, specify that
 2083 ompi_info should use level 9:
 2085   shell$ ompi_info --param btl tcp --level 9
 2087 These values can be overridden at run-time in several ways.  At
 2088 run-time, the following locations are examined (in order) for new
 2089 values of parameters:
 2091 1. <prefix>/etc/openmpi-mca-params.conf
 2093    This file is intended to set any system-wide default MCA parameter
 2094    values -- it will apply, by default, to all users who use this Open
 2095    MPI installation.  The default file that is installed contains many
 2096    comments explaining its format.
 2098 2. $HOME/.openmpi/mca-params.conf
 2100    If this file exists, it should be in the same format as
 2101    <prefix>/etc/openmpi-mca-params.conf.  It is intended to provide
 2102    per-user default parameter values.
 2104 3. environment variables of the form OMPI_MCA_<name> set equal to a
 2105    <value>
 2107    Where <name> is the name of the parameter.  For example, set the
 2108    variable named OMPI_MCA_btl_tcp_frag_size to the value 65536
 2109    (Bourne-style shells):
 2111    shell$ OMPI_MCA_btl_tcp_frag_size=65536
 2112    shell$ export OMPI_MCA_btl_tcp_frag_size
 2114 4. the mpirun/oshrun command line: --mca <name> <value>
 2116    Where <name> is the name of the parameter.  For example:
 2118    shell$ mpirun --mca btl_tcp_frag_size 65536 -np 2 hello_world_mpi
 2120 These locations are checked in order.  For example, a parameter value
 2121 passed on the mpirun command line will override an environment
 2122 variable; an environment variable will override the system-wide
 2123 defaults.
 2125 Each component typically activates itself when relevant.  For example,
 2126 the usNIC component will detect that usNIC devices are present and
 2127 will automatically be used for MPI communications.  The SLURM
 2128 component will automatically detect when running inside a SLURM job
 2129 and activate itself.  And so on.
 2131 Components can be manually activated or deactivated if necessary, of
 2132 course.  The most common components that are manually activated,
 2133 deactivated, or tuned are the "BTL" components -- components that are
 2134 used for MPI point-to-point communications on many types common
 2135 networks.
 2137 For example, to *only* activate the TCP and "self" (process loopback)
 2138 components are used for MPI communications, specify them in a
 2139 comma-delimited list to the "btl" MCA parameter:
 2141    shell$ mpirun --mca btl tcp,self hello_world_mpi
 2143 To add shared memory support, add "vader" into the command-delimited
 2144 list (list order does not matter):
 2146    shell$ mpirun --mca btl tcp,vader,self hello_world_mpi
 2148 (there is an "sm" shared memory BTL, too, but "vader" is a newer
 2149 generation of shared memory support; by default, "vader" will be used
 2150 instead of "sm")
 2152 To specifically deactivate a specific component, the comma-delimited
 2153 list can be prepended with a "^" to negate it:
 2155    shell$ mpirun --mca btl ^tcp hello_mpi_world
 2157 The above command will use any other BTL component other than the tcp
 2158 component.
 2160 ===========================================================================
 2162 Common Questions
 2163 ----------------
 2165 Many common questions about building and using Open MPI are answered
 2166 on the FAQ:
 2168     https://www.open-mpi.org/faq/
 2170 ===========================================================================
 2172 Got more questions?
 2173 -------------------
 2175 Found a bug?  Got a question?  Want to make a suggestion?  Want to
 2176 contribute to Open MPI?  Please let us know!
 2178 When submitting questions and problems, be sure to include as much
 2179 extra information as possible.  This web page details all the
 2180 information that we request in order to provide assistance:
 2182      https://www.open-mpi.org/community/help/
 2184 User-level questions and comments should generally be sent to the
 2185 user's mailing list (users@lists.open-mpi.org).  Because of spam, only
 2186 subscribers are allowed to post to this list (ensure that you
 2187 subscribe with and post from *exactly* the same e-mail address --
 2188 joe@example.com is considered different than
 2189 joe@mycomputer.example.com!).  Visit this page to subscribe to the
 2190 user's list:
 2192      http://lists.open-mpi.org/mailman/listinfo/users
 2194 Developer-level bug reports, questions, and comments should generally
 2195 be sent to the developer's mailing list (devel@lists.open-mpi.org).
 2196 Please do not post the same question to both lists.  As with the
 2197 user's list, only subscribers are allowed to post to the developer's
 2198 list.  Visit the following web page to subscribe:
 2200      http://lists.open-mpi.org/mailman/listinfo/devel
 2202 Make today an Open MPI day!