"Fossies" - the Fresh Open Source Software Archive

Member "berkeley_upc-2019.4.2/gasnet/README" (27 May 2019, 75486 Bytes) of package /linux/misc/berkeley_upc-2019.4.2.tar.gz:


As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. See also the last Fossies "Diffs" side-by-side code changes report for "README": 2.28.0_vs_2019.4.0.

    1 =======================================================================
    2 =======================================================================
    3 
    4               *** GASNet-EX Software Release ***
    5 
    6 GASNet-EX is the next generation of the GASNet-1 communication system. 
    7 The GASNet interfaces are being redesigned to accommodate the emerging
    8 needs of Exascale supercomputing, providing communication services to a
    9 variety of PGAS programming models on current and future HPC architectures.
   10 This upgrade is being done over the next several years as part of the DOE
   11 Exascale Computing Program (ECP).
   12 
   13 GASNet-EX is a work-in-progress.
   14 Many features remain to be specified, implemented, and/or tuned.
   15 
   16 Users interested in learning what's new in GASNet-EX are recommended to peruse 
   17 the ChangeLog file in this directory for a summary of recent developments.
   18 The docs/GASNet-EX.spec file provides more detailed information about
   19 the evolving EX specification.
   20 
   21 GASNet-EX notably includes a backwards-compatibility layer to assist in
   22 migration of current GASNet-1 client software. Existing GASNet clients can
   23 get started by relying on this layer (provided in gasnet.h), and incrementally 
   24 add calls to the new gex_* interfaces (defined in gasnetex.h, which is 
   25 automatically included by gasnet.h) to access new EX features and capabilities.
   26 
   27 Feedback or questions on any matters related to the GASNet-EX project are
   28 welcomed at: gasnet-devel@lbl.gov
   29 
   30 EX-Specific Implementation Notes:
   31 
   32 * gasnet_fwd.h provides a dependency-free useful subset of the API
   33   declarations, suitable for inclusion into application-level code.
   34 
   35 * The following public headers may be included in any order:
   36   - gasnetex.h       : GASNet-EX API + Tools
   37   - gasnet.h         : GASNet-1 API (also includes gasnetex.h)
   38   - gasnet_fwd.h     : Forward declaration subset for app inclusion
   39   - gasnet_ratomic.h : Remote Atomics API
   40   - gasnet_vis.h     : Non-contiguous RMA (VIS) API
   41   - gasnet_coll.h    : Collectives API
   42 
   43 =======================================================================
   44 =======================================================================
   45 
   46 README file for GASNet
   47 https://gasnet.lbl.gov
   48 
   49 This is a user manual for GASNet. Anyone planning on using GASNet (either
   50 directly or indirectly) should consult this file for usage instructions.
   51 
   52 Other documentation:
   53  * In this README the "docs directory" means either docs/ in the source
   54    directory or ${prefix}/share/doc/gasnet/ in an installation of GASNet.
   55  * For GASNet licensing and usage terms, see license.txt.
   56  * For documentation on a particular GASNet conduit, see the README file in the
   57    conduit directory (also installed as README-<conduit> in the docs directory).
   58  * For documentation on the communication-independent GASNet-tools library,
   59    see README-tools.
   60  * Additional information, including the GASNet specification and our bug
   61    tracking database, is available from https://gasnet.lbl.gov
   62  * Anyone planning to modify or add to the GASNet code base should also read
   63    the developer documents, available in the GASNet git repository, which 
   64    can be browsed online: https://bitbucket.org/berkeleylab/gasnet/src/develop
   65    + README-devel: GASNet design information and coding standards
   66    + README-git: Rules developers are expected to follow when committing
   67    + template-conduit: A fill-in-the-blanks conduit code skeleton
   68 
   69 Contents of this file:
   70  * Introduction
   71  * Building and Installing GASNet
   72  * Basic Usage Information
   73  * Conduit Status
   74  * Single-node Development Options
   75  * Supported Platforms
   76  * Recognized Environment Variables
   77  * GASNet exit
   78  * GASNet tracing & statistical collection
   79  * GASNet Collectives
   80  * GASNet debug malloc services
   81  * GASNet inter-Process SHared Memory (PSHM)
   82  * MPI Interoperability
   83  * Contact Info and Support
   84 
   85 Introduction
   86 ============
   87 
   88 GASNet is a language-independent, low-level networking layer that provides
   89 network-independent, high-performance communication primitives tailored for
   90 implementing parallel global address space SPMD languages and libraries such as
   91 UPC, UPC++, Co-Array Fortran, Legion, Chapel, and many others. The interface is
   92 primarily intended as a compilation target and for use by runtime library
   93 writers (as opposed to end users), and the primary goals are high performance,
   94 interface portability, and expressiveness. GASNet stands for "Global-Address
   95 Space Networking".
   96 
   97 The GASNet API is defined in the specification, which is included with this 
   98 archive in docs/, and the definitive version is located on the GASNet webpage:
   99 
  100   https://gasnet.lbl.gov/
  101 
  102 This README accompanies the GASNet source distribution, which includes
  103 implementations of the GASNet API for various popular HPC and general-purpose
  104 network hardwares.  We use the term "conduit" to refer to any complete
  105 implementation of the GASNet API which targets a specific network device or
  106 lower-level networking layer. A conduit is comprised of any required headers,
  107 source files and supporting libraries necessary to provide the functionality of
  108 the GASNet API to GASNet clients. This distribution additionally includes
  109 a library of communication-independent portability tools called the "GASNet tools",
  110 which are used in the conduit implementations and also made available to clients
  111 (see README-tools for details).
  112 
  113 System Requirements
  114 ===================
  115 GASNet is extremely portable, and runs on most systems that are relevant to HPC
  116 production or development (and many that are not).
  117 
  118 The minimum system requirements are:
  119 
  120  * A POSIX-like environment, e.g. Linux or another version of Unix.
  121    For Mac systems, the free 'XCode command-line tools' from the Apple Store.
  122    For Windows systems one needs either of two options:
  123    + The free 'Cygwin' toolkit (https://www.cygwin.com/)
  124    + Windows 10 Subsystem for Linux, a.k.a. WSL
  125      (https://docs.microsoft.com/en-us/windows/wsl/)
  126  * Make (we recommend GNU make version 3.79 or newer).
  127  * Perl (version 5.005 or newer).
  128  * The following standard Unix tools: 'awk', 'sed', 'env', 'basename', 'dirname',
  129    and a Bourne-compatible shell (e.g. bash).
  130  * A C compiler with at least minimal C99 support.
  131 
  132 We explicitly support most OS's, architectures and compilers in widespread use today.
  133 See the 'Supported Platforms' section for details on systems we've recently validated.
  134 
  135 Most distributed-memory GASNet conduits have additional requirements, based on their
  136 interactions with network hardware and other implementation details. For example:
  137 
  138  * mpi-conduit requires an MPI-1.1 or newer compliant MPI implementation.
  139  * udp-conduit requires POSIX socket libraries and a C++98 or newer compiler.
  140 
  141 See each conduit README for additional details on system requirements.
  142 
  143 Building and Installing GASNet
  144 ==============================
  145 Here are the steps to build GASNet:
  146 
  147 * Step 0 (optional): ./Bootstrap  
  148   Runs the autoconf tools to build a configure script (this can be done on any
  149 system and may already have been done for you).
  150   If you are keeping a copy of the GASNet sources in your own source control
  151 repository (CVS, svn, Hg, Git, etc.), then please also see "Source Control and
  152 GASNet" in README-devel (in the GASNet git repo).
  153 
  154 * Step 1: ./configure (options)
  155   Generate the Makefiles tailored to your system, creating a build tree in the
  156 current working directory.  You can run configure from a different directory to
  157 place your build files somewhere other than inside the source tree (a nice
  158 option when maintaining several build trees on the same source tree). 
  159 Any compiler flags required for correct operation on your system (e.g. to 
  160 select the correct ABI) should be included in the values of CC, CXX and MPI_CC.
  161 For example to build 32-bit code when your gcc and g++ default to 64-bit:
  162   configure CC='gcc -m32' CXX='g++ -m32' MPI_CC='mpicc -m32'
  163 or similarly, if you want libgasnet to contain debugging symbols:
  164   configure CC='gcc -g' CXX='g++ -g'   (however also see --enable-debug, below)
  165 
  166   Some of the useful configure options:
  167 
  168     --help - display all available configure options
  169     --prefix=/install/path - set the directory where GASNet will be installed
  170     --enable-debug - build GASNet in a debugging mode. This turns on C-level
  171       debugger options and also enables extensive error and sanity checking 
  172       system-wide, which is highly recommended for developing and debugging 
  173       GASNet clients (but should NEVER be used for performance testing). 
  174       --enable-debug also implies --enable-{trace,stats,debug-malloc},
  175       but these can still be selectively --disable'd.
  176     --enable-trace - turn on GASNet tracing (see usage info below)
  177     --enable-stats - turn on GASNet statistical collection (see usage info below)
  178     --enable-debug-malloc - use GASNet debugging malloc (see usage info below)
  179     --enable-segment-{fast,large,everything} - select a GASNet segment 
  180       configuration (see the GASNet spec for more info)
  181     --enable-pshm - Build GASNet with inter-Process SHared Memory (PSHM) support.
  182       This feature uses shared memory communication among the processes (aka
  183       GASNet nodes) within a single compute node (where the other alternatives
  184       are multi-threading via a PAR or PARSYNC build; or use of the conduit's
  185       API to perform the communication).
  186       Note that not all conduits and operating systems support this feature.
  187       For more information, see the section below entitled "GASNet inter-Process
  188       SHared Memory (PSHM)".
  189     --with-max-segsize=<val> - configure-time default value for GASNET_MAX_SEGSIZE,
  190       which is used when GASNET_MAX_SEGSIZE is not set at runtime. See the description 
  191       of the GASNET_MAX_SEGSIZE environment variable below for details.
  192       
  193   Configure will detect various interesting features about your system and
  194 compilers, including which GASNet conduits are supported.
  195 
  196   For cross-compilation support, look for an appropriate cross configure
  197 script in other/contrib/ and link it into your top-level source directory and
  198 invoke it in place of configure.  Example for the IBM BlueGene/Q:
  199      cd <path-to-gasnet-src>
  200      ln -s other/contrib/cross-configure-bgq .
  201      cd <your-build-dir>
  202      <path-to-gasnet-src>/cross-configure-bgq (configure_options)
  203 
  204   For cross-compilation support on platforms without scripts in other/contrib
  205 see the instructions in other/cross-configure-help.c.
  206 [NOTE: We currently don't distribute the cross-configure-help.c in our
  207 normal distribution.  If you think you need it, contact us at gasnet-devel@lbl.gov]
  208 
  209 * Step 2: make all
  210   Build the GASNet libraries. A number of other useful makefile targets 
  211   are available from the top-level:
  212     make {seq,par,parsync}
  213       build the conduit libraries in a given mode
  214     make tests-{seq,par,parsync}  
  215       build all the GASNet tests in a given mode
  216     make run-tests-{seq,par,parsync}  
  217       build and run all the GASNet tests in a given mode
  218     make run-tests
  219       run whatever tests are already built in the conduit directories
  220     make run-tests TESTS="test1 test2..."
  221       run specifically-listed tests that are already built in the conduit directories
  222     make DO_WHAT="<makefile target>"
  223       build selected makefile target in all supported conduit directories
  224   Each conduit directory also has several useful makefile targets:
  225     make {seq,par,parsync}
  226       build the conduit libraries in a given mode
  227     make tests-{seq,par,parsync}  
  228       build the conduit tests in a given mode
  229     make testXXX 
  230       build just testXXX, in SEQ mode
  231     make testXXX-{seq,par,parsync}
  232       build just testXXX, in a given mode
  233     make run-tests-{seq,par,parsync}  
  234       build and run the conduit tests in a given mode
  235     make run-tests
  236       run whatever tests are already built in the conduit directory
  237     make run-tests TESTS="test1 test2..."
  238       run specifically-listed tests that are already built in the conduit directory
  239     make run-testexit
  240       build a script to run the testexit tester and run it
  241   Compilation and linker flags for the GASNet libraries and tests can be
  242   augmented from the command-line by setting the following variables in the make
  243   command, as described in more detail in the next section:
  244     make MANUAL_CFLAGS=...    Flags to add on the C compile for GASNet libs, clients & tests
  245     make MANUAL_CXXFLAGS=...  Flags to add on the C++ compile for GASNet libs, clients & tests
  246     make MANUAL_MPICFLAGS=... Flags to add on the mpicc compile for GASNet libs, clients & tests
  247     make MANUAL_DEFINES=...   Flags to add on all compiles for GASNet libs, clients & tests
  248     make MANUAL_LDFLAGS=...   Linker flags to add for GASNet clients & tests
  249     make MANUAL_LIBS=...      Linker library flags to add for GASNet clients & tests
  250   Note this feature should be used sparingly, as some flags can invalidate the
  251   results of tests performed at configure time. The preferred way to add arbitrary
  252   flags is in the $CC, $CXX and $MPI_CC variables passed to configure.
  253   The following misc make variables can also be set to affect GASNet compilation:
  254     make SEPARATE_CC=1        
  255       Build libgasnet using separate C compiler invocations, rather than one big one.
  256     make KEEPTMPS=1           
  257       Keep temporary files generated by the C compiler, if supported.
  258 
  259 * Step 3 (optional): make install
  260   Install GASNet to the directory chosen at configure time. This will create an
  261 include directory with a sub-directory for each supported conduit, and a lib
  262 directory containing a library file for each supported conduit, as well as any
  263 supporting libraries.
  264   GASNet may also be used directly from the build directory, as a convenience to
  265 eliminate steps if you are making changes to GASNet or its configuration.
  266 
  267 Manual control over compile and link flags
  268 ==========================================
  269 
  270 As described in the previous section, the recommended mechanism for passing
  271 flags to the compilers used by GASNet is to include them in the definition of
  272 the compiler variable itself (e.g. CC='gcc -m64').  These will be followed on
  273 the command line by flags chosen by GASNet's configure script instead of using
  274 the standard CFLAGS and CXXFLAGS make variables.  If there is a need to pass
  275 flags that override ones chosen by configure, then one may set MANUAL_CFLAGS,
  276 MANUAL_CXXFLAGS and MANUAL_MPICFLAGS on the 'make' command line, and these are
  277 guaranteed to appear on the compilation command line after the ones chosen by
  278 configure.  Additionally, MANUAL_DEFINES is provided to pass flags to all
  279 three compilers, but its order on the command line is not guaranteed (since it
  280 is intended for position-independent command line options such as '-Dfoo=10').
  281 
  282 Note that GASNet does not assume that MPI_CC is the same compiler as CC
  283 (though that is recommended), and therefore invokes it using a distinct
  284 MANUAL_MPICFLAGS, instead of MANUAL_CFLAGS.  The MPI C compiler is used to
  285 compile a portion of mpi-conduit (AMMPI) and the MPI-based spawner code used
  286 by several conduits; and to link executables for mpi-conduit and any conduit
  287 in which the MPI-based spawner is enabled.  Similarly, the C++ compiler is
  288 used to compile a portion of udp-conduit (AMUDP) and to link udp-conduit
  289 executables.
  290 
  291 Finally, the make variables MANUAL_LDFLAGS and MANUAL_LIBS are provided to add
  292 to the link command lines, after the respective configure-detected settings.
  293 They are intended for flags such as '-Ldir' and '-lfoo', respectively.  Since
  294 they do not have compiler-specific variants, their use is limited to
  295 compiler-independent flags unless special care is taken with the selection of
  296 the make target to ensure that make only invokes one compiler as a linker.
  297 
  298 All of the MANUAL_* make variables described above are honored by the Makefile
  299 infrastructure used to build GASNet's libraries and tests.  Additionally, the
  300 makefile fragments (next section) use these variables when compiling client
  301 code.
  302 
  303 Basic Usage Information
  304 =======================
  305 See the README for each GASNet conduit implementation for specific usage
  306 information, but generally client programs should #include <gasnetex.h>
  307 (and nothing else), and use the conduit-provided compilation settings.
  308 
  309 The best way to get the correct compiler flags for your GASNet client is to 
  310 "include" the appropriate makefile fragment for the conduit and configuration 
  311 you want in your Makefile, and use the variables it defines in the Makefile 
  312 rules for your GASNet client code.
  313 
  314 For example:
  315   ----------------------------
  316   include $(gasnet_prefix)/include/mpi-conduit/mpi-seq.mak 
  317 
  318   .c.o:
  319 	  $(GASNET_CC) $(GASNET_CPPFLAGS) $(GASNET_CFLAGS) -c -o $@ $<
  320 
  321   .cc.o:
  322 	  $(GASNET_CXX) $(GASNET_CXXCPPFLAGS) $(GASNET_CXXFLAGS) -c -o $@ $<
  323 
  324   myprog: myprog.o
  325 	  $(GASNET_LD) $(GASNET_LDFLAGS) -o $@ $< $(GASNET_LIBS)
  326   ----------------------------
  327 
  328 See tests/Makefile for another example of compiling GASNet client code.
  329 
  330 For more fine-grained control, the flags variables break-down as follows:
  331   GASNET_CFLAGS is an alias for:
  332      $(GASNET_OPT_CFLAGS) $(GASNET_MISC_CFLAGS)
  333   GASNET_CPPFLAGS is an alias for:
  334      $(GASNET_MISC_CPPFLAGS) $(GASNET_DEFINES) $(GASNET_INCLUDES)
  335   GASNET_CXXFLAGS is an alias for:
  336      $(GASNET_OPT_CXXFLAGS) $(GASNET_MISC_CXXFLAGS)
  337   GASNET_CXXCPPFLAGS is an alias for:
  338      $(GASNET_MISC_CXXCPPFLAGS) $(GASNET_DEFINES) $(GASNET_INCLUDES)
  339 
  340 The content of these variables follow these general guidelines:
  341 * GASNET_[CC,CXX] contain the configure-detected C and C++ compilers used to build
  342   GASNet and guaranteed to work with any compiler-specific flags embedded in the
  343   corresponding variables. Clients are strongly advised to compile all modules
  344   using these same compilers to ensure object compatibility.
  345 * GASNET_INCLUDES contains only and all -I preprocessor flags
  346 * GASNET_DEFINES contains only and all -D or -U preprocessor flags
  347 * GASNET_OPT_* contain compiler-specific flags controlling the debug/opt level
  348 * GASNET_MISC_* contain other needed compiler-specific compile-time flags
  349 * GASNET_LD contains one of the configure-detected compilers (CC, CXX or MPI_CC)
  350   which should be used to link the GASNet client executable, in order to guarantee
  351   satisfaction of conduit library dependencies (eg on C++ or MPI).
  352 * GASNET_LIBS contains only and all -L or -l link-time flags
  353 * GASNET_LDFLAGS contains other needed link-time flags, which may be GASNET_LD-specific
  354 This summary is provided for informational purposes only - altering or omitting
  355 any of the flags contained in these variables could result in a non-functional
  356 build environment.
  357 
  358 Using GASNet with pkg-config 
  359 ----------------------------
  360 
  361 As a convenience, the same variables described in the previous section are also
  362 available via the UNIX pkg-config utility.  For example, if pkg-config is installed
  363 on your system and the GASNet .pc files are in your PKG_CONFIG_PATH, then you
  364 can retrieve the value for GASNET_CC for udp-conduit in GASNET_SEQ mode with a
  365 command like:
  366 
  367    pkg-config gasnet-udp-seq --variable=GASNET_CC
  368 
  369 Note the pkg-config --cflags and --libs arguments retrieve the appropriate subset of 
  370 GASNet variables, but pkg-config does not offer aliases corresponding to GASNET_CC,
  371 GASNET_LD or the GASNET_*CXX* variables. Use the `pkg-config --variable=` syntax 
  372 to retrieve these.
  373 
  374 Here is a complete example of using pkg-config with GASNet in a Makefile:
  375 
  376   PKG_CONFIG_PATH = $(gasnet_prefix)/lib/pkgconfig
  377   pkg = gasnet-udp-seq
  378 
  379   .c.o:
  380           `pkg-config $(pkg) --variable=GASNET_CC` `pkg-config $(pkg) --cflags` -c -o $@ $<
  381 
  382   .cc.o:
  383           `pkg-config $(pkg) --variable=GASNET_CXX` `pkg-config $(pkg) --variable=GASNET_CXXCPPFLAGS` \
  384                          `pkg-config $(pkg) --variable=GASNET_CXXFLAGS` -c -o $@ $<
  385 
  386   myprog: myprog.o
  387           `pkg-config $(pkg) --variable=GASNET_LD` -o $@ $< `pkg-config $(pkg) --libs`
  388 
  389 
  390 Conduit Status
  391 ==============
  392 
  393 The GASNet distribution includes multiple complete implementations of the GASNet API
  394 targeting particular lower-level networking layers. Each of these implementations 
  395 is called a 'conduit'. In some cases the lower-level layer is a proprietary or 
  396 hardware-specific network API, whereas in other cases the target API is a
  397 portable standard (although in some cases this distinction is blurred). The corresponding 
  398 GASNet conduits can be loosely categorized as either 'native' or 'portable' conduits.
  399 
  400 Below is the list of conduits in the current distribution, and their high-level status.
  401 Many conduits are supported on multiple platforms (CPU architectures, operating systems 
  402 and compilers) - see the 'Supported Platforms' section for more details on platforms.
  403 For more detailed info about each conduit, please consult the corresponding conduit README.
  404 
  405 Portable conduits:
  406 -----------------
  407 
  408 smp-conduit: SMP loopback (shared memory)
  409  The conduit of choice for GASNet operation within a single shared-memory node.
  410  Rigorously tested and supported on all current platforms. 
  411   
  412 udp-conduit: UDP/IP (part of the TCP/IP protocol suite)
  413  The conduit of choice for GASNet over Ethernet, supported on any TCP/IP-compliant network.
  414  Rigorously tested over Ethernet and supported on most current platforms (see below).
  415  
  416 mpi-conduit: MPI (Message Passing Interface)
  417  A portable implementation of GASNet over MPI-1.1 or later.
  418  Intended as a reference implementation for systems lacking native conduit support.
  419  Rigorously tested and supported on most current platforms (see below).
  420 
  421 ofi-conduit: Open Fabrics Interfaces
  422  GASNet over the Open Fabrics Interface framework (libfabric).
  423  Rigorously tested and supported over the portable/reference OFI sockets Provider.
  424  Believed to also work over the Intel Omni-Path psm2 Provider.
  425 
  426 portals4-conduit: Portals4   *BETA*
  427  GASNet over the Portals4 network API developed at Sandia National Lab.
  428  The core GASNet developers have been unable to test this conduit recently.
  429   
  430 Native, high-performance conduits:
  431 ---------------------------------
  432 
  433 pami-conduit: IBM Parallel Active Messaging Interface (PAMI)
  434  The conduit of choice for GASNet on the IBM BlueGene/Q.
  435  Rigorously tested on BlueGene/Q, also supported on other select IBM systems.
  436 
  437 gemini-conduit: Cray XE/XK Gemini
  438 aries-conduit:  Cray XC Aries
  439  The conduit of choice for GASNet on the Gemini and Aries networks in recent Cray X* systems.
  440  Rigorously tested and supported.
  441 
  442 ibv-conduit: InfiniBand Verbs
  443  GASNet over the OpenFabrics Verbs API.
  444  Rigorously tested and supported over InfiniBand hardware on all supported systems (see below).
  445  Believed to also work on other hardware offering a standard-compliant Verbs layer.
  446 
  447 mxm-conduit: Mellanox InfiniBand Messaging Accelerator (MXM)
  448  GASNet over the MXM API for recent Mellanox InfiniBand adapters.
  449  The core GASNet developers have limited access to this hardware, but have completed 
  450  minimal validation on x86_64/Linux.
  451 
  452 psm-conduit: Intel Omni-Path PSM2
  453  GASNet over the PSM2 API for Intel Omni-Path adapters.
  454  The core GASNet developers have limited access to this hardware, but have completed 
  455  minimal validation on x86_64/Linux. 
  456  
  457 Single-node Development Options
  458 ===============================
  459 
  460 GASNet supports hardware configurations ranging from HPC supercomputers and
  461 clusters to individual workstations and laptops.  Often the easiest way to
  462 develop and debug code is in a single-node environment, so this section
  463 provides conduit recommendations for development.  It should also be noted
  464 that configure option --enable-debug is highly recommended for finding bugs
  465 when developing GASNet client code.
  466 
  467 Option 1.  smp-conduit
  468 ----------------------
  469 This is a pure shared-memory implementation and should be the fastest and
  470 easiest to use option.  GASNet's shared memory support should work "out of
  471 the box" (no special configuration options required) on common laptop,
  472 desktop, and workstation environments, including Linux, macOS, Windows with
  473 Cygwin, Windows 10 Subsystem for Linux (WSL), and Solaris.
  474 
  475 Option 2.  udp-conduit
  476 ----------------------
  477 By default this will be a shared-memory implementation within your single
  478 node, indistinguishable from smp-conduit except for small issues like job
  479 launch and stdin/out/err handling.  However, one can disable shared memory
  480 (by passing --disable-pshm at configure time or by setting the environment
  481 variable GASNET_SUPERNODE_MAXSIZE=1 at runtime).  This may be more realistic
  482 for testing of the expected behaviors in multi-node systems.
  483 
  484 Option 3.  mpi-conduit
  485 ----------------------
  486 If you have a MPI installed (and in your $PATH when GASNet is configured)
  487 then everything said for udp-conduit holds true for mpi-conduit, including
  488 the ability to disable shared-memory support.  Additionally, since MPI will
  489 very likely use shared-memory internally, you can get multi-node like
  490 isolation at the GASNet level with the performance of shared-memory
  491 communication (much better than udp).
  492 
  493 Supported Platforms
  494 ===================
  495 
  496 Platforms where GASNet and Berkeley UPC have been successfully tested:
  497 
  498       OS/Architecture/compiler/ABI: network conduits
  499       ----------------------------------------------
  500     * Linux/x86-Ethernet/{gcc,clang}32: smp, mpi, udp
  501     * Linux/x86-Mellanox/gcc/32:     smp, mpi, udp, ibv
  502     * Linux/x86/IntelC/32:           smp, mpi, udp
  503     * Linux/x86/PortlandGroupC/32:   smp, mpi, udp
  504     * Linux/x86_64-Ethernet/{gcc,clang,PGI,SunC}/{32,64}: smp, mpi, udp, ofi
  505     * Linux/x86_64-Mellanox/{gcc,clang,PathScale,IntelC,SunC}/64: smp, mpi, udp, ibv, mxm
  506     * Linux/x86_64-OmniPath/gcc/64: smp, mpi, udp, ofi, psm
  507     * Linux/x86_64/x86-Open64/64:   smp, udp, #
  508 
  509     * Linux/PowerPC-Ethernet/{gcc,xlc,clang}/{32,64}: smp, mpi, udp
  510     * Linux/PowerPC-HFI/{xlc,gcc}/32: smp, mpi, udp, pami &
  511     * Linux/PPC64le/gcc/{32,64}:     smp, udp, #
  512 
  513     * Linux/MIPS/gcc/{32,n32,64}:       smp, udp, #
  514     * Linux/MIPS64el/gcc/{32,n32,64}:   smp, udp, #
  515 
  516     * FreeBSD/{x86,amd64}/{gcc,clang}/{32,64}: smp, mpi, udp
  517     * OpenBSD/{x86,amd64}/{gcc,clang}/{32,64}: smp, mpi, udp
  518     * NetBSD/{x86,amd64}/{gcc,clang}/{32,64}:  smp, mpi, udp
  519 
  520     * Solaris10/SPARC/gcc,SunProC/{32,64}: smp, udp, mpi
  521     * Solaris10/x86/gcc,SunProC/{32,64}:   smp, udp, mpi
  522     * OpenSolaris/x86/gcc,SunStudio12/{32,64}: smp, udp, #
  523     * Solaris11Express/x86/gcc,SunStudio12/{32,64}: smp, udp, mpi, ibv
  524     * Solaris11/x86/gcc,SunStudio12/{32,64}: smp, udp, mpi, ibv
  525 
  526     * MSWindows-Cygwin/{x86,x86_64}/{gcc,clang}/{32,64}: smp, udp, mpi (OpenMPI)
  527     * MSWindows10-Linux(WSL)/{gcc,clang}/64: smp, udp, mpi
  528 
  529     * macOS/PowerPC/{gcc,xlc}/{32,64}: smp, udp, mpi &
  530     * macOS/{x86,x86_64}/icc/{32,64}: smp, udp, mpi &
  531     * macOS/{x86,x86_64}/{gcc,clang}/{32,64}: smp, udp, mpi
  532     * macOS/{x86,x86_64}/PGI/{32,64}: smp, udp, #
  533 
  534     * CNL/Cray-XT/{gcc,PGI,PathScale,Intel}/64: smp, mpi &
  535 
  536     * CNL/Cray-XE/{gcc,PGI,PathScale,Intel,Cray}/64: smp, mpi, gemini
  537     * CNL/Cray-XK/{gcc,PGI,PathScale,Intel,Cray}/64: smp, mpi, gemini
  538 
  539     * CNL/Cray-XC/{gcc,Intel,Cray}/64: smp, mpi, aries, ofi
  540 
  541     * Linux/Cray-XD1/{gcc,PGI}/64: smp, mpi &
  542 
  543     * CNK/IBM-BlueGeneQ-PowerPC/{gcc,xlc,clang}/64: smp, mpi, pami
  544  
  545     * ucLinux/Microblaze/gcc/32: smp, udp, #%&
  546 
  547     * Linux/ARM/gcc/32:               smp, udp, #
  548     * Linux/AARCH64/{gcc,clang}/64:   smp, udp, #
  549 
  550 # = No working MPI implementation available, although we have every
  551     reason to believe mpi conduit would work if one was available.
  552 % = System lacks pthreads or they are broken
  553 & = System has not been tested in recent releases due to lack of access.
  554     Reports of success or failure on this system are strongly encouraged.
  555 BETA = Support for this system is in beta state, please report your experiences.
  556 
  557 This list is not meant to be exhaustive.
  558 Other combinations of the platforms above are likely to also work, these are
  559 just the systems we've personally tested. Several of the systems listed using a
  560 vendor-specific C compiler can also use gcc as the underlying C compiler,
  561 although we generally recommend the vendor C compiler for performance reasons.
  562 
  563 Recognized Environment Variables
  564 ================================
  565 
  566 Users of language- or application-specific wrappers for job launch should also
  567 consult the wrapper's documentation.  Such wrappers often have options to set
  568 these environment variables while also enabling any corresponding language- or
  569 application-specific support.
  570 
  571 * GASNET_VERBOSEENV: set to 1 to output information about environment variable settings
  572  read by the conduit that affect conduit behavior.
  573 
  574 * GASNET_FREEZE: set to 1 to make GASNet pause and wait for a debugger to attach on
  575  startup
  576 
  577 * GASNET_FREEZE_ON_ERROR: set to 1 to make GASNet pause and wait for a debugger
  578  to attach on any fatal errors or fatal signals
  579 
  580 * GASNET_FREEZE_SIGNAL: set to a signal name (e.g. "SIGINT" or "SIGUSR1") to
  581  specify a signal that will cause the process to freeze and await debugger attach.
  582 
  583 * GASNET_TRACEFILE, GASNET_TRACEMASK, GASNET_STATSFILE, GASNET_STATSMASK, 
  584   GASNET_TRACEFLUSH, GASNET_TRACELOCAL, GASNET_TRACENODES, GASNET_STATSNODES:
  585    control tracing & statistical features, if enabled at configure time.
  586    See usage information below.
  587 
  588 * GASNET_TEST_POLITE_SYNC: set to 1 to enable polite-mode synchronization
  589   for the GASNet tests (only), for running with overcommitted CPUs.
  590 
  591 * GASNET_MALLOC_* : control the GASNet debug malloc features, if enabled at configure time.
  592  See usage information below.
  593 
  594 * GASNET_MAX_SEGSIZE - control the upper limit for FAST/LARGE segment size on most conduits
  595  This setting defaults to the value passed to configure: --with-max-segsize=<val>
  596  In FAST and LARGE segment configurations, GASNet probes each compute node at
  597  startup to determine an upper-limit on the available space for use in the
  598  GASNet segment (and some other large internal objects). This value provides
  599  one upper-limit to that probe, which also has the effect of limiting the 
  600  space available for client segments (as reported by gasnet_getMaxLocalSegmentSize()).
  601  <val> has the following format: size_spec ( / opt_suffix )
  602  where 'size_spec' is either an absolute memory size: [0-9]+{KB,MB,GB}
  603  or a fraction of compute node physical memory: 0.85
  604  and 'opt_suffix' is one of the following: (or empty, which means "P")
  605   "P" : means the limit is per-process and EXCLUDES internal GASNet objects
  606   "H" : means the limit is host-wide and INCLUDES internal GASNet objects 
  607  Examples:
  608   "0.85/H" : limit host-wide use at 85% of physical memory (this is also the default)
  609   "4GB/P"  : try to ensure 4GB per process of GASNet shared segment space
  610  The default behavior of this option has grown considerably smarter over time, so
  611  it's anticipated that most clients will never need to set this.
  612 
  613 * GASNET_DISABLE_MUNMAP - requests the use of mallopt() on glibc systems to disable
  614  mmap-based allocation for satisfying malloc. This can be used to work-around a
  615  known bug in firehose (bug 495) that could lead to incorrect behavior after 
  616  free()ing out-of-segment memory areas previously used for communication. Note that
  617  on some systems (32-bit Linux in particular) the disable is only partly effective
  618  because once the sbrk()-controlled heap reaches the bottom of shared libraries,
  619  glibc will use mmap() used to obtain memory regardless of any options one can
  620  control. The default is conduit-dependent.
  621 
  622 * GASNET_MAX_THREADS - per-node limit on the number of GASNet client pthreads 
  623   (in PAR and PARSYNC modes) that can simultaneously be live on each GASNet node.
  624   This is subject to the hard limit established by configure --with-max-pthreads-per-node,
  625   and pthread limits that may be imposed by specific conduits (see conduit README).
  626 
  627 * GASNET_SUPERNODE_MAXSIZE - limit on size of a GASNet "supernode".
  628  This is the maximum number of processes (GASNet "nodes") that will be grouped
  629  into a shared-memory "supernode", as reported by gasnet_getNodeInfo() and used
  630  by shared-memory communication (PSHM).  A value of zero means no limit.
  631 
  632 * GASNET_PSHM_BARRIER_HIER - enable/disable hierarchical shared-memory barrier.
  633  When shared-memory communication (PSHM) is enabled, the default behavior of
  634  most of GASNet's barrier implementations is to use a two-stage barrier which
  635  coordinates within each supernode before communicating across the network.
  636  This variable can be set to 0 to disable this optimization.
  637 
  638 * GASNET_CATCH_EXIT - set to 0 to prevent GASNet from forcing global job
  639  termination (via atexit() or on_exit()) when a process calls exit() or returns
  640  from main().  GASNet's default behavior helps prevent orphaned processes that
  641  can occur in some systems after an incomplete job termination, but may interfere 
  642  with some profiling tools that write output inside atexit handlers. Setting this
  643  variable may allow those tools to operate, but the client code (or other entity) 
  644  must assume responsibility for ensuring no orphan processes are left behind.
  645 
  646  Note this variable does not affect the behavior of explicit calls to gasnet_exit() 
  647  (either directly or indirectly via calls like upc_global_exit(), returning from 
  648  UPC main(), or reaching the GASNet default fatal signal handler) which will still bypass
  649  atexit handlers and kill the job. The recommended method to ensure the execution of 
  650  atexit handlers is to run with GASNET_CATCH_EXIT=0 and collectively invoke libc exit().
  651 
  652 * GASNET_NO_CATCH_SIGNAL - specify a comma separated list of signals
  653  to exclude from GASNet default signal handling. Formats "SIGSEGV",
  654  "SEGV", "sigsegv", "segv", and "11" are accepted. If the value of
  655  this environment variable is "*" (alone, with no leading or trailing
  656  whitespace), GASNet will not register the default handler for any
  657  signal.
  658  
  659  The default handler provides GASNet's backtrace support and ensures clean
  660  exits when fatal signals are received.  Disabling this handler may
  661  allow use of other tools for debugging of signals, but is not intended
  662  for production use.
  663 
  664 * GASNET_BACKTRACE - requests the generation of stack backtraces on most fatal
  665  errors.  The format/content of these backtraces varies by platform.  On some
  666  platforms no backtrace support is available and this variable will be
  667  ignored.  Backtraces are sent to stderr and to the trace file if tracing is
  668  active (see below).
  669  WARNING: Some fatal errors may involve memory corruption or other abnormal
  670  conditions that could cause the backtrace code to hang. For this reason we
  671  do not recommend setting GASNET_BACKTRACE by default (though there is no
  672  performance penalty for doing so).
  673  When reporting bugs, one is strongly encouraged to include a backtrace if
  674  possible.  The backtrace is almost always more detailed if GASNet is built
  675  with debugging enabled, but may still be useful to a GASNet developer in a
  676  non-debug build.  If tracing is active (see below) then a copy of the
  677  backtrace will be sent to the trace file.  This file may provide developers
  678  with potentially useful information about activities prior to the error.
  679 
  680 * GASNET_BACKTRACE_NODES - if enabled by GASNET_BACKTRACE then this provides
  681  an optional list of nodes on which to permit backtraces.  The list may
  682  contain one or more integers or ranges separated by commas, such as "0,2-4,6".
  683  If unset, empty, or equal to "*" then all nodes may generate backtraces.
  684 
  685 * GASNET_BACKTRACE_SIGNAL: set to a signal name (e.g. "SIGINT" or "SIGUSR1") to
  686  specify a signal that will cause the process to generate an immediate backtrace,
  687  and then continue executing. This is useful for getting a convenient
  688  backtrace for a "hung" process.
  689 
  690 * GASNET_BACKTRACE_TYPE - set to a comma-delimited, ordered list of mechanisms 
  691  (i.e. different debugger tools) to try when generating a backtrace for
  692  GASNET_BACKTRACE. The default value (visible via GASNET_VERBOSEENV) includes
  693  all mechanisms detected as supported on the current platform.
  694 
  695 * GASNET_DISABLE_ENVDECODE/GASNET_DISABLE_ARGDECODE - disable the automatic decoding 
  696  of environment variable values/command-line arguments. Some GASNet spawners 
  697  automatically encode shell meta-characters passing through non-GASNet spawn scripts, 
  698  in to order to ensure their safe delivery to the GASNet client program.
  699 
  700 * GASNET_BARRIER - select the communication algorithm for use in GASNet barriers.
  701  The following values are available on all conduits:
  702   AMCENTRAL - uses Active Messages to manipulate a single centralized counter.
  703   AMDISSEM - uses Active Messages to implement the Dissemination barrier
  704    algorithm as described in section 3.3 of
  705     John M. Mellor-Crummey and Michael L. Scott. "Algorithms for scalable
  706     synchronization on shared-memory multiprocessors." ACM ToCS, 9(1):21 65, 1991.
  707   RDMADISSEM - uses Put operations to implement the Dissemination algorithm.
  708   DISSEM - auto-selects either AMDISSEM or RDMADISSEM
  709    The AMDISSEM or RDMADISSEM algorithm is selected automatically based on
  710    conduit-specific criteria.  In general RDMADISSEM is favored when the GASNet
  711    Extended API is implemented natively.
  712  In addition to those choices, many conduits have additional network-specific
  713  barrier algorithms documented in the corresponding conduit READMEs.
  714  The default is DISSEM, unless the conduit README documents another default.
  715 
  716 * GASNET_PSHM_BARRIER_RADIX - set radix for intra-node barrier algorithm
  717  For configurations using PSHM (a function of OS and conduit) the GASNet barrier
  718  is performed in intra-node and inter-node stages.  This environment variable
  719  is the radix of the tree-based intra-node (shared-memory) barrier.
  720    If zero (default) then radix = size - 1, resulting in a "flat tree" (linear time)
  721    If positive, then the given value is the out-degree of an N-ary tree.
  722    If negative, then a tree is built with the processes in groups of size = -radix.
  723      The first process in each group is the parent of the others in that group.
  724      The rank==0 process is the parent of the other group-representatives (in
  725      addition to being the parent of the others in its own group).
  726  The default is 0 (linear) on most platforms.
  727 
  728 * GASNET_PSHM_NETWORK_DEPTH - set depth of the intra-node AM network
  729  For configurations using PSHM (a function of OS and conduit) GASNet implements
  730  intra-node Active Messages using a shared-memory queue.  This variable sets the
  731  "network depth" of this implementation: the minimum number of outgoing AMs it
  732  must be capable of buffering before possibly stalling.
  733  The default is 32 and the minimum is 4.
  734 
  735 * GASNET_NODEMAP_EXACT - enables exact algorithm for discovery of shared memory nodes.
  736  Several GASNet conduits use mmap() and/or conduit-specific memory registration
  737  resources to establish the GASNet segment.  When multiple GASNet nodes (processes)
  738  run on the same O/S node, there is a potential for competition for resources which
  739  can be managed by coordinating among the processes.
  740  When PSHM support is enabled, multiple processes on the same O/S node must be
  741  identified so they can cross-mmap() their GASNet segments.
  742  By default processes sharing the same O/S node are discovered using an algorithm
  743  that runs in time linear in the number of GASNet nodes, and which is sufficient
  744  for all common process layout patterns.  However, the default may fail to discover
  745  sharing in unusual cases.  Setting this variable to 1 enables an algorithm that is
  746  certain to find all sharing of memory, but has an expected running time
  747  proportional to N*log(N).
  748  The default is currently 0 for all but udp-conduit, which has an unpredictable
  749  process layout.
  750 
  751 * GASNET_VIS_AMPIPE - set to 1 to enable packing of most non-contiguous
  752  put/gets into AMMediums, with each packet of size approx MaxMedium (the only
  753  exception being cases where both sides happen to be fully contiguous, in
  754  which case we skip packing). The default is conduit-dependent.
  755 
  756 * GASNET_VIS_{PUT,GET}_MAXCHUNK - limits the max size of a contiguous chunk which
  757  will be packed by AM pipelining in a strided put/get or indexed put/get, respectively.
  758  The chunk size may additionally be limited based on the size that will fit in one MaxMedium.
  759  The default value is conduit-specific.
  760 
  761 * GASNET_VIS_MAXCHUNK - Provides a default value for GASNET_VIS_{PUT,GET}_MAXCHUNK,
  762  to be used when the more specific knob is unset.
  763 
  764 * GASNET_VIS_REMOTECONTIG - enables a pack & RDMA algorithm for gather puts and
  765  scatter gets - i.e. cases that are locally non-contiguous but remotely
  766  contiguous. The default is conduit-dependent.
  767 
  768 * GASNET_COLL_SCRATCH_SIZE - Specifies the size of the scratch space allocated
  769  on each rank for internal use in collective communications.  This is the
  770  preferred size allocated for the initial team, and is the value returned from
  771  a query using GEX_FLAG_TM_SCRATCH_SIZE_RECOMMENDED.
  772  If the size is set too low, then the performance of collectives may suffer.
  773  Defaults to 2MB per rank.
  774 
  775 * GASNET_COLL_MIN_SCRATCH_SIZE - Specifies the minimum size of the scratch
  776  space allocated on each rank for internal use in collective communications.
  777  This is the minimum size permitted for the initial team (or job startup will
  778  fail), and is the value returned from a query using GEX_FLAG_TM_SCRATCH_SIZE_MIN.
  779  Defaults to 1KB per rank.
  780 
  781 * GASNET_COLL_ENABLE_SEARCH - enable autotuning of collectives
  782 * GASNET_COLL_TUNING_FILE - file to read and/or write collective autotuning data
  783  For usage information, see the file autotuner.txt in the docs directory.
  784 
  785 * GASNET_FS_SYNC - set to 1 enable a sync() call (or equivalent) at exit time.
  786  Default is 0.  Try this setting if you experience truncated output.
  787 
  788 * GASNET_TMPDIR: if set to a valid directory name this is used instead of TMPDIR
  789  or "/tmp" as a location for creating temporary files.
  790 
  791 * GASNET_SD_INIT, GASNET_SD_INITVAL and GASNET_SD_INITLEN
  792  These variables are *only* recognized in a build configured with --enable-debug.
  793 
  794  When using the negotiated-payload AM APIs, GASNet can optionally initialize the
  795  start of buffers it allocates at Prepare time with a defined pattern, and verify
  796  at Commit time that the client has over-written this "canary".  To reduce
  797  overhead, only a fixed-length pattern is used.  However, to avoid false alarms,
  798  this length must be sufficiently long to make the probability of matching the
  799  client data very low.
  800 
  801  GASNET_SD_INIT - Defaults to 1 in debug builds (and is effectively 0 otherwise)
  802    Setting to 0 disables the initialization and checking
  803  GASNET_SD_INITVAL - Defaults to "NAN"
  804    Specifies the initialization pattern.
  805    See GASNET_MALLOC_INITVAL for more info.
  806  GASNET_SD_INITLEN - Defaults to 128
  807    Specifies the length of the "canary" used for this checking.
  808    If 'nbytes' passed to the Commit call is less than this length, then no
  809    checking will be performed.
  810    If set too small then the likelihood of random payloads corresponding to the
  811    initialization pattern may lead to false positives.
  812    The minimum is 1 byte and smaller values will be silently replaced by 1.
  813 
  814 * OPTIONAL: GASNET_EXITTIMEOUT, GASNET_EXITTIMEOUT_MAX, GASNET_EXITTIMEOUT_MIN, and
  815  GASNET_EXITTIMEOUT_FACTOR - control exit-coordination timeout.  Some conduits use a
  816  timeout to distinguish orderly job exit from uncoordinated failure of one or more
  817  nodes.  This is important to help avoid leaving "orphan" processes on the nodes.
  818  These environment variables allow the user to adjust the length of time that the
  819  conduit will wait to establish contact among all nodes before deciding that an exit
  820  is uncoordinated.  By default, the timeout is computed as
  821    GASNET_EXITTIMEOUT = min(GASNET_EXITTIMEOUT_MAX,
  822                             GASNET_EXITTIMEOUT_MIN + nodes * GASNET_EXITTIMEOUT_FACTOR)
  823  Setting GASNET_EXITTIMEOUT provides a specific value, ignoring this formula.
  824  Setting one or more of the others will compute a value for GASNET_EXITTIMEOUT using
  825  the formula above (and defaults for any variables not set in the environment).
  826  Currently most conduits honor these settings.
  827  Smp-conduit honors these settings only when PSHM-support is enabled.
  828  Default values differ among these conduits.
  829 
  830 * OPTIONAL: GASNET_THREAD_STACK_MIN, GASNET_THREAD_STACK_PAD
  831  Some conduits spawn additional threads internally for various purposes, which may
  832  include running client's AM handlers.  If such AM handlers have unusually large
  833  stack space requirements, a mechanism is required to ensure these internal threads
  834  will have large enough stacks.
  835  These settings allow one to control the stack sizes of these internal threads.
  836  The stack size requested from the system will be computed as
  837     stack_size = MAX(GASNET_THREAD_STACK_MIN,
  838                      GASNET_THREAD_STACK_PAD + default) 
  839  where "default" denotes the system's default thread stack size, and values
  840  of both variables have been rounded-up to a multiple of the page size.
  841  The actual stack size may be smaller (e.g. to satisfy rlimits) or larger
  842  (for instance, the next power of two in some implementations).
  843  The default for both settings is zero, which results in using the system
  844  default size for thread stacks.
  845 
  846 * see conduit-specific documentation in conduit directories for more settings
  847 
  848 GASNet exit
  849 ===========
  850 
  851 GASNet clients desiring robustness and portability should not call _exit().
  852 Normal process termination should be done using gasnet_exit(), exit() or
  853 return from main().  Use of these paths allows GASNet to ensure proper
  854 shutdown.  This includes efforts to avoid zombie/orphan processes in the case
  855 of non-collective exits, and proper release of network resources.  This second
  856 point is important because we are aware of multiple network stacks which can
  857 permanently leak system resources if they are not explicitly released.
  858 
  859 We acknowledge that this represents a restriction that may be hard to satisfy.
  860 We fully intend to separate library termination from process termination in a
  861 future release.
  862  
  863 GASNet tracing & statistical collection
  864 =======================================
  865 
  866 GASNet includes an extensive tracing and statistical collection 
  867 utility for monitoring communication events in GASNet applications.
  868 To use, configure with --enable-stats and/or --enable-trace 
  869 (or --enable-debug, which implies both of these by default), and
  870 run your program as usual. In order to see the trace/stats output, 
  871 you must set the environment variable GASNET_TRACEFILE and/or 
  872 GASNET_STATSFILE as explained below.
  873 
  874 Note that system performance is likely to be degraded as a result of tracing and 
  875 statistical collection. This is still true even when output is disabled
  876 by not setting this GASNET_TRACEFILE/GASNET_STATSFILE (so production builds
  877 should not enable tracing/stats at GASNet configure time).
  878 
  879 Optional environment variable settings:
  880 
  881  GASNET_TRACEFILE - specify a file name to receive the trace output
  882    may also be "stdout" or "stderr", (or "-" to indicate stderr)
  883    each node may have its output directed to a separate file, 
  884    and any '%' character in the value is replaced by the node number at runtime
  885    (e.g. GASNET_TRACEFILE="mytrace-%")
  886    unsetting this environment variable (or setting it to empty) disables 
  887    tracing output (although the trace code still has performance impact)
  888 
  889  GASNET_TRACENODES - specify an optional list of nodes on which to generate
  890    tracing output (if enabled by GASNET_TRACEFILE).  List may contain one
  891    or more integers or ranges separated by commas, such as "0,2-4,6".
  892    If empty or equal to "*" then all nodes may generate tracing output.
  893 
  894  GASNET_STATSFILE - specify a filename to receive statistical output
  895    operates analogously to GASNET_TRACEFILE
  896 
  897  GASNET_STATSNODES - limit nodes to generate statistical output
  898    operates analogously to GASNET_TRACENODES
  899 
  900  GASNET_TRACEMASK - specify the types of trace messages to report
  901    A string containing one or more of the following letters:
  902      G - gets
  903      P - puts
  904      R - remote atomics
  905      S - non-blocking synchronization
  906      W - collective operations (excluding barriers)
  907      B - barriers
  908      L - locks
  909      A - AM requests/replies (and handler execution, if conduit-supported)
  910      I - informational messages about system status or performance alerts
  911      C - conduit-specific (low-level) messages
  912      D - Detailed message data for gets/puts/AMreqrep
  913      N - Line number information from client source files
  914      H - High-level messages from the client
  915      U - Unsuppressable messages, which are always output (use with caution)
  916   default: (all of the above)
  917 
  918  GASNET_STATSMASK - specify the types of statistics to collect and report
  919    operates analogously to GASNET_TRACEMASK
  920 
  921  GASNET_TRACEFLUSH - set this variable to force a file system flush 
  922    after every write to the tracefile. This seriously degrades tracing
  923    performance, but ensures any final trace messages before a crash
  924    are flushed into the tracefile.
  925  
  926  GASNET_TRACELOCAL - set to control whether the PG trace messages for
  927    local (i.e. loopback) put/get operations are entered into the trace file 
  928    or suppressed (they are included by default). This can be used to reduce 
  929    tracing overhead and trace file size for clients with a large number of 
  930    local put/get operations which are not of interest.
  931 
  932 GASNet Collectives
  933 ==================
  934 
  935 GASNet includes interfaces for collective operations, which is still
  936 evolving.  The design for the interfaces is located under the source directory
  937 in docs/collective_notes.txt.  Anyone planning to implement a client that
  938 uses these collective should contact us (gasnet-devel@lbl.gov) first to determine
  939 the completeness and stability of the relevant portions of the implementation.
  940 
  941 In GASNet 1.14.0 we introduce a mechanism for auto-tuning of collectives
  942 in which a variety of algorithms are tried for each collective operation in
  943 a user's application and the best choice can be recorded in a file for use
  944 in future runs.  For more information, see the file autotuner.txt in the
  945 docs directory.
  946 
  947 GASNet debug malloc services
  948 ============================
  949 
  950 GASNet includes a debug malloc implementation that can be used to find local
  951 heap corruption bugs in GASNet itself and also in GASNet client applications
  952 (notably Berkeley UPC). To use, configure GASNet with --enable-debug-malloc
  953 (or --enable-debug which implies --enable-debug-malloc), and run your
  954 program as usual.  GASNet will regularly scan the heap for corruption at malloc
  955 events and AMPoll's and report any detected problems.  If you're a GASNet
  956 client writer, you should see the gasnett_debug_malloc functions in
  957 gasnet_tools.h to tie your applications into the GASNet debug malloc services.
  958 
  959 Note the debug malloc implementation imposes some CPU and memory consumption
  960 overhead relative to the system malloc implementation (it's implemented as a
  961 thin wrapper around the system malloc).  This means heap behavior is likely to
  962 be somewhat perturbed relative to the normal mode of operation, however the
  963 overhead is probably less than it would be with third-party heap corruption
  964 tools such as efence or purify (at some cost in lost precision).
  965 
  966 Optional environment variable settings (recognized only when debugging
  967 malloc is enabled):
  968 
  969 GASNET_MALLOC_INIT: When set to 1, every byte of allocated memory is
  970  initialized to the value specified by GASNET_MALLOC_INITVAL. This can be useful
  971  for detecting use of uninitialized data.
  972 
  973 GASNET_MALLOC_CLOBBER: When set to 1, every byte of freed memory is overwritten
  974  with the value specified by GASNET_MALLOC_CLOBBERVAL. This can be useful for
  975  detecting inadvertent use of freed data.
  976 
  977 GASNET_MALLOC_INITVAL, GASNET_MALLOC_CLOBBERVAL: The data value to use for
  978  allocation init or free clobbering, respectively. The value may be specified
  979  as a decimal integer matching the pattern: "-?[0-9]+" or a hexadecimal integer
  980  matching the pattern: "0x[0-9A-F]+".  If the value is between 0 and 255
  981  (inclusive) it is taken to be an 8-bit value which is used to overwrite every
  982  byte in the given region. Otherwise, it is taken to be a 64-bit value which is
  983  used to overwrite the given region in 8-byte chunks.  The value may also be one
  984  of "NAN", "sNAN" or "qNAN", which select a double-precision signaling or quiet
  985  NAN, which is likely to propagate through floating point calculations and cause
  986  segmentation faults when used as a pointer. Both values default to "NAN". 
  987 
  988 GASNET_MALLOC_LEAKALL: set to 1 to leak all freed objects, ensuring they are
  989  not re-allocated during subsequent mallocs. This has an obvious cost in memory
  990  consumption, but can be helpful for tracking bugs in conjunction with
  991  GASNET_MALLOC_CLOBBER for tracking usage of dead objects.
  992 
  993 GASNET_MALLOC_SCANFREED: set to 1 to enable scanning of freed objects, to
  994  detect write-after-free errors. This option implies GASNET_MALLOC_LEAKALL and
  995  GASNET_MALLOC_CLOBBER.
  996 
  997 GASNET_MALLOC_EXTRACHECK: set to 1 to enable more frequent checking for memory
  998  corruption (at a cost in performance)
  999 
 1000 GASNET_MALLOCFILE: specify a file name to receive a report of malloc heap
 1001  behavior and utilization at process exit. Includes a list of leaked objects.
 1002  The filename may also be "stdout" or "stderr", (or "-" to indicate stderr)
 1003  Each node may have its output directed to a separate file,
 1004  and any '%' character in the value is replaced by the node number at runtime
 1005  (e.g. GASNET_MALLOCFILE="mallocreport-%"). These output files are
 1006  human-readable, but can be passed to gasnet_trace for summarization.
 1007 
 1008 GASNET_MALLOCNODES: limit which nodes will generate a malloc report.
 1009    operates analogously to GASNET_TRACENODES
 1010 
 1011 GASNet inter-Process SHared Memory (PSHM)
 1012 =========================================
 1013 
 1014 GASNet's PSHM support provides mechanism to communicate through shared memory among
 1015 processes on the same compute node (ie sharing a cache-coherent physical memory).
 1016 Inter-process communication through shared memory is usually the fastest way for
 1017 processes sharing a compute node to communicate. GASNet clients are free to use
 1018 any combination of processes communicating through PSHM and client pthreads within 
 1019 processes (in GASNET_PAR mode) to implement on-node parallelism, and all such 
 1020 execution contexts may perform network communication (with an appropriate conduit).
 1021 
 1022 GASNet's PSHM support is enabled by default unless one is cross-compiling, or
 1023 running Cygwin prior to version 2.0.  However, most cross-compilation scripts
 1024 enable it with any additional settings as required.  If desired, PSHM can be
 1025 disabled by passing --disable-pshm at configure time.
 1026 
 1027 In addition to configure-time control, the environment variable
 1028 GASNET_SUPERNODE_MAXSIZE sets a limit on the number of processes that will be
 1029 grouped into a shared-memory "supernode".  By default all co-located processes
 1030 are grouped into one supernode, but the value can be set to 1 to disable the
 1031 use of shared-memory for communication.
 1032 
 1033 The PSHM support in GASNet can operate via three generic mechanisms: POSIX
 1034 shared memory, SystemV shared memory, or mmap()ed disk files.
 1035 
 1036 POSIX shared memory is recommended as the best option in most cases, but is
 1037 not always available (many kernels can be configured not to support it), or
 1038 may not permit large-enough allocations.  See "System Settings for POSIX
 1039 Shared Memory", below, for information on sizing of allocations.
 1040 
 1041 In the absence of the POSIX shared memory, users are advised to use the
 1042 SystemV shared memory as the next-best option.  However, see "System Settings
 1043 for SystemV Shared Memory", below, for information about configuring a system
 1044 to use SystemV shared memory.
 1045 
 1046 In the absence of both POSIX and SystemV shared memory, a user may try
 1047 using mmap()ed disk files.  However, on some systems we see significant
 1048 performance degradation when using files (apparently due to committing the
 1049 changes from memory to disk).
 1050 
 1051 Unless PSHM is disabled, the default behavior of the configure step is to probe
 1052 for support via exactly one preferred mechanism.  For Solaris the preferred
 1053 mechanism is SystemV, while for all other platforms it is POSIX.  If the
 1054 preferred mechanism is not supported (or is explicitly disabled) the then there
 1055 is no automatic fallback to any other mechanism.  So, if one wishes to use
 1056 another mechanism, one should explicitly disable the default (POSIX or SysV)
 1057 support and enable the desired mechanism:
 1058 
 1059         Common Usage Summary (flags to pass to the configure script):
 1060         ----------------------------------------------------------
 1061           OFF: --disable-pshm
 1062         POSIX: no flags required (default unless cross-compiling)
 1063          SYSV: --disable-pshm-posix --enable-pshm-sysv
 1064          FILE: --disable-pshm-posix --enable-pshm-file
 1065 
 1066         Solaris Usage Summary (as above except that SysV is default)
 1067         ----------------------------------------------------------
 1068           OFF: --disable-pshm
 1069         POSIX: --disable-pshm-sysv --enable-pshm-posix
 1070          SYSV: no flags required (default)
 1071          FILE: --disable-pshm-sysv --enable-pshm-file
 1072 
 1073 There are also implementations which are specific to certain platforms:
 1074 
 1075 For Cray XE, XK and XC systems, PSHM-over-XPMEM is enabled automatically
 1076 by the corresponding cross-configure scripts.  However, one can also
 1077 optionally enable PSHM-over-XPMEM on an SGI Altix as follows:
 1078         XPMEM: --enable-pshm --disable-pshm-posix --enable-pshm-xpmem
 1079 
 1080 For BlueGene/Q, PSHM via the "common heap" is enabled automatically by
 1081 the corresponding cross-configure scripts.  However, one can also use:
 1082         GHEAP: --enable-pshm --disable-pshm-posix --enable-pshm-gheap
 1083 Note on BG/Q one must set BG_MAPCOMMONHEAP=1 in the compute node environment.
 1084 This is handled automatically by the gasnetrun_mpi and gasnetrun_pami
 1085 scripts, but must be handled manually when using runjob directly (or
 1086 a wrapper like qsub from ANL's Cobalt suite).
 1087 
 1088 For Linux systems with libhugetlbfs and a hugetlbfs mount, there is also
 1089 experimental support for mmap()ed files on hugetlbfs, which can be
 1090 enabled with:
 1091     HUGETLBFS: --disable-pshm-posix --enable-pshm-hugetlbfs
 1092 
 1093 PSHM includes an environment variable for controlling use of memory for
 1094 intra-memory AM traffic:
 1095 
 1096   GASNET_PSHMNET_QUEUE_DEPTH
 1097      Minimum number of PSHM-AMs a process must be capable of sending before
 1098      it may stall (default 32).  The shared memory allocated on a compute
 1099      node with P processes is roughly (2 * P * Depth * MaxMsgSz).
 1100 
 1101 System Settings for POSIX Shared Memory:
 1102 ---------------------------------------
 1103 On most systems (all but Solaris and some cross-compiled platforms), the
 1104 default implementation of PSHM uses POSIX shared memory.  On many operating
 1105 systems the amount of available POSIX shared memory is controlled by the
 1106 sizing of a pseudo-filesystem that consumes space in memory (and sometimes
 1107 swap space) rather than stable storage.  If this filesystem is not large
 1108 enough, it can limit the amount of POSIX shared memory which can be allocated
 1109 for the GASNet segment.
 1110 
 1111 Insufficient available POSIX shared memory may either lead to failures at
 1112 start-up, or to SIGBUS or SIGSEGV later in a run (if the OS has permitted
 1113 allocation of more virtual address space than is actually available).  If one
 1114 encounters either of these failure modes when GASNet is configured to use
 1115 POSIX shared memory, then one should check the space available (see below) and
 1116 may need to increase the corresponding system settings.  Setting these
 1117 parameters is system-specific and requires administrator privileges.
 1118 
 1119 On most modern Linux distribution, POSIX shared memory allocations reside in
 1120 the /dev/shm filesystem (of type 'tmpfs'), though /var/shm and /run/shm are
 1121 also used in some cases.  The mechanism for sizing of this filesystem varies
 1122 greatly between distributions.  Please consult the documentation for your
 1123 specific Linux distribution for instructions to resize.  In particular, be
 1124 advised that distributions may mount this filesystem early in the boot
 1125 process, without regards to any entry in /etc/fstab.
 1126 
 1127 On macOS, Cygwin and Solaris, there are no settings (known to us at this time)
 1128 required to increase the space available for allocation as POSIX shared
 1129 memory.
 1130 
 1131 On FreeBSD, POSIX shared memory allocations reside in the /tmp filesystem,
 1132 which is normally of type 'tmpfs'.  This filesystem often defaults to having
 1133 no limit on its size other than the sum of physical memory and swap space.
 1134 However, the size can be set in /etc/fstab.  See the fstab(5) and tmpfs(5)
 1135 manpages for more information.
 1136 
 1137 On NetBSD, POSIX shared memory allocations reside in the /var/shm filesystem
 1138 (of type 'tmpfs'), which can be sized in /etc/fstab.  See the fstab(5) and
 1139 mount_tmpfs(8) manpages for more information.
 1140 
 1141 On OpenBSD, POSIX shared memory allocations reside in the /tmp filesystem,
 1142 which is normally a filesystem of type 'mfs' and can be sized in /etc/fstab.
 1143 See the fstab(5) and mount_mfs(8) manpages for more information.
 1144 
 1145 System Settings for SystemV Shared Memory:
 1146 -----------------------------------------
 1147 SystemV shared memory is the second most widely used implementation of PSHM
 1148 after POSIX shared memory.  It is used by default on Solaris, and is
 1149 recommended on most other systems when POSIX does not work for any reason.
 1150 On most operating systems the amount of available SystemV shared memory and
 1151 the number of shared memory segments is controlled by the kernel parameters:
 1152 shmmax, shmall and shmmni.
 1153    shmmax = largest size of a shared memory segment (in bytes)
 1154    shmall = total amount of memory allocatable as shared (in pages)
 1155    shmmni = maximum number of shared memory segments
 1156 
 1157 Insufficient amount of SystemV shared memory will lead to failures at
 1158 start-up of any application using a runtime configured to use PSHM over
 1159 SystemV.  Setting these parameters is system-specific and requires
 1160 administrator privileges.
 1161 
 1162 * Examples of configuration:
 1163 
 1164 These examples are for 64-bit systems, and essentially allow
 1165 unlimited amounts of memory to be used for GASNet segments.
 1166 
 1167 These assume you are running no more than 127 processes per
 1168 node.  If you have more cores (or cpu threads) then increase
 1169 the shmmni value to the maximum number of processes you wish
 1170 to be be able to run, plus 1.  Some examples below do not
 1171 include a setting for shmmni because the system defaults are
 1172 large enough in general, and often cannot be reduced.
 1173 
 1174 For 32-bit systems, use 2147483647 for the shmmax value to
 1175 avoid a potential overflow.
 1176 
 1177    - Linux:
 1178         Add the following three lines to /etc/sysctl.conf (sudo required):
 1179 kernel.shmmax=1099511627776
 1180 kernel.shmall=268435456
 1181 kernel.shmmni=128
 1182 
 1183         To activate the new settings reboot, or run the following:
 1184         sudo /sbin/sysctl -p /etc/sysctl.conf
 1185 
 1186    - macOS:
 1187         Add the following three lines to /etc/sysctl.conf (sudo required):
 1188 kern.sysv.shmmax=1099511627776
 1189 kern.sysv.shmall=268435456
 1190 kern.sysv.shmmni=128
 1191 
 1192         To activate the new settings reboot, or run the following:
 1193         grep ^kern.sysv.shm /etc/sysctl.conf | sudo xargs /usr/sbin/sysctl -w
 1194 
 1195    - FreeBSD:
 1196         Add the following two lines to /etc/sysctl.conf (sudo required):
 1197 kern.ipc.shmmax=1099511627776
 1198 kern.ipc.shmall=268435456
 1199 
 1200         To activate the new settings reboot, or run the following:
 1201         sudo /sbin/sysctl -f /etc/sysctl.conf
 1202 
 1203    - NetBSD:
 1204         Add the following two lines to /etc/sysctl.conf (sudo required):
 1205 kern.ipc.shmmax=1099511627776
 1206 kern.ipc.shmmaxpgs=268435456
 1207 
 1208         To activate the new settings reboot, or run the following:
 1209         sudo /sbin/sysctl -f /etc/sysctl.conf
 1210 
 1211    - OpenBSD:
 1212         Add the following two lines to /etc/sysctl.conf (sudo required):
 1213 kern.shminfo.shmmax=2147483647
 1214 kern.shminfo.shmall=268435456
 1215 
 1216         To activate the new settings reboot, or run the following:
 1217         grep ^kern.shminfo /etc/sysctl.conf | sudo xargs /sbin/sysctl -w
 1218 
 1219    - Solaris:
 1220         Depends on version.  Please see the Sun/Oracle documentation.
 1221         However, on recent releases the defaults are sufficient.
 1222 
 1223    - Cygwin:
 1224         If you have not done so yet, please Read the Cygwin documentation
 1225         on "cygserver-config" to create an initial /etc/cygsever.conf
 1226         and start the server as a Windows service (optional).
 1227         You may then edit /etc/cygserver to edit
 1228           kern.ipc.shmmaxpgs
 1229           kern.ipc.shmmni
 1230           kern.ipc.shmseg
 1231         Except for shmmaxpgs, the defaults are often large enough.
 1232         These configuration values are only read when cygserver starts.
 1233         So, read the Cygwin documentation to determine if/how to restart
 1234         the cygserver service.
 1235         Under Cygwin-1.5 you may also need to add "server" to the value
 1236         of the CYGWIN environment variable.  Again, you should see the
 1237         Cygwin documentation for more information on this subject.
 1238 
 1239 Warning to users of macOS:
 1240 -------------------------
 1241 We have evidence which suggests that there is a kernel bug in all releases of
 1242 macOS though at least 10.12 (Sierra) which leads to a small leak of kernel
 1243 memory (perhaps a few 10s of bytes) each time POSIX or SystemV shared memory
 1244 is used.  This may, after tens of thousands of runs of GASNet applications
 1245 using PSHM, lead to a low memory condition contributing to slow performance
 1246 and eventually to total memory exhaustion.  In normal usage, it is reasonable
 1247 to expect that a Mac laptop or desktop system will be rebooted for software
 1248 updates frequently enough that this leak will not impact normal users.
 1249 Also note that --enable-pshm-file is NOT recommended for use on macOS, where
 1250 it has been seen to cause VERY slow startup times.
 1251 
 1252 IMPORTANT: SYSTEM CLEANING OF PSHM OBJECTS:
 1253 ------------------------------------------
 1254 If a GASNet application using PSHM is terminated before ending the
 1255 initialization phase, there is a possibility that the shared memory objects
 1256 will remain in the system.  A large amount of memory or disk space can
 1257 remain allocated, preventing users from fully utilizing all available
 1258 hardware resources.
 1259 
 1260 In the SystemV case, the allocated (but not released) shared memory
 1261 segments can be listed via the "ipcs" command, and can be removed via the
 1262 "ipcrm" command.  Note that on the systems with a batch scheduler, the
 1263 "ipcs" and "ipcrm" instructions need to be run on the compute nodes.
 1264 
 1265 In the mmap()ed file case, the allocated but not released shared memory files
 1266 can be found in the directory pointed by the TMPDIR environment variable
 1267 (default: /tmp). These files are named with the prefix GASNT (the lack of an
 1268 'E' is not a typo), and can be deleted using the "rm" command.
 1269 
 1270 The case of mmap()ed file on hugetlbfs is like the regular mmap()ed file
 1271 case except that the directory will be the hugetlbfs mount point.  This is
 1272 typically something like /dev/hugepages.  If uncertain of the mount point,
 1273 the command "mount|grep hugetlbfs" should locate it for you.
 1274 
 1275 The case of POSIX shared memory is OS-specific, but in most cases the
 1276 shared memory objects are visible in the file system.  On systems we use
 1277 for development, we have observed the following defaults:
 1278 * Linux:   located in /dev/shm with a filename prefix of "GASNT"
 1279 * Cygwin:  located in /dev/shm with a filename prefix of "GASNT"
 1280 * Solaris: located in /tmp with a filename prefix of ".SHMDGASNT"
 1281 * OpenBSD: located in /tmp with long random names suffixed by ".shm"
 1282 * NetBSD:  located in /var/shm with a filename prefix of ".shmobj_GASNT"
 1283 
 1284 MPI Interoperability
 1285 ====================
 1286 
 1287 The Message Passing Interface (MPI) is a ubiquitous portable interface for
 1288 communication in scientific computing and is used both directly and indirectly
 1289 by various applications and libraries.  In most cases GASNet is implemented
 1290 natively on the various SAN interconnects, bypassing the MPI layer in order to
 1291 provide the best possible performance. This means that in general GASNet does
 1292 NOT use MPI for performing communication (the only notable exception being
 1293 mpi-conduit, which is a portable implementation of GASNet over MPI).  In some
 1294 cases GASNet can use MPI to assist in parallel job creation and termination
 1295 (thereby taking advantage of the existing MPI infrastructure), but in those
 1296 cases it will not use MPI during the steady state of operation - only at
 1297 startup and shutdown.
 1298 
 1299 Although GASNet is not implemented using MPI, it is *compatible* with MPI -
 1300 GASNet and MPI can be used together within the same network and even within the
 1301 same program.  However, there are a few subtle issues to be dealt with when
 1302 using both communication systems in the same process, because in those cases
 1303 GASNet and MPI must both be initialized properly and cooperate to "share" the
 1304 resources of the network and physical memory. This section describes issues
 1305 that arise when combining the use of MPI-based communication and GASNet
 1306 communication within a single process. The information described here applies
 1307 to any case where a single process attempts to access the network hardware
 1308 using both GASNet and MPI, even when such usage occurs at different levels
 1309 within a layered application (for example, a UPC application which uses GASNet
 1310 via the Berkeley UPC compiler and makes calls to a helper library that
 1311 communicates using MPI).
 1312 
 1313 MPI requires exactly one initialization call before MPI communication can be used
 1314 within a process.  The GASNet implementation may or may not automatically
 1315 initialize the MPI library, depending on the conduit and various configuration
 1316 settings. Therefore, processes using both communication libraries must be
 1317 prepared to correctly handle either situation - the recommended way to
 1318 arbitrate this is using the MPI_Initialized() call which reports whether or not
 1319 the MPI layer has been initialized.  Here is some example code for
 1320 accomplishing this:
 1321 
 1322 int isMPIinit;
 1323 int main(int argc, char **argv) {
 1324   size_t segment_size = 64*1024*1024; /* want 64MB segment in this example */
 1325   size_t segment_max;
 1326 
 1327   gasnet_init(&argc, &argv);
 1328 
 1329   segment_max = gasnet_getMaxGlobalSegmentSize();
 1330   if (segment_size > segment_max) segment_size = segment_max;
 1331 
 1332   gasnet_attach(NULL, 0, segment_size, 0);
 1333 
 1334   gasnet_barrier_notify(0, GASNET_BARRIERFLAG_ANONYMOUS);
 1335   gasnet_barrier_wait(0, GASNET_BARRIERFLAG_ANONYMOUS);
 1336 
 1337   if (MPI_Initialized(&isMPIinit) != MPI_OK) { /* test if MPI already init */
 1338     fprintf(stderr, "Error calling MPI_Initialized()\n");
 1339     abort();
 1340   }
 1341   if (!isMPIinit) MPI_Init(argc, argv); /* MPI not init, so do it */
 1342 
 1343   /* ... use MPI as usual ... */
 1344 
 1345   MPI_Barrier(MPI_COMM_WORLD);
 1346 
 1347   /* ... use GASNet as usual ... */
 1348 
 1349   gasnet_barrier_notify(0, GASNET_BARRIERFLAG_ANONYMOUS);
 1350   gasnet_barrier_wait(0, GASNET_BARRIERFLAG_ANONYMOUS);
 1351 
 1352   ...
 1353 
 1354   if (!isMPIinit) MPI_Finalize(); 
 1355   return 0;
 1356 }
 1357 
 1358 The MPI_Initialized call checks to make sure MPI hasn't already been
 1359 initialized by GASNet, which will be the case on mpi-conduit and possibly on
 1360 other conduits that use MPI for job startup. Any MPI usage internal to GASNet
 1361 uses a private MPI communicator, preventing any potential interference with MPI
 1362 use by the GASNet client. The code which initializes the MPI layer is also
 1363 responsible for finalizing it before exit.
 1364 
 1365 An extensive example is available in tests/testmpi.c, which implements various
 1366 interoperability tests between GASNet and MPI.
 1367 
 1368 There are a few other important caveats to be aware of when mixing GASNet and MPI:
 1369 
 1370 * GASNet must be configured with MPI_CC set to the exact *same* MPI
 1371  installation that will be used when building any GASNet client code using MPI.
 1372  Note that many MPI implementations are not binary-compatible across versions,
 1373  so upgrading the MPI compiler usually requires a fresh configure/build/install
 1374  of GASNet with the new MPI implementation if you intend to mix GASNet and MPI.
 1375 
 1376 * When a node is inside a blocking MPI call, most conduits will be prevented
 1377  from servicing many GASNet requests from remote nodes. Similarly, GASNet
 1378  blocking calls will generally not ensure progress of MPI communication. This
 1379  means that arbitrarily interleaving blocking MPI calls with GASNet calls
 1380  (e.g. UPC shared accesses) can easily lead to deadlock. In order to ensure
 1381  safety and portably correct operation across networks, GASNet communication
 1382  operations and user MPI calls should be isolated from each other by barriers, 
 1383  by strictly adhering to the following "Time-Phasing" protocol:
 1384 
 1385    - When the application starts, the first MPI or GASNet call issued from any
 1386      node should be considered to put the application in 'MPI' or 'GASNet'
 1387      mode, respectively. 
 1388 
 1389    - When an application is in 'MPI' mode, and needs to switch to using GASNet,
 1390      it should collectively execute an 'MPI_Barrier()' as the last MPI call
 1391      before issuing any GASNet communication calls. Once any GASNet communication has
 1392      occurred from any node, the program should be considered to have switched to
 1393     'GASNet' mode. 
 1394 
 1395    - When an application is in 'GASNet' mode, and an MPI call that may cause
 1396      network traffic is needed, a collective call to 'gasnet_barrier_notify()' followed
 1397      by a 'gasnet_barrier_wait()' should be executed as the last GASNet communication
 1398      before any MPI calls are made. Once any MPI functions have been called from any
 1399      thread, the program should be considered to be in 'MPI' mode. 
 1400 
 1401  If this simple construct--GASNet code must be followed by a GASNet barrier,
 1402  and MPI code must be followed by an MPI_Barrier--is followed, this deadlock should
 1403  not occur. This protocol is demonstrated in the code above.
 1404 
 1405 * GASNet and MPI both number processes with a unique non-negative index (i.e. gasnet_mynode()
 1406   and MPI_Comm_rank(MPI_COMM_WORLD)). Every attempt is made to ensure this
 1407   numbering matches between layers, but it might not always be possible to ensure this. 
 1408   To maximize portability, apps using both layers should use a GASNet or MPI collective 
 1409   (e.g. AllGather) to build a table of ids and translate rank identifiers as needed.
 1410 
 1411 * Note that GASNet language clients such as UPC or Titanium might include 
 1412   compilation modes where multiple language-level threads are implemented as pthreads
 1413   within a single process, which will map to a single GASNet node id (and hence
 1414   a single MPI rank). Clients using such threading should ensure they are using
 1415   a pthread-safe implementation of MPI (GASNet is thread-safe by virtue of PAR mode),
 1416   and be aware that MPI messages sent to a particular node might be received by any
 1417   pthread at the target (unless steps are taken to avoid this, e.g. using an MPI tag).
 1418 
 1419 * There are various network-specific issues that may arise when combining
 1420  GASNet with specific MPI implementations, most notably related to job creation
 1421  and spawning mechanisms. Please see the README file in each conduit directory
 1422  for network-specific discussion of issues with MPI interoperability.
 1423 
 1424 Contact Info and Support
 1425 ========================
 1426 
 1427 For the latest GASNet downloads, publications, and specifications, please visit:
 1428 
 1429   https://gasnet.lbl.gov
 1430 
 1431 For bug reports and feature requests, please submit a ticket in the GASNet Bugzilla:
 1432 
 1433   https://gasnet-bugs.lbl.gov
 1434 
 1435 You may find an instant solution to your problem by searching the bug database!
 1436 
 1437 GASNet has several mailing lists for support:
 1438 
 1439   gasnet-users@lbl.gov     General questions or inquiries regarding the installation or 
 1440                            use of GASNet. Any users can join the list from the web site above.
 1441 
 1442   gasnet-devel@lbl.gov     GASNet developers list (multi-institution)
 1443 
 1444   gasnet-announce@lbl.gov  GASNet release announcements. Join on the web site above.
 1445 
 1446 --------------------------------------------------------------------------
 1447   The canonical version of this document is located here:
 1448     https://bitbucket.org/berkeleylab/gasnet/src/master/README
 1449 
 1450   For more information, please email: gasnet-users@lbl.gov
 1451   or visit the GASNet home page at:   https://gasnet.lbl.gov
 1452 --------------------------------------------------------------------------