"Fossies" - the Fresh Open Source Software Archive

Member "berkeley_upc-2019.4.2/gasnet/mxm-conduit/README" (27 May 2019, 8580 Bytes) of package /linux/misc/berkeley_upc-2019.4.2.tar.gz:


As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. See also the last Fossies "Diffs" side-by-side code changes report for "README": 2.28.0_vs_2019.4.0.

    1 GASNet mxm-conduit documentation
    2 ============================================
    3 
    4 =====================================================================
    5 NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE
    6 NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE
    7 =====================================================================
    8 
    9 The mxm-conduit is now DEPRECATED.
   10 No further development of this conduit is anticipated.
   11 It may be removed in a future release.
   12 
   13 Users with InfiniBand hardware are recommended to use ibv-conduit
   14 by passing the following to GASNet's configure script.
   15   --enable-ibv --disable-mxm
   16 
   17 =====================================================================
   18 NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE
   19 NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE
   20 =====================================================================
   21 
   22 
   23 User Information:
   24 -----------------
   25 
   26 Mxm-conduit implements GASNet over the Mellanox Messaging Accelerator (MXM)
   27 API for recent Mellanox InfiniBand Host Channel Adapters (HCAs).
   28 
   29 See http://www.mellanox.com/products/mxm for general information on MXM,
   30 including supported hardware and OS releases.
   31 
   32 Where this conduit runs:
   33 -----------------------
   34 
   35 While nearly all InfiniBand HCAs can run ibv-conduit, mxm-conduit is able to
   36 take fuller advantage of the underlying infrastructure in Mellanox's switches
   37 and HCAs.  One significant difference is improved scalability to large node
   38 counts, due to MXM's support for Dynamic Connection (DC) transport mode.
   39 
   40 Mxm-conduit is expected to work with all releases of MXM.
   41 
   42 The relative performance of mxm and ibv conduits can depend on many factors.
   43 For this reason we cannot categorically recommend one over the other.  We
   44 encourage users concerned with obtaining the best possible performance to
   45 benchmark your own workload when selecting a conduit.
   46 
   47 Optional compile-time settings:
   48 ------------------------------
   49 
   50 The default spawner to be used by the gasnetrun_mxm utility can be
   51 selected by configuring '--with-mxm-spawner=VALUE', where VALUE is one
   52 of 'mpi', 'pmi' or 'ssh'.  If this option is not used, mpi is the
   53 default when available, and ssh otherwise.
   54 Here are some things to consider when selecting a default spawner:
   55   + mpi-spawner is the default when MPI is available precisely because it
   56     is so frequently present on systems where GASNet is to be installed.
   57     Additionally, very little (if any) configuration is required and the
   58     behavior is highly reliable.
   59   + pmi-spawner uses the same "Process Management Interface" which forms
   60     the basis for many mpirun implementations.  When support is available,
   61     this spawner can be as easy to use and as reliable as mpi-spawner, but
   62     without the overheads of initializing an MPI runtime.
   63   + ssh-spawner depends only on the availability of a remote shell command
   64     such as ssh.  For this reason ssh-spawner support is always compiled.
   65     However, it can be difficult (or impossible) to use on a cluster which
   66     was not setup to allow ssh to (and among) its compute nodes.
   67 For more information on configuration and use of these spawners, see
   68    README-{ssh,mpi,pmi}-spawner (installed)
   69 or
   70    other/{ssh,mpi,pmi}-spawner/README (source).
   71 
   72 Job Spawning:
   73 -------------
   74 
   75 If using UPC, Titanium, etc. the language-specific commands should be used
   76 to launch applications.  Otherwise, applications can be launched using
   77 the gasnetrun_mxm utility:
   78 + usage summary:
   79     gasnetrun_mxm -n <n> [options] [--] prog [program args]
   80     options:
   81       -n <n>                 number of processes to run (required)
   82       -N <N>                 number of nodes to run on (not supported by all MPIs)
   83       -E <VAR1[,VAR2...]>    list of environment vars to propagate
   84       -v                     be verbose about what is happening
   85       -t                     test only, don't execute anything (implies -v)
   86       -k                     keep any temporary files created (implies -v)
   87       -spawner=(ssh|mpi|pmi) force use of a specific spawner (if available)
   88 
   89 There are as many as three possible methods (ssh, mpi and pmi) by which one
   90 can launch an mxm-conduit application.  Ssh-based spawning is always
   91 available, and mpi- and pmi-based spawning are available if the respective
   92 support was located at configure time.  The default is established at
   93 configure time (see section "Optional compile-time settings").
   94 
   95 To select a non-default spawner one may either use the "-spawner=" command-
   96 line argument or set the environment variable GASNET_MXM_SPAWNER to "ssh",
   97 "mpi" or "pmi".  If both are used, then the command line argument takes
   98 precedence.
   99 
  100 Recognized environment variables:
  101 ---------------------------------
  102 
  103 * All the standard GASNet environment variables (see top-level README)
  104 
  105 * GASNET_PHYSMEM_MAX (default is set at configure time, see below)
  106   If set non-zero this parameter tells mxm-conduit the maximum amount of
  107   memory to pin for the segment in a (default) GASNET_SEGMENT_FAST build.  The
  108   value may specify either a relative or absolute size.  If the value parses
  109   as a floating-point value less than 1.0 (including fractions such as "5/8"),
  110   then this is taken as a fraction of the (estimated) physical memory.
  111   Otherwise the value is taken as an absolute memory size, with "M", "G" and
  112   "T" suffixes accepted to indicate units of Megabytes, Gigabytes, and
  113   Terabytes, respectively.
  114   For backwards compatibility, "0" means 2/3 of estimated physical memory.
  115   The default may be set at configure time using --with-mxm-physmem-max=VALUE,
  116   and otherwise is "2/3" (pin up to 2/3 of the estimated physical memory).
  117 
  118   This is a per-node value - if there are multiple processes per shared memory
  119   node the value given by this variable will be divided by the number of
  120   processes per shared memory node to determine the maximum segment size.
  121 
  122 * GASNET_PHYSMEM_PROBE (default is set at configure time, see below)
  123   This gives a boolean: "0" to disable or "1" to force a probe to verify the
  124   value given by GASNET_PHYSMEM_MAX.  Enabling this setting may greatly slow
  125   startup, but can avoid unexpected runtime failures if GASNET_PHYSMEM_MAX
  126   exceeds the limits imposed by the O/S and HCA.
  127   The default is OFF (probe disabled).  However, this can be changed to a
  128   default of ON by configuring using --enable-mxm-physmem-probe.
  129 
  130 * GASNET_MXM_MAX_OUTSTANDING_MSGS (default = 500)
  131   This gives the maximum number of MXM-level operations which can be in-flight
  132   at any given time due to GASNet NBI Puts and Gets.
  133 
  134 * GASNET_MXM_SPAWNER (default set at configure time)
  135   To override the default spawner for mxm-conduit jobs, one may set this
  136   environment variable as described in the section "Job Spawning", above.
  137   There are additional settings which control behaviors of the various
  138   spawners, as described in the respective READMEs (listed in section
  139   "Optional compile-time settings", above).
  140 
  141 * MXM environment variables documented in the Release Notes which accompany
  142   your release of MXM.
  143 
  144 Known problems:
  145 ---------------
  146 
  147 * See the GASNet Bugzilla server for details on known bugs:
  148   https://gasnet-bugs.lbl.gov/
  149 
  150 Future work:
  151 ------------
  152 
  153 ==============================================================================
  154 
  155 Design Overview:
  156 ----------------
  157 
  158 Header Files
  159 ------------
  160 
  161     <gasnet_core_fwd.h>
  162         Definitions of basic constants and types used by the core API and
  163         included early in the gasnet.h file (gasnet.h provides some default
  164         values for various constants and types that can be overridden here).
  165 
  166     <gasnet_core.h>
  167         Provides the public interface for the GASNet core API (function
  168         prototypes and/or inline functions for implementing the core API).
  169 
  170     <gasnet_core_help.h>
  171         This is the core's entry point to provide access to private symbols
  172         which might be required by the core header (e.g.: node's node id,
  173         number of nodes in job), but which should not really be publicly
  174         visible to client code (i.e. not part of GASNet API).
  175 
  176     <gasnet_extended.h>
  177         Provides the public interface for the GASNet extended API
  178 
  179     <gasnet_core_internal.h>
  180         This file is NOT included by gasnet.h (and therefore not visible to
  181         client code) and are exclusively for use in building the conduit
  182         libraries.
  183         <gasnet_internal.h> is the conduit-independent version of this file
  184         at the top level.
  185         Internal GASNet implementation should include <gasnet_internal.h>
  186         instead of <gasnet.h>.
  187 
  188 
  189 << TBD >>