"Fossies" - the Fresh Open Source Software Archive

Member "relax-5.0.0/multi/__init__.py" (2 Dec 2019, 14645 Bytes) of package /linux/privat/relax-5.0.0.src.tar.bz2:


As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) Python source code syntax highlighting (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. For more information about "__init__.py" see the Fossies "Dox" file reference documentation and the latest Fossies "Diffs" side-by-side code changes report: 4.1.3_vs_5.0.0.

    1 ###############################################################################
    2 #                                                                             #
    3 # Copyright (C) 2007 Gary S Thompson                                          #
    4 # Copyright (C) 2008,2012 Edward d'Auvergne                                   #
    5 #                                                                             #
    6 # This file is part of the program relax (http://www.nmr-relax.com).          #
    7 #                                                                             #
    8 # This program is free software: you can redistribute it and/or modify        #
    9 # it under the terms of the GNU General Public License as published by        #
   10 # the Free Software Foundation, either version 3 of the License, or           #
   11 # (at your option) any later version.                                         #
   12 #                                                                             #
   13 # This program is distributed in the hope that it will be useful,             #
   14 # but WITHOUT ANY WARRANTY; without even the implied warranty of              #
   15 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #
   16 # GNU General Public License for more details.                                #
   17 #                                                                             #
   18 # You should have received a copy of the GNU General Public License           #
   19 # along with this program.  If not, see <http://www.gnu.org/licenses/>.       #
   20 #                                                                             #
   21 ###############################################################################
   22 
   23 # Package docstring.
   24 """The multi-processor package.
   25 
   26 1 Introduction
   27 ==============
   28 
   29 This package is an abstraction of specific multi-processor implementations or fabrics such as MPI via mpi4py.  It is designed to be extended for use on other fabrics such as grid computing via SSH tunnelling, threading, etc.  It also has a uni-processor mode as the default fabric.
   30 
   31 
   32 2 API
   33 =====
   34 
   35 The public API is available via the __init__ module.  It consists of a number of functions and classes.  Using this basic interface, code can be parallelised and executed via an MPI implementation, or default back to a single CPU when needed.  The choice of processor fabric is up to the calling program (via multi.load_multiprocessor).
   36 
   37 
   38 2.1 Program initialisation
   39 --------------------------
   40 
   41 The function multi.load_multiprocessor() is the interface for how a program can load and set up a specific processor fabric.  This function returns the set up processor, which itself provides a run() method which is used to execute your application.
   42 
   43 
   44 2.2 Access to the processor instance
   45 ------------------------------------
   46 
   47 The multi.Processor_box class is a special singleton object which provides access to the processor object.  This is required for a number of actions:
   48 
   49     - Queuing of slave commands and memos via Processor_box().processor.add_to_queue().
   50     - Returning results (as a Results_command) from the slave processor to the master via Processor_box().processor.return_object().
   51     - Determining the number of processes via Processor_box().processor.processor_size().
   52     - Waiting for completion of the queued slave processors via Processor_box().processor.run_queue().
   53 
   54 
   55 2.3 Slaves
   56 ----------
   57 
   58 Slave processors are created via the multi.Slave_command class.  This is special base class which must be subclassed.  The run() function should be overridden, this provides the code to execute on the slave processors.
   59 
   60 
   61 2.4 Results handling
   62 --------------------
   63 
   64 The multi.Result_command class is a special base class which must be subclassed.  The run() function should be overridden, this provides the code for the master to process the results from the slaves.
   65 
   66 In addition, the multi.Memo should also be used.  This is a special base class which must be subclassed.  This is a data store used by the Results_command to help process the results from the slave on the master processor.
   67 
   68 
   69 
   70 3 Parallelisation
   71 =================
   72 
   73 The following are the steps required to parallelise a calculation via the multi-processor package API.  It is assumed that the multi.load_multiprocessor() function has been set up at the highest level so that the entire program will be executed by the returned processor's run() method.
   74 
   75 
   76 3.1 Subclassing command and memo objects
   77 ----------------------------------------
   78 
   79 The first step is that the Slave_command, Result_command, and Memo classes need to be subclassed.  The Slave_command.run() method must be provided and is used for running the calculations on the slave processors.  The Result_command is used to unpack the results from the slave.  It is initialised by the Slave_command itself with the results from the calculation as arguments of __init__().  Its run() method processes the results on the master processor.  The Memo object holds data other than the calculation results required by the Result_command.run() method to process the results.
   80 
   81 
   82 3.2 Initialisation and queuing
   83 ------------------------------
   84 
   85 The second step is to initialise the Slave_command and Memo and add these to the processor queue.  But first access to the processor is required.  The singleton multi.Processor_box should be imported, and the processor accessed with code such as::
   86 
   87     # Initialise the Processor box singleton.
   88     processor_box = Processor_box() 
   89 
   90 The slave command is then initialised and all required data by the slave for the calculation (via its run() method) is stored within the class instance.  The memo is also initialised with its data required for the result command for processing on the master of the results from the slave.  These are then queued on the processor::
   91 
   92     # Queue the slave command and memo.
   93     processor_box.processor.add_to_queue(command, memo)
   94 
   95 
   96 3.3 Calculation
   97 ---------------
   98 
   99 To execute the calculations, the final part of the calculation code on the master must feature a call to::
  100 
  101     processor_box.processor.run_queue().
  102 
  103 
  104 4 Example
  105 =========
  106 
  107 See the script 'test_implementation.py' for a basic example of a reference, and full, implementation of the multi-processor package.
  108 
  109 
  110 5 Issues
  111 ========
  112 
  113 For multi-core systems and Linux 2.6, the following might be required to prevent the master processor from taking 100% of one CPU core while waiting for the slaves:
  114 
  115 # echo "1" > /proc/sys/kernel/sched_compat_yield
  116 
  117 This appears to be an OpenMPI problem with late 2.6 Linux kernels.
  118 """
  119 
  120 
  121 __all__ = ['memo',
  122            'misc',
  123            'mpi4py_processor',
  124            'multi_processor_base',
  125            'processor',
  126            'processor_io',
  127            'result_commands',
  128            'result_queue',
  129            'slave_commands',
  130            'uni_processor']
  131 
  132 # Python module imports.
  133 import sys as _sys
  134 import traceback as _traceback
  135 
  136 # Multi-processor module imports.
  137 from multi.memo import Memo
  138 from multi.misc import import_module as _import_module
  139 from multi.misc import Verbosity as _Verbosity; _verbosity = _Verbosity()
  140 from multi.result_commands import Result_command
  141 from multi.slave_commands import Slave_command
  142 
  143 
  144 #FIXME error checking for if module required not found.
  145 #FIXME module loading code needs to be in a util module.
  146 #FIXME: remove parameters that are not required to load the module (processor_size).
  147 def load_multiprocessor(processor_name, callback, processor_size, verbosity=1):
  148     """Load a multi processor given its name.
  149 
  150     Dynamically load a multi processor, the current algorithm is to search in module multi for a
  151     module called <processor_name>.<Processor_name> (note capitalisation).
  152 
  153 
  154     @todo:  This algorithm needs to be improved to allow users to load processors without altering the relax source code.
  155 
  156     @todo:  Remove non-essential parameters.
  157 
  158     @param processor_name:  Name of the processor module/class to load.
  159     @type processor_name:   str
  160     @keyword verbosity:     The verbosity level at initialisation.  This can be changed during program execution.  A value of 0 suppresses all output.  A value of 1 causes the basic multi-processor information to be printed.  A value of 2 will switch on a number of debugging printouts.  Values greater than 2 currently do nothing, though this might change in the future.
  161     @type verbosity:        int
  162     @return:                A loaded processor object or None to indicate failure.
  163     @rtype:                 multi.processor.Processor instance
  164     """
  165 
  166     # Store the verbosity level.
  167     _verbosity.set(verbosity)
  168 
  169     # The Processor details.
  170     processor_name = processor_name + '_processor'
  171     class_name = processor_name[0].upper() + processor_name[1:]
  172     module_path = '.'.join(('multi', processor_name))
  173 
  174     # Load the module containing the specific processor.
  175     modules = _import_module(module_path)
  176 
  177     # Access the class from within the module.
  178     if hasattr(modules[-1], class_name):
  179         clazz = getattr(modules[-1], class_name)
  180     else:
  181         raise Exception("can't load class %s from module %s" % (class_name, module_path))
  182 
  183     # Instantiate the Processor.
  184     object = clazz(callback=callback, processor_size=processor_size)
  185 
  186     # Load the Processor_box container and store the details and Processor instance.
  187     processor_box = Processor_box()
  188     processor_box.processor = object
  189     processor_box.processor_name = processor_name
  190     processor_box.class_name = class_name
  191 
  192     # Return the Processor instance.
  193     return object
  194 
  195 
  196 def fetch_data(name=None):
  197     """API function for obtaining data from the Processor instance's data store.
  198 
  199     This is for fetching data from the data store of the Processor instance.  If run on the master, then the master's data store will be accessed.  If run on the slave, then the slave's data store will be accessed.
  200 
  201 
  202     @attention:     No inter-processor communications are performed.
  203 
  204     @keyword name:  The name of the data structure to fetch.
  205     @type name:     str
  206     @return:        The value of the associated data structure.
  207     @rtype:         anything
  208     """
  209 
  210     # Load the Processor_box.
  211     processor_box = Processor_box()
  212 
  213     # Forward the call to the processor instance.
  214     return processor_box.processor.fetch_data(name=name)
  215 
  216 
  217 def fetch_data_store():
  218     """API function for obtaining the data store object from the Processor instance.
  219 
  220     If run on the master, then the master's data store will be returned.  If run on the slave, then the slave's data store will be returned.
  221 
  222 
  223     @attention:     No inter-processor communications are performed.
  224 
  225     @return:        The data store of the processor (of the same rank as the calling code).
  226     @rtype:         class instance
  227     """
  228 
  229     # Load the Processor_box.
  230     processor_box = Processor_box()
  231 
  232     # Return the data store.
  233     return processor_box.processor.data_store
  234 
  235 
  236 def send_data_to_slaves(name=None, value=None):
  237     """API function for sending data from the master to all slaves processors.
  238 
  239     @attention:     Inter-processor communications are performed.
  240 
  241     @keyword name:  The name of the data structure to store.
  242     @type name:     str
  243     @keyword value: The data structure.
  244     @type value:    anything
  245     """
  246 
  247     # Load the Processor_box.
  248     processor_box = Processor_box()
  249 
  250     # Forward the call to the processor instance.
  251     processor_box.processor.send_data_to_slaves(name=name, value=value)
  252 
  253 
  254 
  255 class Application_callback(object):
  256     """Call backs provided to the host application by the multi processor framework.
  257 
  258     This class allows for independence from the host class/application.
  259 
  260     @note:  B{The logic behind the design} the callbacks are defined as two attributes
  261             self.init_master and self.handle_exception as handle_exception can be null (which is
  262             used to request the use of the processors default error handling code). Note, however,
  263             that a class with the equivalent methods would also works as python effectively handles
  264             methods as attributes of a class. The signatures for the callback methods are documented
  265             by the default methods default_init_master & default_handle_exception.
  266     """
  267 
  268     def __init__(self, master):
  269         """Initialise the callback interface.
  270 
  271         @param master:  The data for the host application. In the default implementation this is an
  272                         object we call methods on but it could be anything...
  273         @type master:   object
  274         """
  275 
  276         self.master = master
  277         """The host application."""
  278 
  279         self.init_master = self.default_init_master
  280         self.handle_exception = self.default_handle_exception
  281 
  282 
  283     def default_handle_exception(self, processor, exception):
  284         """Handle an exception raised in the processor framework.
  285 
  286         The function is responsible for aborting the processor by calling processor.abort() as its
  287         final act.
  288 
  289         @param processor:   The processor instance.
  290         @type processor:    multi.processor.Processor instance
  291         @param exception:   The exception raised by the processor or slave processor. In the case of
  292                             a slave processor exception this may well be a wrapped exception of type
  293                             multi.processor.Capturing_exception which was raised at the point the
  294                             exception was received on the master processor but contains an enclosed
  295                             exception from a slave.
  296         @type exception:    Exception instance
  297         """
  298 
  299         # Print the traceback.
  300         _traceback.print_exc(file=_sys.stderr)
  301 
  302         # Stop the processor.
  303         processor.abort()
  304 
  305 
  306     def default_init_master(self, processor):
  307         """Start the main loop of the host application.
  308 
  309         @param processor:   The processor instance.
  310         @type processor:    multi.processor.Processor instance
  311         """
  312 
  313         self.master.run()
  314 
  315 
  316 
  317 class Processor_box(object):
  318     """A storage class for the Processor instance and its attributes.
  319 
  320     This singleton contains Processor instances and information about these Processors.  Importantly
  321     this container gives the calling code access to the Processor.
  322     """
  323 
  324     # Class variable for storing the class instance.
  325     instance = None
  326 
  327     def __new__(self, *args, **kargs): 
  328         """Replacement function for implementing the singleton design pattern."""
  329 
  330         # First initialisation.
  331         if self.instance is None:
  332             self.instance = object.__new__(self, *args, **kargs)
  333 
  334         # Already initialised, so return the instance.
  335         return self.instance