"Fossies" - the Fresh Open Source Software Archive

Member "opensaf-5.21.09/src/imm/README" (14 Sep 2021, 164161 Bytes) of package /linux/misc/opensaf-5.21.09.tar.gz:


As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. See also the latest Fossies "Diffs" side-by-side code changes report for "README": 5.21.06_vs_5.21.09.

    1 #
    2 #      -*- OpenSAF  -*-
    3 #
    4 # (C) Copyright 2008 The OpenSAF Foundation
    5 #
    6 # This program is distributed in the hope that it will be useful, but
    7 # WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
    8 # or FITNESS FOR A PARTICULAR PURPOSE. This file and program are licensed
    9 # under the GNU Lesser General Public License Version 2.1, February 1999.
   10 # The complete license can be accessed from the following location:
   11 # http://opensource.org/licenses/lgpl-license.php
   12 # See the Copying file included with the OpenSAF distribution for full
   13 # licensing terms.
   14 #
   15 # Author(s): Ericsson AB
   16 #
   17 
   18 GENERAL
   19 =======
   20 This directory (osaf/services/saf/immsv) contains an implementation of the
   21 SAF IMM service version A.02.01.
   22 
   23 Application programmers intending to interface with immsv should primarily read
   24 the document "OpenSAF IMM Service Release 4 Programmer's Reference" and of
   25 course the IMM standard (SAI-AIS-IMM-A.02.01).
   26 
   27 This document provides an overview of the design and internals of the immsv.
   28 It is intended for maintainers or trouble-shooters of the immsv in OpenSAF.
   29 Familiarity with the IMM standard is obviously also needed for understanding
   30 this document.
   31 
   32 The IMM service follows the three tier OpenSAF "Service Director" framework
   33 (see OpenSAF Overview User's Guide).
   34 As such, the process structure consists of:
   35 
   36         * The IMM Director process type (IMMD).
   37         * The IMM Node Director process type (IMMND).
   38         * The application procesesses linked with the IMM Agent library (IMMA).
   39 
   40 IMMD
   41 ====
   42 The IMMD is implemented as a server process executing on the controllers using
   43 a 2N redundancy model. The IMMD on the active controller can be seen as the
   44 "central master" of the IMM service. The standby IMMD is kept up-to-date by the
   45 active IMMD using the message based checkpointing service. If the active
   46 controller goes down, the standby controller becomes active by order from the
   47 AMF. A crash of any director process, such as the IMMD, will escalate to a
   48 restart of the active controller, which of course implies a fail-over of the
   49 directors for all services.
   50 
   51 The active IMMD tracks the cluster membership (using MDS) and controls
   52 which nodes/IMMNDs currently can provide access to the IMM service.
   53 Note: the current implementation of IMM does not use the SAF CLM service.
   54 That service is only at the node level. The cluster that is tracked by the IMMD
   55 will loose and gain membership not only when nodes leave and join, but also when
   56 IMMND processes crash and restart. The crash of an IMMND process does *not*
   57 imply the restart of the node where it executes.
   58 
   59 The set of IMMNDs known by the IMMD is the basis for the two main functions of
   60 the IMMD:
   61 
   62         * FEVS, a reliable multicast between IMMNDs.
   63         * Election of IMMND coordinator.
   64 
   65 To understand the current implementation of the IMM service, it helps to know
   66 that it was first developed in a different context. That context used Extended
   67 Virtual Synchrony (EVS) as the basic cluster communication protocol.
   68 It had only one server process type implementing the IMM service and it executed
   69 symmetrically, one server process on each node. That implementation has now been
   70 ported to the OpenSAF middle-ware. The single symmetrically distributed process
   71 now corresponds to the IMMND process type. The IMMD provides a replacement for
   72 EVS that we call FEVS.
   73 
   74 FEVS (Fake EVS) is so named because it roughly emulates the semantics (but not
   75 the API) of EVS. That is, FEVS tries to provide a reliable multicast over a set
   76 of members plus control over changes of membership. By "reliable" we mean that
   77 any FEVS message arriving at a member, must arrive at all current members and
   78 in the same order. Thus it provides non-loss and uniform ordering of messages.
   79 In addition, when a member joins or leaves, the other members see that join or
   80 leave at the same point in the FEVS message sequence.
   81 
   82 The other function of the IMMD, election of IMMND coordinator, concerns a minor
   83 asymmetry between the IMMNDs. One of the current IMMND members has to take on
   84 the role of IMMND coordinator. Exactly what this means is explained in the next
   85 section. Here it suffices to say that it does not matter which one of the
   86 current IMMND members takes on this role, only that one and only one of the
   87 IMMNDs has this role at all times. The IMMD will always decide which IMMND is
   88 the current coordinator.
   89 
   90 The IMMD is quite lean, simple and small in its functionality. It is almost not
   91 imm service specific at all. This increases the probability that it is robust
   92 and stays so. This is good given the serious escalation caused by the failure
   93 of any director process, in OpenSAF.
   94 
   95 
   96 IMMND
   97 =====
   98 The IMMND is implemented as a server process executing on all nodes (both
   99 controllers and payloads) according to the "No Redundancy" model. Each IMMND
  100 handles all connections made by clients to the IMM service, at that node.
  101 This off-loads the central IMMD server from handling any client related data.
  102 In fact, an IMMD fail-over will not impact any clients, except the clients at
  103 the active controller itself which has to restart.
  104 
  105 The IMMND process is the "real" IMM server. It contains the IMM data model,
  106 which holds the configuration and runtime objects that is the core of the
  107 IMM service. It also has information about all implementers, admin-owners and
  108 CCBs currently registered. In the current implementation, this repository is
  109 symmetrically replicated over the IMMND at all processors. This will optimise
  110 for read access, since all such accesses can be kept purely node-local.
  111 
  112 Updates on the other hand, must be replicated to all IMMNDs. To ensure that all
  113 IMMNDs are kept consistent and complete, the IMM service uses the FEVS protocol
  114 provided by the IMMD. When a client issues a request that implies a mutation of
  115 the IMM repository, the local IMMND that receives the request does not
  116 immediately apply the request. Instead it FEVS forwards the request to the IMMD.
  117 The IMMD in turn will add a sequence number to the message and then forward the
  118 message to all IMMNDs (including the originating IMMND), using MDS broadcast.
  119 
  120 Since all such messages pass through the single active IMMD process, a total
  121 order is enforced. The sequence number added by the IMMD is only used as a
  122 redundant check for completeness (non-loss). Any IMMND that receives an
  123 out-of-sequence message will not apply that message. Instead it will either
  124 restart and request a "sync"; or it may request a resend from the IMMD of the
  125 missing message(s). [The resend variant is not yet fully implemented.]
  126 
  127 The IMMND processes are in essence symmetrically equivalent. But there are a few
  128 important exceptions. First, the details of connections are maintained only at
  129 the local IMMND. Other IMMNDs will, for example, know which node an attached
  130 implementer resides on, but not the exact connection for a remote implementer.
  131 Any need to reach a remote implementer is solved by communicating with the
  132 IMMND at that node. Second, at all times when the IMM service is operational,
  133 one and only one of the IMMNDs is designated the IMMND-coordinator. The IMMND
  134 that is currently the coordinator will take on the task of conducting a few
  135 cluster global operations when needed:
  136 
  137         * Initial loading;
  138         * Sync of restarted IMMNDs;
  139         * Aborting CCBs waiting on apparently hung implementors.
  140 
  141 A very good question here is: why not move the IMMND coordinator role to the
  142 active IMMD? This is indeed possible to do, but there are at least a few
  143 arguments against it.
  144 
  145 First, these three tasks depend on knowledge of the current state of CCBs.
  146 At the start of a sync, there is a period of grace where the IMMSv will not
  147 accept new CCBs, but will allow already started CCBs to complete. A CCB can not
  148 be aborted if it is in the critical phase of distributed commit. Loading also
  149 has to be coordinated with sync, because the loading is often followed
  150 immediately by a sync of any straggler nodes that missed the start of the
  151 loading sequence. The dependency on CCB state could be solved by involving the
  152 IMMD in the CCB life cycle. But since all IMMNDs are already involved in the CCB
  153 handling and sync, there is no real simplification in also involving the IMMD.
  154 The only way to really simplify would be to move the entire repository to the
  155 IMMD, converting to the "Service Server" framework, or possibly keeping the
  156 IMMNDs but only for client connection management. Indeed, we may end up there in
  157 the future.
  158 
  159 Second, there is a strong argument for keeping the IMMDs simple and by
  160 implication robust. A crash of an IMMD causes a controller fail-over, but a
  161 crash of an IMMND, even on the active controller, only causes a restart of the
  162 IMMND.
  163 
  164 Finally, another reason why the IMMND coordinator role does not reside with the
  165 active IMMD is historical, that it was ported from an EVS based implementation.
  166 
  167 The coordinator IMMND can theoretically be any IMMND in the system. In the
  168 current implementation we have restricted it to reside on a controller node.
  169 The only reason for this restriction is that the task of loading requires
  170 access to the file system and we do not wish to assume that payloads have such
  171 access. In the future we may enhance the reliability of the IMM service by
  172 allowing any IMMND to become coordinator once the IMMNDs have loaded.
  173 
  174 The current limitation, that the IMMND coordinator must reside on a controller,
  175 has a downside.  If the IMMNDs at both controllers fail, the IMMD can not elect
  176 any coordinator. Instead the IMMD has to order all IMMNDs to restart, which
  177 currently causes a cluster restart.
  178 
  179 
  180 IMMA
  181 ====
  182 
  183 The client interface conforms to the OpenSAF agent library framework. The IMMA
  184 library (libSaImmOm.so and libSaImmOi.so) only communicate with the node local
  185 IMMND process. As explained previously, all read accesses made by the client
  186 are kept node-local. In addition, the values for basic immsv handles:
  187 
  188         SaImmHandleT
  189         SaImmSearchHanldeT
  190         SaImmAccessorHandleT
  191         SaImmOiHandleT
  192 
  193 are all set by the local IMMND without going remote to the IMMD or other IMMNDs.
  194 The following handles are related to resource allocation, need global validity
  195 and therefore are initialised over FEVS.
  196 
  197         SaImmAdminOwnerHandleT (linked to SaImmHandleT)
  198         SaImmCcbHandleT        (linked to SaImmAdminOwnerHandleT & SaImmHandleT)
  199         SaImmOiImplemenerNameT (linked to SaImmOiHandleT).
  200 
  201 Because the basic handles are allocated and known only by the local IMMND, a
  202 crash or termination of that IMMND will cause the IMMA library instantiations
  203 at the node to mark all its handles as stale. Even global handles allocated via
  204 the crashed IMMND are effectively stale because they depend on the local handles.
  205 
  206 The restart of an IMMND should still normally be transparent to the clients
  207 at that node. The IMMA library tries to resurrect the handles once the IMMND
  208 at the node has restarted. In some cases the IMMA will not be able to
  209 resurrect the handle. One example is if the handle is related to an admin-owner
  210 handle with the 'release-on-finalize' option set ot true.
  211 Some clients may then get their handles invalidated. Any attempt by a client
  212 to use such a stale and unsuccessfully resurected handle will result in a
  213 reply of SA_AIS_ERR_BAD_HANDLE.
  214 
  215 Note that it is at least not necessary for the client applications to restart.
  216 They may reconnect by simply allocating a new handle. Of course, CCBs that
  217 originated at the node and that where not applied, are aborted. Implementers
  218 need to re-attach. Admin-ownership needs to be reclaimed etc, if you have
  219 allocated a new handle to replace a stale one.
  220 
  221 From the perspective of the other nodes, the termination of a remote IMMND is
  222 handled in the same way as if the node it executed on terminated, i.e. the
  223 resources allocated by any client at that node are released. Note however that
  224 such a release of resources by the other IMMNDs is not done immediately or
  225 autonomously by each node. Instead a release order is sent over FEVS.
  226 The actual release is done when the FEVS message arrives at the IMMNDs,
  227 ensuring that all IMMNDs release the resource at the same point in the FEVS
  228 message sequence.
  229 
  230 Common missunderstandings about the OpenSAF handles and IMM handles.
  231 --------------------------------------------------------------------
  232 (i) Handles are resources.
  233 OpenSAF client handles in general and IMM handles in particular(SaImmHandleT
  234 and SaImmOiHandleT) are not thread independent. That is, the developer must
  235 avoid using the same handle concurrently from several application threads.
  236 Using different handles from different application threads is of course allowed
  237 and using one handle in different application threads at *different* times
  238 is allowed. In the latter case, the threads typically need to synchronize their
  239 use of the common handle to avoid overlapped use. And they need to understand
  240 the meaning of the term 'use' for the handle. An application thread that is
  241 blocked in a poll that uses a selection object obtained from a handle,
  242 is using that handle even when that thread is not executing.
  243 
  244 Each handle should be seen as a resource, which mutates its state
  245 when used by a thread. Part of the resource is a communications channel
  246 to/from the IMMND server. If more than one thread tries to send (synchronously
  247 or asynchronously) or receive (includes waiting for synchronous reply, poll
  248 and dispatch), using the same handle at the same time, the results are
  249 unpredictable. Messages may get lost and a reply may end up at the "wrong"
  250 receiver, the process may crash on assertion in the library.
  251 Each handle is also associated with some heap memory containing a handle
  252 state.
  253 
  254 Interleaved use of a handle between several threads is thus possible,
  255 if done correctly. But we strongly recommend against it because any
  256 incorrect use will cause such strange and unpredictable symptoms.
  257 
  258 Note that, used correctly(!) OpenSAF libraries are (should be) thread safe.
  259 But thread safe does not mean immune to any and all forms of incorrect use.
  260 
  261 (ii) Sub-handles depend on the root handle resources.
  262 An OM-handle (SaImmHandleT) can be used to generate sub-handles such as:
  263 
  264   SaImmAdminOwnerHandleT
  265   SaImmCcbHandleT
  266   SaImmSearchHandleT
  267   SaimmAccessorHandleT
  268 
  269 Such sub-handles will share the *same* communication channel towards the
  270 IMMND server. Thus sub-handles to one SaImmHandleT should only be used by
  271 the thread currently using (owning) the root SaImmHandleT.
  272 
  273 (iii) Asynchronous admin-operations.
  274 A special case of (ii) is how to program the use of asynchronous admin-
  275 operations. The "vanilla" case is to simply use one thread, both for sending
  276 the asynchronous request and for receiving the asynchronous reply. Both events
  277 of course use the same handle. This is the recommended use if possible.
  278 It is possible when the thread is event driven from a poll. External
  279 requests arrive causing the thread to sometimes send asynchronous admin-
  280 operation requests to some OI. The asynchronous reply from the OI is picked
  281 up by the same poll and typically forwarded out to the external user.
  282 Some form of continuation record would be kept by the requesting thread
  283 to identify the reply.
  284 
  285 But in some cases the developer wants to receive the replies on
  286 asynchronous operations in a separate thread from the thread that sent
  287 the request. This is possible, provided that the sending thread somehow
  288 submitts the handle for exclusive use by the receiving thread, immediately
  289 after the send. Until the receiving thread has has received the reply,
  290 the original sending thread should not use the same handle, and it must
  291 not use the same handle for receiving anything (i.e. not poll on a selection
  292 object obtained from that handle and not invoke any synchronous calls using
  293 that handle).
  294 
  295 (iv) Blocking & timeouts.
  296 Synchronous downcalls will block the client untill there is a reply.
  297 If there is any resource problem detected in the IMMA library or
  298 in the IMMND server that prevents the request from being processed
  299 currently, then an immediate reply with SA_AIS_ERR_TRY_AGAIN is generated.
  300 The client must be prepared to get the TRY_AGAIN error on any downcall.
  301 The apropriate way to handle TRY_AGAIN is to have a limited loop with
  302 a small delay (at least 10 ms) for each iteration. Preferrably also
  303 increase the delay for each iteration (back off). The client must also
  304 consider the realtime requirements placed on the client itself.
  305 When time spent in the retry loop approaches the realtime limit
  306 placed on the client, then it has to abandon the request.
  307 
  308 A synchronous call may also get blocked for a longer time (without getting
  309 TRY_AGAIN) due to resource congestion in the system, or due to slow
  310 processing by one or more OIs, or due a large request such as the apply
  311 of a large CCB. There is a timeout placed on each synchronous downcall,
  312 inside the IMMA library. The default timeout is 10 seconds. This timeout
  313 can be altererd by setting the IMMA_SYNCR_TIMEOUT environment variable
  314 (see below).
  315 
  316 There is also a timeout in the IMMND server placed on each OI callback that
  317 is generated. This timeout is currently hard-coded to 6 seconds.
  318 There is an enhancement planned to allow this global timeout value
  319 to be configured (http://devel.opensaf.org/ticket/2021).
  320 If the OI does not reply in time, the server will reply with error to the
  321 client, without waiting for the reply on the callback. Exactly which error is
  322 generated depends on the type of call. Ccb related calls tend to generate
  323 FAILED_OPERATION, i.e. an abort of the CCB/transaction by the server.
  324 But with PBE enabled and congestion at the PBE, the server can not abort
  325 any CCB that is being committed by the PBE. So there is the unfortunate
  326 possibility of getting TIMEOUT on a ccb-apply when applying a CCB with
  327 PBE enabled. TIMEOUT of course means that the client then does not
  328 know if the CCB committed, aborted, or is still being processed.
  329 
  330 (v) Return codes from OiCcbCallbacks.
  331 An OI returning a specific error code from a ccb-callback will often
  332 not result in that exact error code being transmitted to and returned by
  333 the originating OM interface. The OI is typically just a participant in
  334 the distributed thing (in space and time) that a CCB/transaction is. Thus
  335 the IMM server consolidates replies from several Ois and must also provide
  336 clear and unambiguous communication of the CCB state towards the CCB end user.
  337 This inevitably results in return codes from Ois being converted by the imm
  338 server to a different singe error code returned to the om-user. The cardinal
  339 example is the ERR_FAILED_OPERATION meaning CCB aborted, resulting from one
  340 or more Ois returning a different specific error on a callback. If the OIs
  341 return error-strings allong with the error code then these error strings
  342 will be provided over the om interface. The OI implementer should therefore
  343 avoid including a specific error code in any error string returned from
  344 a ccb-oi-callback.
  345 
  346 Common missunderstandings about attribute defaults.
  347 ---------------------------------------------------
  348 Imm class definitions allow the declaration of a default value to be defined
  349 as part of an attribute definition.
  350 
  351 (i) A default declaration is only allowed for single valued attributes (no
  352 concept of a multivalued default exists).
  353 
  354 (ii) Default values are assigned at object creation, except the creation comes
  355 from IMM loader.
  356 
  357 Common missunderstandings about "system attributes" of an imm object.
  358 ---------------------------------------------------------------------
  359 Besides the attributes defined in its class, every object also gets assigned
  360 three system attributes (see section 4.2.19 in the SAF IMM A.2.1 spec):
  361 
  362     SaImmAttrClassName - SaStringT - holds the class-name for the object.
  363     SaImmAttrAdminOwnerName - SaStringT - holds admin-owner (if it exists).
  364     SaImmAttrImplementerName - SaStringT - holds implemener-name (if it exists).
  365 
  366 These system attributes are maintained by the imm service and are not writable
  367 by any Om or OI client. They are readable at any time. The SaImmAttrClassName
  368 attribute is in essence static, since no object can ever change class. It is
  369 also a persistent attribute in that the value is dumped to external format
  370 and the value of course survives cluster restarts.
  371 
  372 The SaImmAttrAdminOwnerName and SaImmAttrImplementerName get assigned when/if 
  373 an admin-owner or implementer is set for the object. These attributes are not
  374 full-cycle persistent. That is, a cluster restart will drop their value. For
  375 the SaImmAttrAdminOwnerName attribute, imm-loading always assigns the value
  376 'IMMLOADER'. That admin-owner is special in that it may be overwritten by
  377 anyone setting a new and normal admin-owner for the object. The admin-owner
  378 value 'IMMLOADER' can be used to see if an object has been modified since 
  379 the last cluster restart. For SaImmAttrImplementerName the value will always
  380 be empty/null after a cluster restart.
  381 
  382 Note however that both SaImmAttrImplementerName and SaImmAttrAdminOwnerName
  383 are written to export format when the imm is dumped. This can be usefull
  384 for debugging purposes, but the immloader discards these values so that
  385 the system comes up "clean" with respect to admin-owners and implementers.
  386 
  387 One conclusion from this is that an OI that is started/restarted must always
  388 invoke not just saImmOiImplementerSet, but also any relevant
  389 saImmOiClassImplementerSet or saImmOiObjectImplementerSet. If the OI only
  390 restarted as a process, the call will be 'idempotent' and succeed quickly.
  391 If the OI is started after a cluster-start then these calls are necssary
  392 to rebuild the implementer-map. See ticket defect ticket #543 for details.
  393 (http://sourceforge.net/p/opensaf/tickets/543)
  394 
  395 Minor discrepancy between OpenSAF and SAF on error codes for admin-operations.
  396 ------------------------------------------------------------------------------
  397 According to the SAF spec ERR_BUSY should be given as response to an
  398 AdminOwnerRelease/Clear operation if there is currently an admin-operation 
  399 on-going directed at some object that the release/clear is targeting.
  400 But The AdminOwner mechanism is an access-control mechanism, not a concurrency
  401 control mechanism. After access has been granted, in this case for an 
  402 admin-operation, there is no real point in insisting that the admin-ownership
  403 must be valid until the admin-op completes. An AdminOwnerRelease/Clear will
  404 return ERR_BUSY if there is an ongoing CCB, since it is unknown if additional
  405 operations are to be added. Opeartions may be added to the CCB by the OI
  406 itself using the AugmentedCcb interface. With respect to interference
  407 from other overlapping admin-operation requests or other ccb-requests, the
  408 OpenSAF IMM leaves it open to the OI implementation to handle or reject.
  409 The OI dispatch is always via one single thread, but the OI may internally be
  410 multithreaded. It may also postpone reply on an admin-op while taking on other
  411 queued callbacks, since the reply to an admin-op is not done by returning from
  412 the admin-op callback, but by a separate downcall to immsv.
  413 
  414 
  415 
  416 PERSISTENCE
  417 ===========
  418 The IMM standard mandates that configuration attributes and persistent runtime
  419 attributes shall be persistent in the sense that such values must survive a
  420 cluster restart and that this must hold for every applied CCB.
  421 
  422 The default option for the current implementation of the IMMSv in OpenSAF is
  423 that it does not fully support persistence at the CCB granularity. Instead
  424 it supports persistence at the coarse granularity of dump/backup. Thus the
  425 user must ensure that the IMM is dumped after important changes are made to
  426 such data. The configuration data and persistent runtime data will survive
  427 cluster restart, but only the latest dumped version will be recovered.
  428 
  429 One of the main new immsv features available in OpenSAF4.0 is the
  430 Persistent Back-End (PBE). By default this feature is not enabled.
  431 See the instructions below for how to enable it. Use of the PBE
  432 has a signifigant (negative) impact on performance for CCBs and
  433 persistent runtime data, both response time and throughput.
  434 The upside is that the user does not need to do frequent dumps
  435 to get persistence. But it is still advisable to dump the IMM
  436 (to the imm-xml form) periodically, for backup to an external
  437 filesystem, to support recovery from disasters.
  438 
  439 -----------------------------------------------
  440 
  441 The IMM-PBE feature is an optional feature.
  442 To enable this feature the folowing three steps are needed.
  443 
  444 	1) ++++++BUILD WITH PBE+++++++
  445 
  446 	In the configure step for OpenSAF enable the pbe:
  447 
  448 	./configure --enable-imm-pbe
  449 
  450 	You would typically also give other options to configure at the same time.
  451 
  452 	Ensure that OpenSAF builds without problems. In particular that it finds
  453 	the sqlite3 include files and the sqlite3 library, which is used by PBE.
  454 
  455 	2) ++++++CONFIGURE PBE DEPLOYMENT+++++++++
  456 
  457 	Edit 'osaf/services/saf/immsv/config/immnd.conf' or in the
  458 	deployed system typically '/etc/opensaf/immnd.conf'.
  459 
  460 	Ensure the IMMSV_ROOT_DIRECTORY points to a shared file system,
  461 	the sqlite3 database file has to be accessible from both controllers!
  462 	In deployment the IMMSV_ROOT_DIRECTORY would typically be set to
  463 	'/var/lib/opensaf/immsv_store'. In the UML simulated deployment,
  464 	this is normally linked symbolically to 'hostfs/repl-opensaf/immsv_store'.
  465 
  466 	Define IMMSV_PBE_FILE to the file name of the sqlite3 database file,
  467 	normally 'imm.db'. Note that IMMSV_PBE_FILE must be defined in immnd.conf
  468 	for *all* nodes in the cluster, including payloads, even if they cannot
  469 	access 	the file. The fact that this environment variable is nonempty
  470 	is what tells all nodes that PBE is configured.
  471 
  472 	Note also that IMMSV_LOAD_FILE must still also be defined. Even with PBE
  473 	enabled, you typically initial start from an imm.xml file. A restore from
  474 	backup will also typically use an imm.xml file.
  475 
  476 	Optionally define IMMSV_PBE_TMP_DIR (see immnd.conf). The PBE sometimes
  477 	needs to regenerate the imm.db file. This is much faster (10 times) if
  478 	the file is generated in a non replicated file system and then copied
  479 	to the replicated file system.
  480 
  481 	Initial start the system.
  482 	Ensure that it comes up (that the immsv loads properly from imm.xml).
  483 
  484 	3) +++++ENABLE PBE++++++++
  485 
  486 	On one of the nodes execute:
  487 
  488 	immcfg -m -a saImmRepositoryInit=1 \
  489 	   safRdn=immManagement,safApp=safImmService
  490 
  491 	This sets the saImmRepositoryInit attribute in the IMM service
  492 	object to SA_IMM_KEP_RESPOSITORY (1).
  493 
  494 	If this immcfg command succeeds, the pbe is runtime enabled and the
  495 	pbe-daemon process should have started and it should have generated
  496 	the first dump to the pbe-file.
  497 	A 'ps' on the controller where the IMMND coord currently resides
  498 	should show something like:
  499 
  500         618 root 22320 S N  /usr/local/bin/immpbed --daemon --pbe /var/lib/..
  501 
  502 The PBE feature can at any time be disabled by setting the saImmRepositoryInit
  503 attribute back to SA_IMM_FROM_FILE (2).
  504 
  505 	immcfg -m -a saImmRepositoryInit=2 \
  506 	   safRdn=immManagement,safApp=safImmService
  507 
  508 The final transaction/CCB done towards the imm.db sqlite file will be to
  509 set the saImmRepositoryInit attribute also in this sqlite file. So any
  510 cluster restart that tries to load from the imm.db file will detect that
  511 PBE is disabled and instead load from the imm.xml file.
  512 
  513 Finally, if the PBE is hung for some reason, it can also be forced to
  514 disabled, by the administrative operation:
  515 
  516 	immadm -o 100 safRdn=immManagement,safApp=safImmService
  517 
  518 This admin-operation will change the value of the above configuration
  519 attribute to 2, i.e. SaImmRepositoryInitModeT value SA_IMM_INIT_FROM_FILE,
  520 without using a ccb and ignoring any currently active PBE.
  521 But this way of disabling the PBE will NOT update the saImmRepositoryInit
  522 attribute in the sqlite file. So any subsequent cluster restart will use
  523 the imm.db file and come up with PBE enabled. So if possible, use the
  524 recommended way, via immcfg, for disabling PBE.
  525 
  526 
  527 ---------------------
  528 
  529 SCHEMA CHANGES
  530 ==============
  531 OpenSAF 4.1 adds the immsv enhancement of supporting schema changes, that is
  532 changes at runtime to existing class definitions. For more background see
  533 ticket #1310: http://devel.opensaf.org/ticket/1310
  534 
  535 The new class definition must of course be valid in itself, i.e. it must
  536 pass all tests that a first version of a class has to pass.
  537 On top of this there are many restrictions that apply to a schema change.
  538 
  539 The kinds of change allowed in OpenSAF 4.1 are limited to addition of
  540 new attributes and changes to existing attributes. Removal of existing
  541 attributes are not allowed. Furthermore, not all kinds of additions or
  542 changes are allowed. In general, relaxation changes are allowed while
  543 restricting changes are not.
  544 
  545 The following changes are allowed:
  546 
  547 ADDITION of new attributes when the 'initialized' flag is not set.
  548 
  549 CHANGE of an existing attribute if it is one of the following:
  550 
  551    * Change of default-value for an attribute (config or runtime).
  552      New instances of the class will get the new default,
  553      old instances retain the old value.
  554 
  555    * Adding default-value for an attribute (config or runtime).
  556      New instances of the class will get the new default,
  557      old instances retain the null value (untill the next cluster restart).
  558 
  559    * Adding 'multivalued' flag for an attribute (config or runtime).
  560 
  561    * Adding 'writable' flag for an attribute (config).
  562 
  563    * Removing 'initialized' flag for an attribute (config).
  564 
  565    * Removing 'persistent' flag for an attribute (runtime).
  566 
  567 
  568 All other changes, such as change of attribute type, removal of default,
  569 are not allowed.
  570 
  571 There is no API change. The existing saImmOmClassCreate_2 call is used
  572 to insert the new class version. The standard behavior is to reject
  573 a class create call with ERR_EXIST if the class name matches an existing class.
  574 This is also still the defalt behavior. But if schema change is enabled,
  575 (explained below), then a new definition for an existing class is
  576 interpreted as an attempt to change that existing definintion.
  577 The implied change is analyzed to ensure that it is an allowed change
  578 according to the above rules. If it is allowed, the new class definition
  579 replaces the old and all existing instances are migrated to the new
  580 representation. If PBE is enabled, the entire change is made persistent.
  581 
  582 To enable the non standard schema change support, the following admin
  583 operation must be executed:
  584 
  585         immadm -o 1 -p opensafImmNostdFlags:SA_UINT32_T:1 \
  586            opensafImm=opensafImm,safApp=safImmService
  587 
  588 This will set the lowest order bit of the 'opensafImmNostdFlags' runtime
  589 attribute inside the immsv. Operation-id '1' invoked on the object
  590 'opensafImm=opensafImm,safApp=safImmService' has the meaning of 'flags-ON'.
  591 The lowest order flag is the allow-schema-change flag which controls the
  592 schema-change behavior. With this flag switched on, the immsv accepts
  593 legal schema changes. If this flag is off, the immsv follows the standard
  594 behavior of rejecting all implied schema changes with ERR_EXIST.
  595 
  596 The flag is switched off by a corresponding admin operation. Operation-id '2'
  597 invoked on the object 'opensafImm=opensafImm,safApp=safImmService' has the
  598 meaning of 'flags-OFF'.
  599 
  600         immadm -o 2 -p opensafImmNostdFlags:SA_UINT32_T:1 \
  601            opensafImm=opensafImm,safApp=safImmService
  602 
  603 We strongly recommend that the flag is switched off as soon as any intended
  604 schema change has been completed. This to avoid legal but unintended (!)
  605 schema changes. For example, two applications may accidentally use the
  606 same class name. If the first application is already installed when the
  607 second application arrives and the second application redefines the class
  608 in such a way that it is valid when interpreted as an implied upgrade,
  609 then the class could be upgraded. This could have many unintended
  610 consequences which also could be very hard to troubleshoot. The example
  611 also illustrates the importance of propper prefixing of classnames,
  612 since the class name-space is flat (as opposed to the naming tree
  613 for objects).
  614 
  615 If PBE is enabled, the current value of opensafImmNostdFlags can be inspected
  616 via the cached non-persistent runtime attribute 'opensafImmNostdFlags' in the
  617 object  'opensafImm=opensafImm,safApp=safImmService'.
  618 If PBE is not enabled, then this runtime attribute is not accessible, because
  619 there is no regular implementer attached. But the above admin operations are
  620 still handled by the immsv and the schema change behavior is still controlled
  621 the same way.
  622 
  623 MULTIPLE APPLIERS (4.2)
  624 ======================
  625 http://devel.opensaf.org/ticket/1827
  626 OpenSAF 4.2 adds the immsv enhancement of supporting multiple appliers for
  627 configuration objects. An applier is an OI (object implementer) of lower
  628 rank than the primary OI. The primary OI validates the CCB and has veto
  629 rights to abort any CCB it participates in. If the CCB commits then the
  630 pimary OI will apply the changes into the service/application it represents.
  631 If the CCB aborts then the primary OI discards the changes.
  632 
  633 The most signifficant deficiency of having only a primary OI per class/object
  634 is that it is not simple to implement hot-standby solutions.
  635 A hot-standby solution requires the primary OI to inform the standby, by
  636 implementing some form of message protocol at the service/application level.
  637 The concept of an "applier OI" removes this deficiency and allows the
  638 service/application to use the imm to disseminate the changes.
  639 
  640 An applier OI registers the same way as the main OI and receives exactly
  641 the same CCB related callbacks as the primary OI. The difference is that an
  642 applier OI does not participate in CCB validation. The imm-service ignores
  643 any reply on the create,delete,modify and completed callbacks, from applier
  644 OIs. An applier OI does not get callbacks for updating runtime attributes.
  645 This should be obvious since admin-ops or updates to runtime attributes
  646 should only be handled by the primary OI. An applier OI is not allowed to
  647 create or delete runtime objects, or to update cached runtime attributes.
  648 
  649 The completed callback really has no meaning for the applier interface.
  650 For regular Ois, the completed callback has the purpose of allowing the OI
  651 to validate the changes it received for the CCB. But appliers have no validation
  652 rights. Any error code Returned by an applier from a completed callback is
  653 discarded by the imm service. Note also that the immsv does not wait for
  654 replies from the completed callback on appliers, which means that the CCB is
  655 usually already applied in the imm ram database by the time an applier receives
  656 and processes the completed callback. Thus an applier can not use the completed
  657 callback as a trigger to read the pre-apply version of the objects involved in
  658 the CCB. To acheive that requires the primary OI to communicate with the applier
  659 Oi through some alternate mechanism, before the primary OI returns from the
  660 completed callback. 
  661 
  662 The applier OI is thus informed of the same changes as the primary OI
  663 and will be able to apply the changes if the CCB commits and discard them
  664 if the CCB aborts. Exaclty what "apply" here means is up to the application.
  665 The main point here is that it means that the applier has the tools to be
  666 a hot standby, or to perform whatever needs to be done in order to effect
  667 the configuration change that was applied.
  668 
  669 There may be any number of appliers for any given class/object. Thus the
  670 applier OI concept may be used not just to solve the "hot standby" problem
  671 but also for the more general problem of effecting the apply of a CCB over
  672 the applications components in the cluster. In the AMF example above, the
  673 osafamfnd processes (node director processes) could in principle also
  674 attach as applier OIs and register any configuration changes that apply
  675 to the specific node.
  676 
  677 Another way of stating this is that the applier-OI concept allows IMM
  678 users to leverage the IMM CCB protocol for more than the primary purpose of
  679 that protocol.
  680 
  681 The A.02 API is used but the applier concept is only available to an OI
  682 that registers with version A.02.11 (or higher minor version).
  683 An OI-name is recognized as the name for an applier OI by the imm-service
  684 if the first character in the name is '@'. For example, the AMF OI is
  685 named 'safAmfService' and implemented by the osafamfd process at the
  686 active SC. The osafamfd at the standby SC could become more of a hot
  687 standby by registering as the applier OI '@safAmfService'.
  688 
  689 As always, only one process/OI-handle may at any point in time be
  690 associated with a particular OI-name. This holds also for applier-OI
  691 names. And as always, an OI-handle may be associated with at most one
  692 implementer/applier-name at a time. So in case of a switch-over (si-swap),
  693 the osafamfd processes would close their current OI-handles (or invoke
  694 saImmOiImplementerClear). Then the osafamfd processes would attach to
  695 either the main OI (safAmfService) or the applier OI (@safAmfService),
  696 using saImmOiImplementerSet.
  697 
  698 A word of caution about implementer names. Implementer names are to
  699 be re-used. They are never garbage collected by the imm service.
  700 An implementer-name may have no current live implementer handle attached,
  701 but once created, it lives "forever" ready for the next attach,
  702 (saImmOiImplementerSet). Implementer-names could be compared to Unix port
  703 numbers. A port number on some particular host is (in general)
  704 statically mapped to a service on that host. The service may not always
  705 be up and running, but there should never be any *other* service bound
  706 to the port number. Similarly, an implementer-name is in a given cluster
  707 generally mapped to a service. A service should avoid spawning "dynamic"
  708 implementer names, and avoid incorporating volatile name spaces such as
  709 process-id or time in the name. This may have been obvious for regular
  710 OIs. But with the addition of multiple appliers, there is a risk is that
  711 this is interpreted as an open-ended set of appliers, consuming an open
  712 ended ammount of resources. This would be incorrect design.
  713 An application/service must design for a static set of OIs implying a
  714 static set of OI names, including a static set of applier OIs and
  715 a static set of applier names.
  716 
  717 In OpenSAF 5.17.11, a new feature for removing disconnected applies is added
  718 to IMM. The feature can be enabled to remove disconnected appliers from the
  719 cluster after applier timeout expires.
  720 
  721 Another difference between regular regular implementers and appliers,
  722 is that regular implementer names and their binding to a set of objects
  723 will survive a cluster restart, whereas applier names and their binding
  724 to objects does not survive a cluster restart. That is, a regular
  725 implementer name is persistent in the face of a cluster restart, as
  726 long as it has been bound to at least one object. But an applier name
  727 is not persistent in the face of a cluster restart, regardless of
  728 whether it has been bound to anything. The main reason for this is
  729 that the regular implementer name is stored with each object in the
  730 special single valued service attribute: saImmAttrImplementerName.
  731 
  732 Note here that the binding of a regular implementer-name to a class
  733 (as a result of saImmOiClassImplementerSet) does not survive a
  734 cluster restart. So an application that intends to attach as a class
  735 implementer should always exlplicitly set the class implementer.
  736 If the class implementer is already set, this operation returns
  737 SA_AIS_OK anyway. The "feature" of attachign to an already set
  738 up implementer structure is of dubious value, since the application
  739 must then be aware of its starting context.
  740 
  741 Applier bindings are even weaker in this "starting context" sense
  742 because applier bindings to object/class are not synced. The only
  743 applier data that is synced (after a node restart) is the existence of
  744 the applier name and whether ot not any client is currently attached as
  745 that applier.
  746 
  747 The most efficient, simplest and normal way to bind an implementer
  748 or applier to objects is by using the saImmOiClassImplementerSet call.
  749 This ensures that the implementer-name is always bound to all instances
  750 of that class. It ensures that the OI will get the saImmOiObjectCreate
  751 callback, which is only possible for class-implementers, since you can
  752 not register with saImmOiObjectImplementerSet on an object that does
  753 not yet exist.
  754 
  755 The use of saImmOiObjectImplementerSet is mainly for cases when there
  756 are very few instances of a class and each instance is to be linked
  757 to a an implementer that behaves in a way specific to that instance.
  758 
  759 For appliers, attaching as an object-applier could be usefull as a
  760 way of "eavesdropping" on the changes affecting a particular object,
  761 or a few select objects. But invoking saImmObjectImplementerSet
  762 with an applier name, for a large subtree of objects will be much
  763 more expensive in memory than just registering for the class(es)
  764 of these objects.
  765 
  766 
  767 SUPPORT FOR CCB-OI ERROR STRINGS (4.2)
  768 =====================================
  769 http://devel.opensaf.org/ticket/1904
  770 OpenSAF 4.2 adds the immsv enhancement of supporting ccb error strings
  771 from OI to OM client. Two new API calls are available for this for
  772 clients that register with imm version A.2.11:
  773 
  774 (1)
  775 extern SaAisErrorT
  776 saImmOiCcbSetErrorString(SaImmOiHandleT immOiHandle,  /* in */
  777                          SaImmOiCcbIdT ccbId,         /* in */
  778                          const SaStringT errorString);/* in */
  779 
  780   Arguments     :  immOiHandle - IMM OI handle
  781                    ccbId  -  The ccbId for the ccb related callback.
  782                    errorString - The errorString.
  783 
  784   Return Values :  SA_AIS_OK
  785                    SA_AIS_ERR_BAD_HANDLE
  786                    SA_AIS_ERR_INVALID_PARAM
  787                    SA_AIS_ERR_BAD_OPERATION (not inside valid ccb callback)
  788                    SA_AIS_ERR_LIBRARY
  789                    SA_AIS_ERR_TRY_AGAIN
  790                    SA_AIS_ERR_VERSION  (not using A.2.11)
  791 
  792 Allows an OI (not applier) to set an error string in the context of a ccb
  793 related callback. Setting an error string is only relevant if the OI will
  794 return with error (not SA_AIS_OK) on the callback.  The error
  795 string will be transmitted to the OM client. Only one error string
  796 will be accepted per OI callback. Setting additional error strings
  797 in the same OI callback will simply replace the previous one.
  798 Multiple error strings may however be set from different OIs
  799 in different callbacks related to the same OM downcall (e.g.
  800 saImmOmCcbApply). The error SA_AIS_ERR_BAD_OPERATION is returned if the
  801 call is not made from insdie the dynamic context of a valid OI callback.
  802 Setting an error string is only possible inside ccb-oi-callbacks that
  803 have a return code. Thus it is NOT possible inside SaImmOiCcbApplyCallbackT
  804 or SaImmOiCcbAbortCallbackT. For applier OIs, all return values on
  805 callbacks are ignored and an applier OI will get BAD_OPERATION on any
  806 attempt to use this function.
  807 
  808 (2)
  809 extern SaAisErrorT
  810 saImmOmCcbGetErrorStrings(
  811         SaImmCcbHandleT ccbHandle,      /* in */
  812         const SaStringT **errorStrings);/* out */
  813 
  814   Arguments     :  ccbHandle    - The ccb handle.
  815                    errorStrings - Pointer to a pointer to a NULL terminated array
  816                                   of SaStringT. The list of strings is owned by
  817                                   the imma om library and is cleared/deallocated
  818                                   in conjunction with the next downcall using the
  819                                   ccb-handle.
  820 
  821   Return Values :  SA_AIS_OK
  822                    SA_AIS_ERR_BAD_HANDLE
  823                    SA_AIS_ERR_INVALID_PARAM
  824                    SA_AIS_ERR_LIBRARY
  825                    SA_AIS_ERR_TRY_AGAIN
  826                    SA_AIS_ERR_VERSION  (not using A.2.11)
  827 
  828 Allows the OM client to fetch error strings after the return of a
  829 ccb-related downcall and *before* making the next ccb-related
  830 downcall using the same ccb-handle. If any error string where
  831 set by any OIs involved in the operation that returned with
  832 error, then these error strings should be included in the list.
  833 The imm server discards duplicate strings and allows a maximum
  834 of 128 error strings in the list. Any ccb call that returns
  835 SA_AIS_OK will not have any error-strings.
  836 Only errors that could be returned, directly or indirectly
  837 by an OI, such as FAILED_OPERATION or BAD_OPERATION should
  838 be expected to possibly have an error string.
  839 
  840 NOTE: It is very important that the OM client completes its
  841 processing of the errorStrings before invoking the next
  842 ccb downcall using the same ccbHandle. The next such downcall
  843 will free the error strings. If the om user needs the error
  844 strings after re-using the ccbHandle, the user must copy the
  845 strings before such re-use. Note also that the 'errorStrings'
  846 out parameter is declared as 'const', indicating that deallocation
  847 of 'errorStrings' is here done by the library (not the user).
  848 
  849 
  850 IMPROVED ADMIN-OPERATION API (4.2)
  851 ====================================
  852 http://devel.opensaf.org/ticket/1764
  853 Allows the OI to supply a return paramater list in the same way
  854 as the OM client has always been able to supply a parameter list
  855 as part of an admin-operation-invocation. The reply from the OI
  856 is in other respects identical to A.02.01.
  857 Api additions for this are available to imm clients that register
  858 with imm version A.2.11.
  859 
  860 Setting version A.2.11 also means that operationId values higher
  861 than SA_IMM_PARAM_ADMOP_ID_ESC (0x8000000000000000LL) have
  862 special meaning and can not be used by the application for its
  863 own purposes. See point 7 below.
  864 
  865 
  866 (1)
  867 extern SaAisErrorT
  868 saImmOiAdminOperationResult_o2(
  869         SaImmOiHandleT immOiHandle,                        /* in */
  870         SaInvocationT invocation,                          /* in */
  871         SaAisErrorT result,                                /* in */
  872         const SaImmAdminOperationParamsT_2 **returnParams);/* in */
  873 
  874   Arguments     :  immOiHandle  - Same as saImmOiAdminOperationResult.
  875                    invocation   - Same as saImmOiAdminOperationResult.
  876                    result       - Same as saImmOiAdminOperationResult.
  877                    returnParams - Parameters to return (new for A.2.11)
  878                                   See A.02.01 spec for the 'params' arg
  879                                   of saImmOmAdminOperationInvoke_2.
  880 
  881   Return Values : Same as saImmOiAdminOperationResult
  882                   SA_AIS_ERR_VERSION for the case when
  883                   saImmOiAdminOperationResult_o2 is used with
  884                   returnParams!=NULL and *not* with A.2.11 set
  885                   for the immOihandle.
  886 
  887 Allows the OI to reply on the admin-operation with the added list
  888 of return parameters. Works otherwise in the same way as the
  889 original saImmOiAdminOperationResult. If returnParams is set to
  890 NULL then the reply is identical to using the original reply form.
  891 The returnParams argument is constructed in exactly the same way
  892 as the 'params' argument is constructed in
  893 saImmOmAdminOperationInvoke_2.
  894 
  895 
  896 (2)
  897 typedef void
  898  (*SaImmOmAdminOperationInvokeCallbackT_o2) (
  899         SaInvocationT invocation,                          /* in */
  900         SaAisErrorT operationReturnValue,                  /* in */
  901         SaAisErrorT error,                                 /* in */
  902         const SaImmAdminOperationParamsT_2 **returnParams);/* in */
  903 
  904 typedef struct {
  905                SaImmOmAdminOperationInvokeCallbackT_o2
  906                saImmOmAdminOperationInvokeCallback;
  907 } SaImmCallbacksT_o2;
  908 
  909 
  910   Arguments : Same as SaImmOmAdminOperationInvokeCallbackT
  911               plus the 'returnParam' argument (new for A.2.11).
  912               See A.02.01 spec for the 'params' arg of
  913               saImmOmAdminOperationInvoke_2.
  914 
  915 Defines a new OM callback that allows asynchronous admin-ops to
  916 receive the added return parameter list. Note that 'returnParams'
  917 is here a 'const' parameter, indicating that deallocation of the
  918 parameter list will be done by the library (not the user).
  919 The deallocation is done after the user returns from the callback.
  920 
  921 
  922 (3)
  923 extern SaAisErrorT
  924 saImmOmInitialize_o2(
  925         SaImmHandleT *immHandle,               /* in */
  926         const SaImmCallbacksT_o2 *immCallbacks,/* in */
  927         SaVersionT *version);                  /* out */
  928 
  929 Arguments	: immHandle - Same as SaImmOmInitialize A.02.01
  930 		  immCallbacks - The new A.02.11 callback.
  931 		  version - Same as SaImmOmInitialize A.02.01
  932 
  933 Return values : Same as SaImmOminitialize A.02.01
  934 
  935 A new OM-handle-initialize call is needed for supporting the new
  936 callback 'SaImmCallbacksT_o2'. Se the discription of saImmOmInitialize
  937 in A.02.01.
  938 
  939 
  940 (4)
  941 extern SaAisErrorT
  942 saImmOmAdminOperationInvoke_o2(
  943         SaImmAdminOwnerHandleT ownerHandle,           /* in */
  944         const SaNameT *objectName,                    /* in */
  945         SaImmContinuationIdT continuationId,          /* in */
  946         SaImmAdminOperationIdT operationId,           /* in */
  947         const SaImmAdminOperationParamsT_2 **params,  /* in */
  948         SaAisErrorT *operationReturnValue,            /* out */
  949         SaTimeT timeout,                              /* in */
  950         SaImmAdminOperationParamsT_2 ***returnParams);/* out */
  951 
  952   Arguments     :  ownerHandle  - Same as saImmOmAdminOperationInvoke_2
  953                    objectname   -       -"-
  954                    continuationId-      -"-
  955                    operationId  -       -"-
  956                    params       -       -"-
  957                    operationReturnValue--"-
  958                    timeout      -       -"-
  959                    returnParams - Parameters to return (new for A.2.11)
  960                                   See A.02.01 spec for the 'params' arg
  961                                   of saImmOmAdminOperationInvoke_2.
  962 
  963   Return Values : Same as saImmOmAdminOperationInvoke_2,
  964                   i.e. no new return codes for the A.2.11 version,
  965                   except SA_AIS_ERR_VERSION for the case when
  966                   saImmOiAdminOperationInvoke_o2 is used but
  967                   not with A.2.11 set for the Om-handle.
  968 
  969 
  970 The synchronous admin-operation downcall. The only difference with
  971 respect to the original saImmOmAdminOperationInvoke_2 is the new
  972 added out parameter 'returnParams'. Note that 'returnParams' is
  973 NOT declared const, indicating that it is the users responsibility
  974 to de-allocate the parameter list. See next point.
  975 
  976 
  977 (5)
  978 extern SaAisErrorT
  979 saImmOmAdminOperationMemoryFree(
  980         SaImmAdminOwnerHandleT ownerHandle,          /* in */
  981         SaImmAdminOperationParamsT_2 **returnParams);/* in */
  982 
  983   Arguments     :  ownerHandle  - Same ownerHandle as used in
  984                                   saImmOmAdminOperationInvoke_o2
  985                    returnParams - Same returnParams as used in
  986                                   saImmOmAdminOperationInvoke_o2
  987 
  988   Return Values :  SA_AIS_OK
  989                    SA_AIS_ERR_BAD_HANDLE
  990                    SA_AIS_ERR_LIBRARY
  991                    SA_AIS_ERR_VERSION  (not using A.2.11)
  992 
  993 Allows the OM client to free the memory for returnParams.
  994 The returnParams must have been received from a call to
  995 saImmOmAdminOperationInvoke_o2.
  996 
  997 (6)
  998 #define SA_IMM_PARAM_ADMOP_ERROR      "SaImmAdminOperationError"
  999 
 1000 If an OI intends to reply with an error code (no SA_AIS_OK) as the
 1001 result of an admin-operation, then the OI may include an error
 1002 string in the list of return parameters. Typically this will be the
 1003 only paramteter included when there is an error, but the API allows
 1004 any number of parameters to be returned regardless of the return code.
 1005 The error string is identified by a parameter with the parameter-name
 1006 of SA_IMM_PARAM_ADMOP_ERROR. This is a convention (and not yet
 1007 standard). But the convention at least allows for the decoupling of
 1008 OM and OI. It allows an independently developed OM client to probe
 1009 for an error string when it receives a non OK result.
 1010 
 1011 (7)
 1012 #define SA_IMM_PARAM_ADMOP_ID_ESC     0x8000000000000000LL
 1013 #define SA_IMM_PARAM_ADMOP_NAME       "SaImmAdminOperationName"
 1014 
 1015 If an OM client sets the 'operationId' to the value of
 1016 SA_IMM_PARAM_ADMOP_ID_ESC, then it indicates the intention to
 1017 identify the admin-operation using a string name instead of the
 1018 numerical operationId. The OM client must then include a parameter
 1019 with the name SA_IMM_PARAM_ADMOP_NAME in the paramter list.
 1020 The type of this parameter must be SaStringT and the value is the
 1021 name of the operation.
 1022 This is a convention, but it allows decoupling between the OI
 1023 client and OM client in the use of admin-operation names.
 1024 In future releases, this convention may also be the basis for
 1025 supporting the inclusion of amdmin-operation definitions associated
 1026 with an imm-class; and for supporting more elaborate runtime type
 1027 checking of admin-operation invocations and replies, by the imm
 1028 service.
 1029 
 1030 
 1031 OI AUGMENTED CCBs (4.2)
 1032 ====================================
 1033 http://devel.opensaf.org/ticket/1963
 1034 
 1035 OpenSAF 4.2 adds the immsv enhancement of supporting OI augmented
 1036 CCBs. This feature is available for OI clients that register with
 1037 imm version A.2.11.
 1038 
 1039 The basic idea is to allow a configuration object implementer to
 1040 add CCB operations to an existing CCB, in the context of create-,
 1041 delete-, or modify-callbacks on a configuration object. The OI
 1042 then extends the set of changes, originating from an OM client,
 1043 that will commit or abort atomically with the original CCB.
 1044 This could be seen as a variant of 'database triggers'. In
 1045 other words, logic that is triggered by changes initiated
 1046 by a client and which extends the set of modifications done by
 1047 the transaction.
 1048 
 1049 Warnings.
 1050 --------
 1051 A note of warning: This feature must be used with prudence
 1052 and care by OI designers. The feature allows the OI to add side
 1053 effects to a CCB, which the originator of the CCB typically is
 1054 unaware of. It may also impact the performance characteristics
 1055 of the ccb use-case, or prevent the ccb from being committed,
 1056 e.g. by causing timeout of the OM client during processing.
 1057 Default timeout for an OM client is 10 seconds. Note also that
 1058 the OI-side callback timeout monitoring (by default 6 seconds)
 1059 is disabled during an augmentation. OI augmentations to the
 1060 same objects operated on by the original OM client could subvert
 1061 or interfere with the effect intended by the originating OM client.
 1062 The OI should keep the augmentation activity "pure"in the sense of
 1063 only doing the ccb-augmentations based on the OIs internal state.
 1064 For example, it should not performe extraneous blocking operations
 1065 such as invoking synchronous admin-operations during the augmentation. 
 1066 
 1067 The OI designer also needs to take care in avoiding deadlocks
 1068 with other OIs. More on this below.
 1069 
 1070 The ideal pattern of use for this feature is when the OI adds
 1071 a small ammount of operations that are directed at 'implementation
 1072 config objects' private to the OI. The risk for interference with
 1073 the intention of the original CCB is then minimal. We are not talking
 1074 about runtime data here. It has always been possible for an OI to
 1075 add/remove/update runtime objects/attributes as a side effect of the
 1076 apply of a configuration change. Runtime data is updated via the OI
 1077 API, may be triggered by CCBs, but are not part of CCBs, i.e. not
 1078 atomic with them. Runtime data is used by the OI to communicate
 1079 information *to* OM clients. Here we are instead talking about
 1080 'private configuration data', or configuration data for a sub service,
 1081 or related service, that needs to be modified as a side effect of
 1082 the original modification and to be part of the atomic commit
 1083 of the CCB.
 1084 
 1085 Api
 1086 ---
 1087 
 1088 The API available to the OI for this feature under IMM interface
 1089 version A.2.11 consists of one new synchronous call:
 1090 
 1091 	extern SaAisErrorT
 1092 	saImmOiAugmentCcbInitialize(
 1093 			     SaImmOiHandleT immOiHandle,          /* in */
 1094 			     SaImmOiCcbIdT ccbId,                 /* in */
 1095 			     SaImmCcbHandleT *ccbHandle,          /* out */
 1096 			     SaImmAdminOwnerHandleT *ownerHandle);/* out */
 1097 
 1098   Arguments     :  immOiHandle  - The immOiHandle received in the callback.
 1099 		   ccbId        - The ccbId received in the callback.
 1100 		   ccbHandle    - Pointer to the CCB handle to be initialized.
 1101                    ownerHandle  - Pointer to the administrative owner handle
 1102                                   to be initialized.
 1103 
 1104   Return Values :  SA_AIS_OK
 1105                    SA_AIS_ERR_BAD_HANDLE
 1106                    SA_AIS_ERR_LIBRARY
 1107                    SA_AIS_ERR_INVALID_PARAM
 1108                    SA_AIS_ERR_BAD_OPERATION (not inside valid ccb callback)
 1109                    SA_AIS_ERR_TRY_AGAIN
 1110                    SA_AIS_ERR_VERSION  (not using A.2.11)
 1111 
 1112 The OI inputs its oi-handle and the ccbId it received in the callback.
 1113 In return it gets an OM CCB handle and an AdminOwner handle.
 1114 The Imm service has marked these handles as special. They are
 1115 only valid for the duration of the callback.
 1116 
 1117 An OI is only allowed to invoke the saImmOiAugmentCcbInitialize call
 1118 inside these callbacks:
 1119 
 1120     SaImmOiCcbObjectCreateCallbackT_2
 1121     SaImmOiCcbObjectDeleteCallbackT
 1122     SaImmOiccbobjectModifyCallbackT_2
 1123 
 1124 It would have been nice to support augmentation also inside the
 1125 SaImmOiCcbCompletedCallbackT callback, but since validation is the
 1126 purpose of that callback and there may be many OIs involved in the
 1127 completion of a CCB, changing the base of validation during
 1128 the validation would make validation practically impossible.
 1129 
 1130 The OI augmentation CCB must be 'applied' by the OI before it
 1131 returns from the callback in which the augmentation is done.
 1132 Such an 'apply' of an augmentation is not the actual final apply,
 1133 since that is dependent on the apply of the originating parent CCB.
 1134 The 'apply' of an augmentation is only the confirmation from the
 1135 OI that the augmentation was completed by the OI.
 1136 
 1137 The typical config OI is already using the OM interface to read its
 1138 initial configuration, so all the normal ccb related calls should
 1139 already be accessible using the provided ccbHandle. (The OI needs
 1140 to include saImmOm.h lexically before the include of saImmOi.h).
 1141 
 1142 If the saImmOiAugmentCcbInitialize call succeeds, the OI can
 1143 use the CCB handle to request create, delete, or modify operations,
 1144 followed by an apply, or possibly a finalize without apply resulting
 1145 in abort.
 1146 
 1147 An abort of the augment CCB by the OI will abort the entire
 1148 originating CCB, unless the augmentation was empty at the time of
 1149 abort/finalize. The augmentation is empty untill at least one
 1150 augmented create, delete or modify operation has succeeded and
 1151 no augment operation has failed with SA_AIS_ERR_FAILED_OPERATION.
 1152 The FAILED_OPERATION return code, as usual, means abort of *entire*
 1153 CCB, not just the operation that resulted in this return code.
 1154 And *entire* CCB here means the OM originated parent CCB, not just
 1155 the augmented CCB done by the OI. Thus augmented CCBs are not
 1156 'nested transactions' in that they can not be aborted with contents,
 1157 independently of the parent CCB.
 1158 
 1159 If the OI adds operations to the CCB for objects which it is itself
 1160 the implementer for, then it will not receive any OI-callback for
 1161 such operations. This is mainly to avoid the OI getting into
 1162 deadlock with itself. But it should also not be necessary for
 1163 the OI to get such callbacks, since it already has the information
 1164 about the changes it itself added. The OI may even create a modify
 1165 on the object that is the basis of a create or modify callback it is
 1166 executing in. (This should be an unusual case and done with care,
 1167 as it is in some sense overriding the 'intention' of the original
 1168 client).
 1169 
 1170 It will still be posible for two or more OIs to get into a
 1171 distributed deadlock with each other by their use of CCB
 1172 augmentation. If such an callback cycle is created, the entire
 1173 OM CCB will timeout and be aborted. This could make desirable
 1174 and correct configuration changes impossible to commit/apply.
 1175 
 1176 The saImmOiAugmentCcbInitialize call also returns 'ownerHandle'.
 1177 This admin owner handle is initialized to the same admin-owner
 1178 used by the originating OM client. It can be used by the OI to
 1179 set admin-owner for objects the OI intends to modify or delete
 1180 and where it does not know that the admin-owner is already set
 1181 by the originating OM client, (create operations always get
 1182 admin-owner set automatically as part of creation).
 1183 The 'ownerHandle' always has 'releaseOwnershipOnFinalize (ROF) set
 1184 to true. This is the case even if the original admin owner has it
 1185 set to false. If the original admin-owner has ROF==true, then
 1186 the augmented CCB actually uses the identical admin-owner. Any
 1187 objects operated on by the augmented ccb, are added to the original
 1188 admin-owner and will be released whenever the original om user
 1189 closes their handle or disconnects. If the original admin-owner
 1190 has ROF==false, then the augmented CCB uses a new admin-owner
 1191 with the same admin-owner-name and with ROF=true. The release
 1192 of the OI augmented admin-owner is done when the parent CCB commits
 1193 (is applied). The admin-owner can not be released when the augmented
 1194 ccb is applied because the augmented operations are still pending
 1195 the apply of the parent ccb.
 1196 
 1197 The OI designer needs to understand that if they set admin-owner
 1198 on some objects which already have admin owner set by the original
 1199 OM user and the original OM user has releaseOnFinalize as false,
 1200 then the OI will cause a release of admin-owner on these objects
 1201 which could be a 'surprise' (cause problems) for the OM user.
 1202 
 1203 Again, the OI designer intending to augment CCBs needs to think
 1204 carefully about what they are trying to do and why, to avoid
 1205 creating an application that appears unmanagable.
 1206 
 1207 
 1208 Support for SA_IMM_CCB_ALLOW_NULL_OI (4.2)
 1209 =========================================
 1210 http://devel.opensaf.org/ticket/1997
 1211 
 1212 OpenSAF 4.2 exteds the SaImmCcbFlagsT type with a new flag value:
 1213 
 1214   #define SA_IMM_CCB_ALLOW_NULL_OI 0x0000000000000100
 1215 
 1216 This flag is available for OI clients that register with
 1217 imm version A.2.11.
 1218 
 1219 The flag should be set in *combination* with the existing
 1220 SA_IMM_CCB_REGISTERED_OI flag. Its effect is that it relaxes
 1221 the requirement on OI to allow no registered OI when there
 1222 is no registered implementer *name* for the class/object.
 1223 If an implementer name exists, but no implemener is currently
 1224 attached, then the semantics is still the same as
 1225 SA_IMM_CCB_REGISTERED_OI, the CCB operation is rejected with
 1226 SA_AIS_ERR_NOT_EXIST.
 1227 
 1228 This will be the setting hardwired into commands such as 'immcfg'
 1229 when invokde in safe mode.
 1230 
 1231 There are then 3 possible flag settings for saImmOmCcbinitialize:
 1232 
 1233 No flags: Allow CCBs to proceed despite missing OIs, regardless
 1234 of the way they are missing. This variant is the most unsafe and
 1235 should really only be used by special applications such as
 1236 immloader. The danger is that an OI that is only temporarily
 1237 detached, say during a failover, will not get the opportunity
 1238 to validate the CCB. This setting is typically too unsafe.
 1239 
 1240 SA_IMM_CCB_REGISTERED_OI: See the IMM spec for A.02.01.
 1241 A CCB operation on an object will only be allowed when there
 1242 is an OI name registered for it AND the OI is attached.
 1243 This is the most safe way of executing a CCB, but it may be
 1244 too restrictive in some cases. The best example is during
 1245 installation when some config objects are created before
 1246 the OI is started.
 1247 
 1248 SA_IMM_CCB_REGISTERED_OI | SA_IMM_CCB_ALLOW_NULL_OI:
 1249 This is the relatively safe variant in the middle.
 1250 A CCB operation on an object is allowed if there is an
 1251 implementer, or if there is no implementer name defined
 1252 for the object.
 1253 
 1254 
 1255 Support for the environment variable IMMA_SYNCR_TIMEOUT (4.2)
 1256 =============================================================
 1257 http://devel.opensaf.org/ticket/1808
 1258 
 1259 The IMMA libraries (both OM and OI) have a hard-wired timeout
 1260 on synchronous downcalls (to the IMMND), of 10 seconds.
 1261 
 1262 This enhancement changes that to a default timeout, with the
 1263 option for the IMMA client to override the default by setting a
 1264 value for the environment variable:
 1265 
 1266             IMMA_SYNCR_TIMEOUT.
 1267 
 1268 The unit of time is 10 milliseconds. Thus the default value
 1269 for IMMA_SYNCR_TIMEOUT is 1000 (10 seconds). The minimum allowed
 1270 value is 10 (0.1 seconds). Trying to set a value less than 10
 1271 will only be interpreted as the value 10.
 1272 
 1273 The 10 second default timeout is a compromize. It is very long for
 1274 some applications to be blocked for up to 10 seconds, for example
 1275 many OIs can get in trouble with such a long wait.
 1276 
 1277 For other applications, 10 seconds is too short, for example
 1278 "immcfg -f huge-batch.xml" with PBE enabled can get ERR_TIMEOUT
 1279 during apply (http://devel.opensaf.org/ticket/1786).
 1280 
 1281 Applications with relatively tight realtime restrictions can set
 1282 IMMA_SYNCR_TIMEOUT to a low value, avoiding the risk of getting
 1283 blocked for 10 seconds, but with an increased risk of getting
 1284 ERR_TIMEOUT on synchronous calls.
 1285 
 1286 Some batch operations, particularly deletes of large subtrees of
 1287 imm data, where one delete downcall is expanded into a cascade of
 1288 deletes of objects in the subtree, can benefit from setting
 1289 a high value on IMMA_SYNCR_TIMEOUT.
 1290 
 1291 The IMMA library samples the IMMA_SYNCR_TIMEOUT environment variable
 1292 at OM/OI handle initialize. The handle is stamped with the value of
 1293 of IMMA_SYNCR_TIMEOUT (if it exists), at the time of handle create.
 1294 
 1295 This means that the timeout can not be altered for every individual
 1296 downcall. But it also means that one process may have different
 1297 timeouts set for different handles. Sub-handles (ccb, admin-owner,
 1298 iteration, accessor) inherit the timeout value of their associated
 1299 om-handle.
 1300 
 1301 Support for search with SA_IMM_SEARCH_GET_CONFIG_ATTR (4.3)
 1302 =============================================================
 1303 http://devel.opensaf.org/ticket/1897
 1304 
 1305 The IMM A.2.11 API included an added flag value for the SaImmSearchOptionsT
 1306 type for requesting only config attributes: SA_IMM_SEARCH_GET_CONFIG_ATTR.
 1307 This feature did not make it into OpenSAF 4.2, but will now instead be part
 1308 of OpenSaf 4.3.
 1309 
 1310 The IMM API version where this is implemented is now IMM A.2.12 in OpenSAF(4.3)
 1311 
 1312 The main driver for this feature is the very common need for an OI implementation
 1313 to fetch the initial configuration, without also asking for the values of
 1314 runtime attributes. If an OI performs a read-request (over the OM API) that
 1315 includes pure runtime attributes for which it itself is the implementer, then
 1316 the OI will currently get into deadlock with itself (if the OI is not multihreaded).
 1317 
 1318 For cached runtime attributes this problem does not exist, but typically the OI
 1319 is not interested in fetching its own cached runtime attributes, which means
 1320 a waste of memory and execution.
 1321 
 1322 The problem has been that the only way for the OI to only fetch its config
 1323 attributes, is to explicitly enumerate all the config attributes it wants
 1324 to fetch.
 1325 
 1326 This new feature will make it much simpler for the OI to express this
 1327 common type of query.
 1328 
 1329 The SA_IMM_SEARCH_GET_CONFIG_ATTR can either be used in the obvious way
 1330 by using it in a saImmOmSearchInitialize_2 request.
 1331 
 1332 For requests of a specific object, the saImmOmAccessorGet_2 api is simpler,
 1333 but that API does not have any searchParameters argument.
 1334 This enhancement still supports the GET_CONFIG_ATTR feature also for
 1335 saImmOmAccessorGet, by what may be called a "hack" or a convention.
 1336 If the 'attributeNames' parameter to saImmOmAccessorGet_2 has only one
 1337 attributeName with the string value: "SA_IMM_SEARCH_GET_CONFIG_ATTR",
 1338 then the immsv implementation will interpret this as request for all
 1339 config attributes and NOT as a request for the attribute named
 1340 "SA_IMM_SEARCH_GET_CONFIG_ATTR".
 1341 
 1342 The SA_IMM_SEARCH_GET_CONFIG_ATTR feature should work over immutils and
 1343 immutills also has an added wrapper for the saImmOmAccessorGet_2 use
 1344 that encapsulates the convention solution just described.
 1345 The new function added to immutils has the signature:
 1346 
 1347 extern SaAisErrorT
 1348 immutil_saImmOmAccessorGetConfigAttrs(SaImmAccessorHandleT accessorHandle,
 1349                                       const SaNameT *objectName,
 1350                                       SaImmAttrValuesT_2 ***attributes);
 1351 
 1352 
 1353 Support for redundant data export special applier OI (4.3)
 1354 ==============================================================================
 1355 http://devel.opensaf.org/ticket/2873
 1356 http://devel.opensaf.org/ticket/2883
 1357 http://devel.opensaf.org/ticket/2892
 1358 
 1359 The 'special applier' enhancement implemented in OpenSAF 4.3 is a mechanism
 1360 intended to assist the replication of chosen parts of the imm database to
 1361 a generic receiver external to the imm service.
 1362 
 1363 As the name implies, it uses the same OI interface that regular appliers use.
 1364 The difference is the way the subset of imm data to be sent to the applier
 1365 is configured and that cached runtime data may be included.
 1366 
 1367 Regular appliers (and OIs) will get callbacks depending on which classes or
 1368 objects they have explicitly registered for, using saImmOiClassImplementerSet
 1369 or saImmOiObjectImplementerSet. This is a two level mapping:
 1370 
 1371         oiHandle->implementerName
 1372         implementerName->classes/objects
 1373 
 1374 The two level maping does not survive cluster restarts (it is not persistent),
 1375 but the second level mapping survives disconnect at the first level. Thus the
 1376 bindings related to an implementerName survives process termination or
 1377 oiHandle close.
 1378 
 1379 Regular appliers (and OIs) are typically application specific, listening only
 1380 for changes on config data that is relevant to their application.
 1381 
 1382 The special applier is generic. It does not use saImmOiClassImplementerSet or
 1383 saImmOiObjectImplementerSet to set up the second level map. Instead it
 1384 will receive callbacks for objects with attribute values if the attribute
 1385 definition in the class definition for the object has the SA_IMM_ATTR_NOTIFY
 1386 flag set to on.
 1387 
 1388 The IMM A.2.12 API extension supported in OpenSAF 4.3 reserves a new flag value,
 1389 labeled:
 1390 
 1391         SA_IMM_ATTR_NOTIFY
 1392 
 1393 in the SaImmAttrFlagsT type. The SaImmAttrFlagsT type is used when defining
 1394 attributes as part of an imm class definition.
 1395 
 1396 The meaning of this new flag is that, if it is on, then any change to the value
 1397 of that attribute in any instance (imm object) of that class, will generate a
 1398 callback for any attached special appliers. Because the flag is associated with
 1399 class definitions, it is persistent. Toggling the flag on an attribute definition
 1400 is in principle an imm schema change (see SCHEMA CHANGE section above). It is an
 1401 update of a class definition. Since class definitions are persistent, the special
 1402 applier second level mapping implementerName->class/attributes will survive a
 1403 cluster restart. This is also necessary since there exists no regular implementer
 1404 that can set up the second level binding dynamically at cluster start.
 1405 
 1406 For the first level mapping oiHandle->implementerName, the special applier is
 1407 currently fixed to only allow two "magic" applier names:
 1408 
 1409         #define OPENSAF_IMM_REPL_A_NAME "@OpenSafImmReplicatorA"
 1410         #define OPENSAF_IMM_REPL_B_NAME "@OpenSafImmReplicatorB"
 1411 
 1412 These applier names are "magic" in the sense that the imm service will recognize
 1413 these names as special and generate the special applier callbacks to them if
 1414 they are attached. The limitation to two names may be relaxed in the future if
 1415 needed. The current vision is that only one logical replicator is needed
 1416 because the service is generic. The generic replicator may then push the data
 1417 on to a tracking service (such as Ntf) that in its turn allows multiple
 1418 consumers. The generic replicator may even itself push the data on to more than
 1419 one outgoing communication service. Thus there should typically not be needed
 1420 more than one logical replicator.
 1421 
 1422 The reason for two special applier names instead of just one, is to allow a
 1423 logically unique replicator to be implemented redundantly as an active/standby
 1424 pair. If only one special applier name would be used, then there would be an
 1425 inevitable gap in time after the current active detaches and the new active
 1426 attaches. An applier or implementer name can at most be attached to one
 1427 oi-handle in the cluster, at any given time. Thus the standby special applier
 1428 needs to receive the same callback stream as the active, using a distinct
 1429 special applier name, but will discard the data based on information it
 1430 receives from the active (e.g. via message based checkpoint).
 1431 
 1432 Having both special applier names attached at the same time on the same node
 1433 should be avoided. The imm service will only generate the special applier
 1434 callbacks towards one special applier at one node at any point in time. Thus
 1435 only one of them will receive the special applier callbacks if both are
 1436 attachhed on the same node. It is implementation dependent which one.
 1437 
 1438 The SA_IMM_ATTR_NOTIFY flag may be toggled using immcfg (#2892), or as part
 1439 of a regular schema change in an SMF campaign. Export/import of the new flag
 1440 to imm xml format is supported by immdump/immload (#2883).
 1441 
 1442 Although the special applier uses the same SaImmOi callback interface as regular
 1443 implementers and regular appliers, there are some special rules concerning the
 1444 contents of the parameters provided from the imm service.
 1445 
 1446 ----------------------------
 1447 
 1448 Config data is changed using Ccb's and the special applier will get the same
 1449 sequence of operational callbacks (create/modify/delete) followed by completed
 1450 and apply. For the operational callbacks the following apply:
 1451 
 1452 SaImmOiCcbObjectCreateCallbackT_2 will contain attribute values for *all*
 1453 attributes that have the SA_IMM_ATTR_NOTIFY flag on. The attribute value may have
 1454 been set explicitly by the om-client in the ceate downcall, or the value may be
 1455 the default value for the attribute, or the value may be the empty value.
 1456 In addition, the callback will contain the Rdn attribute and the system attributes
 1457 named SaImmAttrClassName and SaImmAttrAdminOwnerName. The Rdn attribute is
 1458 necessary even if it is not flagged as notifiable, since the special applier needs
 1459 it for constructing the DN for the object to be created. The class-name and
 1460 admin-owner name are needed for the special applier, or its downstream clients, to
 1461 interpret what kind of object is being created and who is creating it.
 1462 
 1463 SaImmOiCcbObjectModifyCallbackT_2 will contain only attribute values that have
 1464 been modified in this operation *and* that have the SA_IMM_ATTR_NOTIFY flag on.
 1465 In addition, the callback will contain the system attributes SaImmAttrClassName.
 1466 The system attribute SaImmAdminOwnerName will also be included if the value
 1467 has not been provided already in a previous callback for the CCB. Thus if a prior
 1468 create or modify has been sent then the current modify will not add the admin-
 1469 owner system attribute. Admin-owner never changes for a CCB.
 1470 The class-name may be needed if the special applier attached after the object
 1471 was created and had not yet initialized by reading from the imm. Indeed it is
 1472 left open to the implementor of the special applier to decide how and even if
 1473 they should initialize by reading from the imm. The admin-owner allows the
 1474 receiver to interpret who is modifying the object.
 1475 
 1476 SaImmOiCcbObjectDeleteCallbackT can obviosly not contain any attribute values.
 1477 Only the DN of the object to be deleted is provided, since that is how the
 1478 callback is defined. However, the imm service guarantees that a CCB that
 1479 contains delete callbacks, will also contain at least one modify or create
 1480 callback containing at least the SaImmAdminOwnerName system attribute.
 1481 For any ccb (where ccb != 0), the first special applier callback is guaranteed
 1482 to hold the admin-owner-name. If the first operation is a real create or modify
 1483 then that will hold the admin-owner-name. But if the first operation in the real
 1484 ccb is a delete-operation, then the immsv will first generate a fake modify to
 1485 the special applier, using the same object/dn to be deleted, containing the
 1486 admin-owner-name. After the fake modify, the real delete will be generated
 1487 towards the special applier.
 1488 
 1489 This allows the receiver to interpret who is deleting the objects. The class-name
 1490 for the object should not be needed since if the receiver is already aware of the
 1491 object to be deleted, then they should already know the class for that object
 1492 and if they are not aware of the object, then they can simply discard the delete
 1493 operation since it is a delete of an object that the receiver does not have.
 1494 
 1495 The SaImmOiCcbCompletedCallbackT, saImmOiCcbApplyCallbackT and
 1496 SaImmOiCcbAbortCallbackT have exactly the same semantics as always. The completed
 1497 callback is typically ignored by appliers.
 1498 
 1499 ----------------------------
 1500 
 1501 Cached runtime data is created/updated and deleted by an implementer/OI. Each such
 1502 OI downcall operation will generate only the single corresponding OI callback on
 1503 the special applier, but with the ccbId parameter set to *zero*. No completed,
 1504 apply or abort callbacks are generated for cached runtime data operations. Persistent
 1505 runtime data is always also cached, so the same rules apply for persistent runtime
 1506 data. Non-cached runtime attributes are not allowed to have the SA_IMM_ATTR_NOTIFY
 1507 flag set.
 1508 
 1509 SaImmOiCcbObjectCreateCallbackT_2 will contain attribute values for *all* cached
 1510 runtime attributes that have the SA_IMM_ATTR_NOTIFY flag on. The attribute value
 1511 may have been set explicitly by the OI in the ceate downcall, or the value may be
 1512 the default value for the attribute. Cached runtime attributes without default
 1513 value must get a value provided by the OI in a saImmOiRtObjectCreate_2 call.
 1514 In addition, the callback will contain the Rdn attribute and the system attributes
 1515 named SaImmAttrClassName and SaImmAttrImplementerName. The Rdn attribute is
 1516 necessary even if it is not flagged as notifiable, since the special applier needs
 1517 it for constructing the DN for the runtime object to be created. The class-name and
 1518 implementer name are needed for the special applier, or its downstream clients, to
 1519 interpret what kind of runtime object is being created and who (what implementer)
 1520 is creating it. The ccbId parameter will be zero.
 1521 
 1522 SaImmOiCcbObjectModifyCallbackT_2 will contain only cached runtime attribute values
 1523 that have been updated in this operation *and* that have the SA_IMM_ATTR_NOTIFY
 1524 flag on. In addition, the callback will contain the system attributes
 1525 SaImmAttrClassName and SaImmImplementerName. The class-name may be needed if the
 1526 special applier attached after the object was created and had not yet initialized
 1527 by reading from the imm. Indeed it is left open to the implementor of the special
 1528 applier to decide how and even if they should initialize by reading from the imm.
 1529 The implementer-name allows the receiver to interpret who is updating the runtime
 1530 attributes. Note that the object containing the cached runtime attributes may be
 1531 either a runtime object or a config object. This since config objects may contain
 1532 runime attributes. The ccbId parameter will be zero.
 1533 
 1534 SaImmOiCcbObjectDeleteCallbackT can obviosly not contain any attribute values.
 1535 Only the DN of the object to be deleted is provided, since that is how the
 1536 callback is defined. For runtime object deletes, the imm can not provide any
 1537 'fake' modify callback to provide class-name and implementer-name. This is because
 1538 runtime data is not operated on using Ccbs/transactions. So there is no way to
 1539 group and corelate several operations as belonging to the same ccb. But the
 1540 class-name for the runtime object should not be needed since if the receiver is
 1541 already aware of the runtime object to be deleted, then they should already know
 1542 the class for it and if they are not aware of the runtime object, then they can
 1543 simply discard the delete operation, since it is a delete of an object that the
 1544 receiver does not have. Similarly, the implementer-name should not be needed.
 1545 Unlike admin-owner-name, which may be different for different ccbs operating on
 1546 the same config object during its life cycle, implementer-name will never change
 1547 for a runtime object. The ccbId parameter will be zero.
 1548 
 1549 
 1550 Support for attribute definitions with SA_IMM_ATTR_NO_DUPLICATES (4.3)
 1551 ======================================================================
 1552 http://devel.opensaf.org/ticket/1545
 1553 
 1554 The IMM A.2.11 API included an added flag value for the SaImmAttrFlagsT type
 1555 used in attribute definitions for class create: SA_IMM_ATTR_NO_DUPLICATES.
 1556 This feature did not make it into OpenSAF 4.2, but will now instead be part
 1557 of OpenSaf 4.3
 1558 
 1559 The IMM API version where this is implemented is now IMM A.2.12 in OpenSAF(4.3)
 1560 
 1561 This can only be set in conjunction with the SA_IMM_ATTR_MULTI_VALUE flag.
 1562 The effect of setting also the SA_IMM_ATTR_NO_DUPLICATES flag is that the
 1563 attribute behaves as a 'set' instead of as a 'bag'. The SAF IMM standard
 1564 defines multivalued attributes very loosely as attributes that may have many
 1565 values. Nothing is said about uniquenes of values, hence duplicates must be
 1566 allowed. In most cases the intention with a multi valued attribute is likely
 1567 to be to not have duplicate values. It is now possible to get this more
 1568 restricted semantics by settign the SA_IMM_ATTR_NO_DUPLICATES_FLAG.
 1569 
 1570 It is possible to add the NO_DUPLICATES flag to a MULTI_VALUE attribute
 1571 definition in an existing class, via a schema change. But such an upgrade
 1572 will only succeed if no current instances of the class have duplicates in
 1573 this attribute. The imm service will perform a runtime check of this before
 1574 allowing the upgrade. Application maintainers that intend to use this flag
 1575 need to consider the fact that adding the flag may cause an upgrade campaign
 1576 to fail due to site specific values on such multivalued attributes.
 1577 
 1578 There is a reason why the imm service does not go one step further and
 1579 consolidate duplicates instead of just check and reject upgrades that
 1580 encounter duplicates. If the imm service where to do such consolidation,
 1581 it would mutate the values of application data without informing the OI
 1582 of such changes. We dont see that as acceptable and predictable behavior.
 1583 The approach taken is instead to make it the applications responsibility to
 1584 eliminate duplicates before the schema change attempt, possibly as a prior
 1585 application step in the campaign.
 1586 
 1587 
 1588 IMMA_MAX_OPEN_SEARCHES_PER_HANDLE (4.3)
 1589 ==============================================================
 1590 http://devel.opensaf.org/ticket/2977
 1591 http://devel.opensaf.org/ticket/3036
 1592 
 1593 A relatively common error in imm applications is to omitt closing of
 1594 search handles or accesor handles. This results in a "resource leak"
 1595 which will be removed only if/when the application closes the om-handle
 1596 under which the search/accessor handles where created, or when the
 1597 application terminates.
 1598 
 1599 Such resource leaks will manifest themselves as increasing use of memory
 1600 in the local IMMND server. How large such leaks will be depends on the
 1601 applicaiton behavior. But if a search is initialized for iteration over
 1602 a large part of the imm database and is dropped before it is consumed,
 1603 then very large ammounts of memory could remain allocated.
 1604 
 1605 To make early detection of such resource leaks possible, a limit on
 1606 the maximum number concurrently allocated search handles per om handle
 1607 has been introduced in OpenSAF 4.3.
 1608 
 1609 A default maximum of 100 concurrently open search handles per om-handle
 1610 are allowed. That is, at most 100 open handles of type SaImmSearchHandleT
 1611 or SaImmAccessorHandleT created using the same SaImmHandleT are allowed.
 1612 This limit, associated with each om-handle, is checked by the imma library
 1613 each time a search/accessor handle is created.
 1614 
 1615 Applications that have special and legitimate needs to provide many
 1616 concurrent imm searches, can change the limit from the default value.
 1617 The application does this by setting a higher value on the environment
 1618 variable IMMA_MAX_OPEN_SEARCHES_PER_HANDLE.
 1619 
 1620 Applications can also avoid the limit by using more than one om-handle
 1621 as the root resource for allocating search handles. For applications
 1622 that legitimately need several hundred concurrent search handles,
 1623 we recommend this solution rather than raising the limit per om-handle,
 1624 as it is more efficient in server processing. It also allows the searches
 1625 to be controlled by more than one client thread.
 1626 
 1627 It may also be usefull for applications to *lower* the value in early
 1628 testing. This increases the chances that application resource leaks are
 1629 detected early in development.
 1630 
 1631 The IMMA library samples the IMMA_MAX_OPEN_SEARCHES_PER_HANDLE environment
 1632 variable at OM handle initialize. The om-handle is stamped with the value
 1633 of this variable, if it exists, at the time of handle create.
 1634 
 1635 There is also a much higher sanity limit defined on the server side.
 1636 Currently defined as 5000 open searches per om-handle. That limit is
 1637 not configurable.
 1638 
 1639 
 1640 Optimize subtree search           (4.3)
 1641 ==============================================================================
 1642 http://devel.opensaf.org/ticket/1844
 1643 
 1644 
 1645 Allow saImmOiImplementerSet during imm-sync       (4.3)
 1646 =======================================================
 1647 http://devel.opensaf.org/ticket/1871
 1648 
 1649 
 1650 Catch some cases of thread interference over handles    (4.3)
 1651 =============================================================
 1652 http://devel.opensaf.org/ticket/1984
 1653 
 1654 
 1655 Pbe & immload uses pre-compiled & parametrized queries  (4.3)
 1656 =============================================================
 1657 http://devel.opensaf.org/ticket/2467
 1658 http://devel.opensaf.org/ticket/2696
 1659 
 1660 
 1661 immcfg -L validate imm-xml file as xml loadable  (4.3)
 1662 ======================================================
 1663 http://devel.opensaf.org/ticket/2766
 1664 
 1665 
 1666 AdminOp gets TRY_AGAIN if OI is transiently detached (4.3)
 1667 ==========================================================
 1668 http://devel.opensaf.org/ticket/2593
 1669 
 1670 
 1671 IMM server must protect itself from invalid data received from imm clients (4.3)
 1672 ================================================================================
 1673 http://devel.opensaf.org/ticket/2747
 1674 
 1675 IMMNDs should subscribe in normal mode towards IMMD (4.3)
 1676 =========================================================
 1677 http://devel.opensaf.org/ticket/2446
 1678 
 1679 saImmOiImplementerClear can avoid exposing a stale oi-handle (4.3)
 1680 ==================================================================
 1681 http://devel.opensaf.org/ticket/2465
 1682 
 1683 immdump should be split into a tool and a daemon (4.3)
 1684 ======================================================
 1685 http://devel.opensaf.org/ticket/2835
 1686 
 1687 
 1688 Notes on upgrading from OpenSAF 4.1/4.2 to OpenSAF 4.3
 1689 ======================================================
 1690 Enhancement #1871 opens up for allowing saImmOiImplementerSet during imm-sync.
 1691 This more liberal behavior does not work well during upgrade from earlier
 1692 OpenSAF releases to the OpenSAF 4.3 release. See ticket #3071.
 1693 
 1694 Because of this upgrade problem, the older more restrictive behavior is still
 1695 enforced in OpenSAF 4.3 unless a flag is turned on in the opensafImmNostdFlags
 1696 runtime attribute in the object: opensafImm=opensafImm,safApp=safImmService.
 1697 The following is the shell command:
 1698 
 1699         immadm -o 1 -p opensafImmNostdFlags:SA_UINT32_T:4 \
 1700            opensafImm=opensafImm,safApp=safImmService
 1701 
 1702 This will set the third bit of the 'opensafImmNostdFlags' runtime attribute
 1703 inside the immsv. Operation-id '1' invoked on the object:
 1704 
 1705  'opensafImm=opensafImm,safApp=safImmService'
 1706 
 1707 has the meaning of 'flags-ON'. Operation-id '2' has the meaning of 'flags-OFF'.
 1708 This flag needs to be turned ON when the upgrade to OpenSAF 4.3 has been
 1709 successfully completed. Thus in the final steps of the upgrade. A cluster
 1710 start/restart of an OpenSAF4.3 system will always automatically turn on
 1711 this flag. 
 1712 
 1713 2PBE Allow IMM PBE to be configured without shared file system (4.4)
 1714 ===================================================================
 1715 http://sourceforge.net/p/opensaf/tickets/21/
 1716 OpenSAF 4.4 adds the enhancement of supporting 2PBE.
 1717 The 2PBE enhancement allows the IMM to have PBE configured so that it
 1718 does not rely on a shared filesystem, such as DRBD.
 1719 
 1720 See: osaf/services/saf/immsv/README.2PBE for details.
 1721 
 1722 immadm supports argument for explicitly setting admin-owner (4.4)
 1723 ================================================================
 1724 http://sourceforge.net/p/opensaf/tickets/613/
 1725 http://sourceforge.net/p/opensaf/tickets/642/
 1726 
 1727 This was actually fixed as a defect made visible by the 2PBE enhancement.
 1728 The IMM service takes stronger ownership of the OpenSAF Imm service object
 1729 mentioned earlier:
 1730 
 1731         'opensafImm=opensafImm,safApp=safImmService'
 1732 
 1733 by more or less permanently setting adminOwner for this object to be
 1734 safImmService. The 'immadm' command has been fixed to be aware of this more
 1735 strict ownership on this particlar object and will automatically use the
 1736 admin-owner 'safImmService' when requested to operate on this object.
 1737 Normal OM API users should not notice any effect of this change.
 1738 
 1739 Support for reference integrity checking - SA_IMM_ATTR_NO_DANGLING (4.4)
 1740 ========================================================================
 1741 http://sourceforge.net/p/opensaf/tickets/49/
 1742 OpenSAF 4.4 adds the immsv enhancement of supporting referential integrity
 1743 checking. The feature is available for clients that register with imm API
 1744 version A.2.13.
 1745 
 1746 A new attribute definition flag has been defined:
 1747 #define SA_IMM_ATTR_NO_DANGLING   0x0000000004000000
 1748 
 1749 See: osaf/services/saf/immsv/README.NO_DANGLING for details.
 1750 
 1751 Support search for objects having NO_DANGLING refrence to a target (4.4)
 1752 ========================================================================
 1753 http://sourceforge.net/p/opensaf/tickets/50/
 1754 OpenSAF 4.4 adds the immsv enhancement of supporting search/iteration
 1755 for finding all objects that have a NO_DANGLING reference to a provided
 1756 target. This feature is available for clients that register with imm API
 1757 version A.2.13.
 1758 
 1759 See: osaf/services/saf/immsv/README.NO_DANGLING for details.
 1760 
 1761 
 1762 Support for saImmOmCcbAbort() and saImmOmCcbValidate() (4.5)
 1763 ========================================================================
 1764 http://sourceforge.net/p/opensaf/tickets/798/
 1765 
 1766 Adds support for an OM client to explicitly request abort of the current
 1767 CCB(id) without finalizing the handle.
 1768 Also support for an OM client to explicitly request validation of the current
 1769 CCB state without any immediate and automatic ccb commit if validation succeeds.
 1770 
 1771 These API additions are available to OM clients that initialize their OM-handle
 1772 with IMM API version A.2.14.
 1773 
 1774 (1)
 1775 extern SaAisErrorT
 1776 saImmOmCcbAbort(SaImmCcbHandleT ccbHandle): /* in */
 1777 
 1778   Arguments     :  ccbHandle  - The ccbHandle for the ccb to be aborted.
 1779 
 1780   Return Values :  SA_AIS_OK
 1781 
 1782                    SA_AIS_ERR_BAD_OPERATION - Not allowed inside a ccb augmentation.
 1783 
 1784                    SA_AIS_ERR_VERSION  (not using A.2.14)
 1785 
 1786                    Returncodes otherwise identical to saImmOmFinalize.
 1787 
 1788 Aborts a CCB(id) without finalizing the ccb-handle.
 1789 Discards any ccb operations currently associated with the ccb-handle.
 1790 Also resets a ccbHandle that has previously received the abort return
 1791 code SA_AIS_ERR_FAILED_OPERATION. 
 1792 Previously it was only possible to explicitly abort an active ccb
 1793 by invoking saImOmCcbFinalize() which also closes the ccb-handle.
 1794 And previously it was not possible to reset a ccbHandle that had
 1795 received the ccb-aborted return code: SA_AIS_ERR_FAILED_OPERATION.
 1796 Such a ccb-handle was doomed and could only be finalized. This is
 1797 no longer the case when the ccbAbort alterative exists. Invoking
 1798 ccbAbort is an explicit handshake between the om-user and the immsv
 1799 that any previous ccb-state is cleared and the current ccb-state is
 1800 empty. If SA_AIS_OK is returned then ccb-handle can continue to be
 1801 used and is in the same empty state as if it had just been initialized.                   
 1802 This is a blocking synchronous call.
 1803 
 1804 (2)
 1805 extern SaAisErrorT
 1806 saImmOmCcbValidate(SaImmCcbHandleT ccbHandle): /* in */
 1807 
 1808   Arguments     :  ccbHandle  - The ccbHandle for the ccb to be validated.
 1809 
 1810   Return Values :  SA_AIS_OK - Validation succeeded. Ccb has *not* been
 1811                                committed. No additional operations can now
 1812                                be added to the current ccb(id). This ccb(id)
 1813                                can now be aborted or an attempt can be made
 1814                                to commit it using saImmOmCcbApply().
 1815 
 1816                    SA_AIS_ERR_FAILED_OPERATION - Validation failed, the ccb
 1817                                can now only be aborted, using either 
 1818                                saImmOmCcbAbort() or saImmOmCcbFinalize().
 1819 
 1820                    SA_AIS_ERR_BAD_OPERATION - Not allowed inside a ccb augmentation.
 1821 
 1822                    SA_AIS_ERR_VERSION  (not using A.2.14)
 1823 
 1824                    Returncodes otherwise identical to saImmOmCcbApply().
 1825 
 1826 Performs only the validation part of a regular saImmOmCcbApply().
 1827 All involved OIs receive the completed-callback and perform their
 1828 validations. If validation fails (FAILED_OPERATION) then the ccb(id)
 1829 is in the same aborted state as if validation had failed during a
 1830 saImmOmCcbApply(). But if validation succeeds (SA_AIS_OK) then the
 1831 OM client has the option to either commit the CCB using saImmOmCcbApply()
 1832 or to abort the CCB using either saImmOmCcbFinalize() or saImmOmCcbAbort().
 1833 The OM client can not add new operations to a CCB that has been validated. 
 1834 The saImmOmCcbValidate call is suitable for clients that wish to perform
 1835 incremental validation. This can be achieved by validating a CCB, aborting
 1836 it then replaying the same operations (that validated), followed by adding
 1837 more operations. Replay is necessary because the OIs ccb-protocol can only
 1838 cope with one invocation of validation (completed callback) for any given
 1839 ccb-id. Another possible use is in integrating IMM transaction commit
 1840 with the commit of some other transaction handling system. By validating
 1841 without committing, the risk is significantly reduced that a final
 1842 commit/apply will fail. The invocation of saImmOmCcbApply after a 
 1843 successfull invocation of saImmOmCcbValidate can still fail/abort,
 1844 but only due to resource problems (such as server crash or filesystem
 1845 unavailability). 
 1846 
 1847 This is a blocking synchronous call.
 1848 
 1849 
 1850 Allow admin-operations directly targeting an implementer or applier (4.5)
 1851 ======================================================================
 1852 http://sourceforge.net/p/opensaf/tickets/799/
 1853 
 1854 
 1855 The existing IMM admin-operation API in OpenSAF:
 1856 
 1857     saImmOmAdminOperationInvoke_2()
 1858     saImmOmAdminOperationInvoke_o2()
 1859     saImmOmAdminOperationInvokeAsync_2()
 1860     SaImmOmAdminOperationInvokeCallbackT()
 1861     SaImmOmAdminOperationInvokeCallbackT_o2()
 1862 
 1863     SaImmOiAdminOperationCallbackT_2()
 1864     saImmOiAdminOPerationResult()
 1865     saImmOiAdminOperationResult_o2()
 1866 
 1867 allows OM clients to invoke procedural requests directed at an imm object.
 1868 The OM client needs to have an admin-owner handle associated with an admin-owner
 1869 name that matches the admin-owner for the object. The actual target/receiver of
 1870 the admin-operation is really the implementer/OI registered for the object or
 1871 class. The object that anchors the request is effectively a parameter in the
 1872 request to the OI.
 1873 
 1874 This enhancement (#799) does not add any new API. It extends the allowed scope
 1875 for the existing admin-operation API to cover the case where the target is an OI
 1876 directly. Instead of supplying the DN of an object as the address/target, an
 1877 implementer-name is provided as the address/target. This allows the OM client to
 1878 invoke admin-operations directly and explicitly on an OI. The implementer-name
 1879 is used as a proxy for the object-name in the admin-operation API.
 1880 
 1881 Because the extended scope does not require the addition of any new API, access
 1882 to this extension is not based on the handle version used. But the enhancement
 1883 is introduced in OpenSAF release 4.5 and thus only available in that release or
 1884 later releases. 
 1885 
 1886 The om-side admin-operation-invoke call takes an admin-owner-handle as argument.
 1887 The admin-owner handle needs to be an initialized handle and it must set the
 1888 admin-owner-name to be the *same* as the implementer-name. This may seem peculiar
 1889 and unnecessary. But it serves both as a confirmation that the OM client has not
 1890 made some mistake in using the new extended scope; and as a possible future
 1891 entry point for authorizing admin-operation access directly targeting the OI. 
 1892 
 1893 The main benefit of this extension of scope, is that it allows a simple way for
 1894 applications to set up cluster scoped remote procedure call communication, where the
 1895 request arives FEVS-synchronously at the target. FEVS is the cluster synchronized
 1896 messaging mechanism used by the Immsv (see earlier in this README).
 1897 
 1898 The receiver process only needs to register as an implementer or applier with IMM.
 1899 In many cases the receiving process has done this anyway for other reasons.
 1900 The sender only needs to know the name of the implementer/applier, the operation-id
 1901 and expected arguments for the call. For this extended scope usage, the OI does not
 1902 necessarily need to be the real implementer/OI for *any* imm object at all.
 1903 
 1904 Another use case is that it allows an om-client to communicate with (or probe) an
 1905 OI/applier as such. An OI may be the OI for many objects and classes. In some cases,
 1906 requests for information, or requests for adjustments in behavior of the OI/applier
 1907 are generic and not associated with any specific object. For example, statistics or
 1908 resource utilization data, last processed ccb, for the OI or Applier.
 1909 
 1910 Appliers are only listeners/trackers of ccb changes. An applier is never reachable
 1911 via any *regular* admin-op invoked via some object. This since the applier will never
 1912 be *the* OI for any object. But this enhancement allows an OM client to communicate
 1913 directly also with appliers. This can for example be usefull if a main-OI needs
 1914 an applier to read the before-ccb-commit versions of objects being changed in a CCB.
 1915 The main OI can achieve this by sending a synchrnous admin-op to that applier from
 1916 inside the completed callback for the main OI. Because the main-OI has not returned
 1917 from the completed callback, the apply of that CCB is suspended waiting on the reply
 1918 on the completed callback. Once the applier has read the before-image data, from
 1919 inside the admin-op callback, it replies on the admin-op towards the main OI. The
 1920 main OI can then finish the validation and reply to the immsv. Note that the main OI
 1921 is in this example using both the OI and the OM API concurrently.
 1922 
 1923 One note of warning. Care needs to be taken when using *synchronous* admin-operations 
 1924 not to create deadlock cycles between the involved processes. Any such deadlock will
 1925 be resolved by the timeout of the synchronous admin-ops, so the problem would be lack
 1926 of progress on some task, not a permanently hung set of processes.
 1927 
 1928 Support for long DNs in existing IMM APIs (4.5)
 1929 ===============================================
 1930 http://sourceforge.net/p/opensaf/tickets/886
 1931 http://sourceforge.net/p/opensaf/tickets/957/
 1932 This enhancement is part of the general enhancement for supporting long DNs in
 1933 OpenSAF tracked by ticket: #191. Ticket #886 adds support for long DNs to IMM.
 1934 
 1935 The long DN enhancement results possibility that the error code:
 1936 
 1937      SA_AIS_ERR_NAME_TOO_LONG 
 1938 
 1939 can be returned by the following APIs:
 1940 
 1941     saImmOmSearchInitialize_2
 1942     saImmOmSearchNext_2
 1943     saImmOmAccessorGet_2
 1944     saImmOmAdminOperationInvoke_2
 1945     saImmOmAdminOperationInvokeAsync_2
 1946     saImmOmAdminOperationInvoke_o2
 1947 
 1948 or be returned in the parameter 'error' by the following callbacks:
 1949 
 1950    SaImmOmAdminOperationInvokeCallbackT
 1951    SaImmOmAdminOperationInvokeCallbackT_o2
 1952 
 1953 This error will be returned if:
 1954 
 1955  - The OM request contains a long DN and the OI is not long DN capable.
 1956  - The reply (from imm-server on search, or from OI on admin-op) contains
 1957    a long DN and the OM-client is not long DN capable.
 1958 
 1959 
 1960 The library implementation of the existing IMM API has been addapted to the 
 1961 "tunneling" solution  provided by the OpenSAF generic patch for ticket #191.
 1962 See the document: 
 1963 
 1964         OpenSAF_Extensions_PR.odt 
 1965 
 1966 on how to use the SaNameT tunneling primitives, allowing DNs longer than 255 bytes 
 1967 to be passed via an SaNameT value.
 1968 
 1969 A new IMM API where all use of SaNameT is replaced by SaStringt and SaConstStringT
 1970 has been reviewed and accepted by the OpenSAF TC. But support for this new API
 1971 did not make it into OpenSAF 4.5. Instead it will be provided in OpenSAF 4.6.
 1972 This is tracked by ticket #643 (http://sourceforge.net/p/opensaf/tickets/643).
 1973 
 1974 An unofficial API similar to #643 has been provided in OpenSAF4.5  as part of
 1975 immutils (ticket #957). The immutils library uses 'char *' and 'const char *'
 1976 instead of SaStringT and SaConstStringT. The immutils library source code is 
 1977 available for copying under the directory:
 1978 
 1979   samples/immsv/immutils
 1980 
 1981 The additions to the immutils API encapsulates use of the SaNameT tunneling over
 1982 the IMMA lib. It can be seen as an example of how to use SaNameT tunneling, or it
 1983 can be copied and incorporated into an application library with the understanding
 1984 that it is part of, maintained, and versioned by each such application. 
 1985 
 1986 
 1987 Attribute 'longDnsAllowed' added to class 'OpensafImm' (4.5)
 1988 ===========================================================
 1989 http://sourceforge.net/p/opensaf/tickets/897
 1990 
 1991 This enhancement is part of the general effort for supporting long DNs in
 1992 OpenSAF tracked by ticket: #191 and adding support for long DNs to IMM tracked by
 1993 ticket: #886.
 1994 
 1995 A config attribute named 'longDnsAllowed' is added to the class 'OpensafImm' and
 1996 is available in the only instance of that class: 
 1997 
 1998    opensafImm=opensafImm,safApp=safImmService
 1999 
 2000 The default value is 0 with the intended meaning that long DNs are not allowed.
 2001 Any other value has the meaning that long DNs are allowed. The attribute is checked
 2002 as part of processing any IMM request that would imply the usage of a long DN. 
 2003 
 2004 Transitioning from true to false/0 for this attribute will only be allowed at a site
 2005 if the imm database at the site currently has no long DNs.
 2006 
 2007 An imm xml file containing the extended class definition for OpensafImm has been
 2008 added at:
 2009 
 2010         samples/immsv/OpensafImm_Upgrade_4.5.xml.
 2011 
 2012 That xml file should be used for upgrading the class when systems running older 
 2013 OpenSAF releases are upgraded to OpenSAF 4.5 and that system needs to be capable
 2014 of supporting long DNs. An initial start of a pristine OpenSAF 4.5 ssytem will
 2015 automatically install this new attribute. 
 2016 
 2017 
 2018 SAF RDN limitation of 64 characters made conditional on 'longDnsAllowed (4.5)
 2019 =============================================================================
 2020 http://sourceforge.net/p/opensaf/tickets/23/
 2021 
 2022 If support for long DNs is enabled (by setting the longDnsAllowed attribute to
 2023 a non zero value) then this will also disable the 64 byte limit on RDN length in
 2024 that system.
 2025 
 2026 
 2027 Add OI functionality for the IMM service to validate its own config data (4.5)
 2028 ==============================================================================
 2029 http://sourceforge.net/p/opensaf/tickets/934/
 2030 http://sourceforge.net/p/opensaf/tickets/951/
 2031 
 2032 Prior to OpenSAF 4.5 there was really no IMM OI implemented. The IMM service
 2033 has two configuration objects for managing the imm service itself:
 2034 
 2035         safRdn=immManagement,safApp=safImmService
 2036         opensafImm=opensafImm,safApp=safImmService
 2037 
 2038 The first one is the SAF defined configuration object for IMM where the only
 2039 used attribute is the config attribute:
 2040 
 2041         saImmRepositoryInit
 2042 
 2043 used for managing enablement of PBE. The second one is the OpenSAF defined
 2044 configuration object for IMM where the config attributes are:
 2045 
 2046         opensafImmSyncBatchSize
 2047         longDnsAllowed
 2048 
 2049 The IMM OI is normally "hosted" in the PBE. If the PBE is disabled, changes to these
 2050 objects are handled by the IMMND.
 2051 
 2052 
 2053 Allow schema change to add attribute default (4.5)
 2054 ==================================================
 2055 http://sourceforge.net/p/opensaf/tickets/895
 2056 
 2057 The restriction of not allowing a default value to be added to an attribute definition
 2058 that previously had no default, has been removed. Only new instances of the class will
 2059 get the default assigned when no value is provided for the attribute at object-create
 2060 time. Old instances with a null-value for the attribute will still have a null-value
 2061 for that attribute even after the schema change. However, a cluster restart will
 2062 cause all instances that used to have a null value to get the default value.
 2063 
 2064 A general rule for schema changes is that the immsv never alters the value of existing
 2065 attributes as part of a schema change. Doing so would require the OI to be notified,
 2066 which is not realistic since the shcema change is not performed as a CCB.
 2067 This new feature is only available after an upgrade to OpensAF 4.5 has completed.
 2068 
 2069 
 2070 Support for configurable OI callback timeout (4.5)
 2071 =================================================
 2072 http://sourceforge.net/p/opensaf/tickets/16/
 2073 
 2074 A new environment variable IMMA_OI_CALLBACK_TIMEOUT is recognized by the OI library.
 2075 In the same way as for the existing IMMA_SYNCR_TIMEOUT, this environment variable
 2076 is sampled by the SaImmOiInitialize function for initializing an oi-handle.
 2077 
 2078 The OI-handle will have that timeout associated with it for the rest of its lifetime,
 2079 for the server side monitoring of callbacks generated over the handle.
 2080 
 2081 The unit used for the value is seconds. 
 2082 
 2083 Different OI handles can have different callback timeouts, by changing the value of
 2084 IMMA_OI_CALLBACK_TIMEOUT prior to each saImmOiIntitialize.
 2085 
 2086 If the environment variable is not defined or is set to less than the default OI 
 2087 timeout, then the default OI timeout defined in the server will be used by the server
 2088 when monitoring callbacks to this OI. Currently the default OI callback timeout is 
 2089 defined to be 6 seconds. It is normally not recommended to set an OI callback timeout
 2090 to a value higher than the om/oi-client side timeout for synchronous downcalls.
 2091 
 2092 IMMA_OI_CALLBACK_TIMEOUT is not applicable to admin operations.
 2093 
 2094 
 2095 SaDoubleT and SaFloatT attributes handled correctly (4.5)
 2096 =========================================================
 2097 http://sourceforge.net/p/opensaf/tickets/53/
 2098 See ticket for details.
 2099 
 2100 
 2101 Auto relase of lingering search handles  (4.5)
 2102 ==============================================
 2103 http://sourceforge.net/p/opensaf/tickets/47/
 2104 See ticket for details.
 2105 
 2106 AdminOp on immsv for dumping resource utilization data (4.5)
 2107 ===========================================================
 2108 http://sourceforge.net/p/opensaf/tickets/35/
 2109 
 2110 This enhancement is mainly intended to assist the maintainers
 2111 of OpenSAF in troubleshouting resource/performance issues.
 2112 It is not intended to be an end-user utility.
 2113 
 2114 See: osaf/services/saf/immsv/README.RESOURCE_DISPLAY for details.
 2115 
 2116 #938 IMM Access control (4.5)
 2117 =============================
 2118 http://sourceforge.net/p/opensaf/tickets/938/
 2119 OpenSAF 4.5 adds the immsv enhancement of supporting access control of imma clients.
 2120 
 2121 See: osaf/services/saf/immsv/README.ACCESS_CONTROL for details.
 2122 
 2123 
 2124 Notes on upgrading from OpenSAF 4.[1,2,3,4] to OpenSAF (4.5)
 2125 ==========================================================
 2126 Several enhancements in OpenSAF4.5 add new message types or add new imm server states
 2127 for ccb handling (#798,  #799, #16, #895). During a rolling upgrade from an earlier OpenSAF
 2128 release to the 4.5 release there will be nodes executing the older release concurrently
 2129 with nodes executing OpenSAF 4.5. Nodes executing the earlier release will not
 2130 recognize new message types originating from nodes executing 4.5 and messages from
 2131 nodes executing the old release may interfere with the extended state machine for ccbs
 2132 in new 4.5 nodes.
 2133 
 2134 Because of this upgrade issue, the older more restrictive behavior is still enforced
 2135 in OpenSAF 4.5 unless a flag is toggled on in the opensafImmNostdFlags runtime
 2136 attribute in the object: opensafImm=opensafImm,safApp=safImmService.
 2137 The following is the shell command:
 2138 
 2139         immadm -o 1 -p opensafImmNostdFlags:SA_UINT32_T:16 \
 2140            opensafImm=opensafImm,safApp=safImmService
 2141 
 2142 This will set bit 5 of the 'opensafImmNostdFlags' runtime attribute inside the immsv.
 2143 Operation-id '1' invoked on the object:
 2144 
 2145  'opensafImm=opensafImm,safApp=safImmService'
 2146 
 2147 has the meaning of 'flags-ON'. Operation-id '2' has the meaning of 'flags-OFF'.
 2148 This flag (and possibly other relevant flags) needs to be toggled ON when the upgrade
 2149 to OpenSAF 4.5 has been successfully completed. This would be in some final step of
 2150 the upgrade. Any cluster start/restart of an OpenSAF4.5 system will always
 2151 automatically toggle on relevant flags. 
 2152 
 2153 In summary:
 2154 
 2155 Bit 1 controls schema (imm class) changes allowed or not (normally off/0).
 2156 Bit 2 controls OpenSAF4.1 protocols allowed or not (normally on/1).
 2157 Bit 3 controls OpenSAF4.3 protocols allowed or not (normally on/1).
 2158 Bit 4 controls 2PBE oneSafe2PBE, see 2PBE feature in OpenSAF4.4 above (normally off/0).
 2159 Bit 5 controls OpenSAF4.5 protocols allowed or not (normally on/1).
 2160 
 2161 Note also that enhancement #897 described above, added a new config attribute to
 2162 the class 'OpensafImm'. An imm xml file containing the extended class definition
 2163 for OpensafImm has been added at samples/immsv/OpensafImm_Upgrade_4.5.xml.
 2164 
 2165 
 2166 Immtools support for saImmOmCcbAbort() and saImmOmCcbValidate() (4.6)
 2167 =====================================================================
 2168 http://sourceforge.net/p/opensaf/tickets/1200
 2169 
 2170 Immutils was missing wrapper functions for saImmOmCcbValidate and saImmOmCcbAbort.
 2171 Immcfg needed a new option (--ccb-validate) for validating created/modified/deleted IMM data.
 2172 The option is only available in explicit commit mode.
 2173 
 2174 
 2175 Cached RTAs show latest cached value when OI is transiently detached (4.6)
 2176 ==========================================================================
 2177 http://sourceforge.net/p/opensaf/tickets/1156
 2178 
 2179 OM clients performing a read (iteration or accessor-get) that fetches a cached
 2180 runtime attribute, will not immediately see the attribute as empty when/if the
 2181 OI detaches. Instead for a period of grace for 6 seconds, the latest set value is shown.
 2182 This allows for failover or switchover or process restart of OI to occur without OM
 2183 clients seeing the "glitch" in the value of the cached runtime attribute. 
 2184 
 2185 
 2186 immdump: Support for dumping instances of selected classes (4.6)
 2187 ================================================================
 2188 http://sourceforge.net/p/opensaf/tickets/927
 2189 
 2190 The immdump tool by default dumps the entire imm database to a file
 2191 (either imm-xml format or imm-sqlite format). There is a need for
 2192 performing more selective dumping. This enhancement adds support
 2193 in immdump for accepting a list of classes in the argument list and to
 2194 selectively dump only the instances that belong to classes in that list.
 2195 
 2196 
 2197 OiRtObject operations with no implementer set should return BAD_OPERATION (4.6)
 2198 ================================================================================
 2199 http://sourceforge.net/p/opensaf/tickets/1064
 2200 
 2201 It used to be the case that if a process/OI:
 2202 
 2203 1) Initializes an IMM-OI handle.
 2204 2) Has not set any implementer-name, i.e. has not yet (?) become any identifiable OI.
 2205 3) Tries to use the oi-handle to create/delete/update runtime data.
 2206 
 2207 Then the resulting error code returned was ERR_BAD_HANDLE.
 2208 This is actually according to the SAF spec. But that error code is here both 
 2209 inconvenient and (we claim) a spec error.
 2210 
 2211 It is a spec error because ERR_BAD_HANDLE should at least mean that the handle as
 2212 such is bad, i.e. unusable for whatever reason. That is the literal meaning of the
 2213 error ERR_BAD_HANDLE. The correct handling of this error is to allocate a new handle,
 2214 or abandon whatever task was intending to use the handle.
 2215 
 2216 But in this error case discussed here the handle is actually not invalid and
 2217 can even be used after the above scenario happens to invoke saImmOiImplementerSet
 2218 to associate an implementer-name with the handle. The missing implementerSet is
 2219 an error of handle state, not an error of handle validity. The correct error code
 2220 for this case is instead declared to be ERR_BAD_OPERATION:
 2221 
 2222  ERR_BAD_OPERATION - The targeted object is not implemented by the invoking process.
 2223 
 2224 The SAF text for this case self contradicting:
 2225 
 2226   The handle immOiHandle is invalid, since it is corrupted, uninitialized,
 2227   has already been finalized, or is not associated with an implementer name.
 2228 
 2229 The last clause 'not associated with an implementer-name' is a VALID state
 2230 for an OI handle that has been initialized but not yet associated with an
 2231 implementer-name! That is in fact the state of the handle demanded by
 2232 saImmOiImplementerSet! So it is not the handle that is invalid, but the
 2233 state of the (valid) handle that is not (yet) appropriate for RT operations.
 2234 
 2235 The error code BAD_OPERATION is defined for similar cases for saImOiRtObjectUpdate
 2236 and saImmOiRtObjectDelete. Unfortunately SAF did not specify that error code for
 2237 saImmOiRtObjectCreate. So this will be a new error code for saImmOiRtObjectCreate.
 2238 
 2239 Fortunately there is not any backwards compatibility issue for this change and 
 2240 addition of error code. The error case is an interface-violation, by definition
 2241 a surprise for the user/application. So it is actually acceptable, even desirable
 2242 (fail fast) for the client process to crash when this happens, since the client
 2243 process definitely has a bug.
 2244 
 2245 An addition will be made to the "non compliance" section of the IMMSV_PR to make
 2246 this clear.
 2247 
 2248 This new behavior is benign and better than the old behavior. It will of course
 2249 be an unexpected error code for the application/user. But since this is an interface
 2250 violation (user/programmer error) the error case itself is unexpected.
 2251 
 2252 
 2253 Immnd timeout on CCB-OI upcall resets client handle (4.6)
 2254 ========================================================
 2255 http://sourceforge.net/p/opensaf/tickets/57
 2256 
 2257 Any IMM API synchronous downcall uses an imm-handle. For the duration of that downcall
 2258 the imm-handle will be blocked for usage by other threads. The blocking is handled
 2259 by rejecting any such overlapped use with an error. This is to prevent multiple threads
 2260 from overlapping use of an imm-handle, which is not allowed for the OpenSAF IMM.
 2261 If different threads have a concurrent need for imm access, they must allocate separate
 2262 imm-handles.
 2263 
 2264 The blocking of the handle is resolved by either:
 2265 
 2266     a) an explicit reply from the server for that request (the normal case).
 2267     b) timeout in the client library for that request (default is 10 seconds).
 2268     c) timeout in the server on processing delegated to an OI (default is 6 seconds).
 2269 
 2270 If case (b) happens, the handle will get unblocked on the client side. But subsequent
 2271 use by the client could still block on case (c) in the server unless the server side
 2272 has resolved the request, either by normal completion or timeout (c). Note that the
 2273 normal completion result would in that case not have reached the client.
 2274 
 2275 Thus the case of (c) can be resolved in the server either by the OI replying or by the
 2276 immnd-server time-out for continuations. The latter will also clean up continuations
 2277 for crashed or hung OIs.
 2278 
 2279 For admin-operation continuations and for search-request-continuations (pure RTA fetch)
 2280 this cleanup is already working. So if (c) occurred or (b) and (c) then a subsequent
 2281 new request would function normally for these requests (admin-operations and searches).
 2282 
 2283 But for CCB-operation requests, a timeout in the immnd server (case c) will simply abort
 2284 the CCB in the server and not clear the handle on the server side. This leaves the handle
 2285 effectively stale for the client. The next attempt by the client to use that handle will
 2286 result in ERR_BAD_HANDLE. The client then has to initialize a new handle. While that is
 2287 an expected error case, this error is actually not necessary in this case, but results
 2288 from a simplified design in the server. This is the issue that this enhancement resolves.
 2289 
 2290 This enhancement improves the cleanup of CCB-operation continuations at OI timeout.
 2291 The fix should also result in a ccb related error reaching the client, instead of the
 2292 ambiguous ERR_TIMEOUT. An OI timeout on a ccb callback will always abort the CCB in
 2293 the server, so any reply forwarded towards the OM CCB client should be an 
 2294 ERR_FAILED_OPERATION. Such a timeout triggered ccb abort will either be sent all the
 2295 way to the client, if the client-side timeout is longer than the server timeout.
 2296 Or (when the client has already timed out in the library) be discarded by MDS.
 2297 In either case, a subsequent attempt by the client to use the handle should work,
 2298 ï.e. not result in ERR_BAD_HANDLE. 
 2299 
 2300 The main points are (a) that the handle gets cleared in the server when
 2301 the continuation times out in the server; and (b) that the CCB OM client
 2302 receives an error reply when that om client has NOT also timed out.
 2303 
 2304 
 2305 Improve error diagnostics when PBE is misconfigured (4.6)
 2306 =========================================================
 2307 http://sourceforge.net/p/opensaf/tickets/1139
 2308 
 2309 Configuration mistakes such as omitting to change immnd.conf to allow PBE
 2310 tends to cause strange behavior that is not immediately apparent as to
 2311 its cause. Symptoms observed where: 
 2312 
 2313 One could attempt to enable PBE by setting  saImmRepositoryInit to 1:
 2314 
 2315      sudo immcfg -a saImmRepositoryInit=1 safRdn=immManagement,safApp=safImmService
 2316 
 2317 Yet the PBE process does not start and no clear error message was logged.
 2318 One could attempt to disable PBE by setting saImmRepositoryInit back to 2:
 2319 
 2320     sudo immcfg -a saImmRepositoryInit=2 safRdn=immManagement,safApp=safImmService
 2321     error - saImmOmCcbObjectModify_2 FAILED: SA_AIS_ERR_TRY_AGAIN (6)
 2322     error - immcfg command timed out (alarm)
 2323 
 2324 In the syslog only this was visible:
 2325 
 2326    Sep 26 09:15:27 NO Precheck of fevs message of type <33> failed with ERROR:18
 2327 
 2328 Error logging been improved and the imm server returns an error string message for the above
 2329 Ccb operation error cases. This should make troubleshooting this issue much faster and easier.
 2330 
 2331 
 2332 IMM API that replaces SaNameT with SaStringT and SA_IMM_ATTR_DN (4.6)
 2333 =====================================================================
 2334 http://sourceforge.net/p/opensaf/tickets/643
 2335 
 2336 See: osaf/services/saf/immsv/README.SASTRINGT_API for details.
 2337 
 2338 
 2339 Notes on upgrading from OpenSAF 4.[1,2,3,4,5] to OpenSAF (4.6)
 2340 ==============================================================
 2341 OpenSAF4.6 adds new message types that avoid using the SaNameT type (#969).
 2342 During a rolling upgrade from an earlier OpenSAF release to the 4.6 release there
 2343 will be nodes executing the older release concurrently with nodes executing OpenSAF 4.6.
 2344 Nodes executing the earlier release will not recognize new message types originating
 2345 from nodes executing 4.6 and messages from nodes executing the old release may interfere
 2346 with the new states in new 4.6 nodes.
 2347 
 2348 Because of this upgrade issue, the new message types added in OpenSAF 4.6 are not used
 2349 unless a flag is toggled on in the opensafImmNostdFlags runtime attribute in the object:
 2350 
 2351    opensafImm=opensafImm,safApp=safImmService.
 2352 
 2353 The following is the shell command:
 2354 
 2355         immadm -o 1 -p opensafImmNostdFlags:SA_UINT32_T:32 \
 2356            opensafImm=opensafImm,safApp=safImmService
 2357 
 2358 This will set bit 6 of the 'opensafImmNostdFlags' runtime attribute inside the immsv.
 2359 Operation-id '1' invoked on the object:
 2360 
 2361  'opensafImm=opensafImm,safApp=safImmService'
 2362 
 2363 has the meaning of 'flags-ON'. Operation-id '2' has the meaning of 'flags-OFF'.
 2364 This flag (and possibly other relevant flags) needs to be toggled ON when the upgrade
 2365 to OpenSAF 4.6 has been successfully completed. This would be in some final step of
 2366 the upgrade. Any cluster start/restart of an OpenSAF4.6 system will always
 2367 automatically toggle on relevant flags. 
 2368 
 2369 In summary:
 2370 
 2371 Bit 1 controls schema (imm class) changes allowed or not (normally off/0).
 2372 Bit 2 controls OpenSAF4.1 protocols allowed or not (normally on/1).
 2373 Bit 3 controls OpenSAF4.3 protocols allowed or not (normally on/1).
 2374 Bit 4 controls 2PBE oneSafe2PBE, see 2PBE feature in OpenSAF4.4 above (normally off/0).
 2375 Bit 5 controls OpenSAF4.5 protocols allowed or not (normally on/1).
 2376 Bit 6 controls OpenSAF4.6 protocols allowed or not (normally on/1).
 2377 
 2378 
 2379 Remove unnecessary detour of accessorGet into ImmModel::searchInitialize (4.7)
 2380 ==============================================================================
 2381 http://sourceforge.net/p/opensaf/tickets/674/
 2382 
 2383 A minor enhancement that refactored some code in ImmModell:accessorGet and 
 2384 ImmModel::searchInitialize.
 2385 
 2386 
 2387 immcfg should support ccbObjModify after ccbObjCreate in same ccb (4.7)
 2388 =======================================================================
 2389 http://sourceforge.net/p/opensaf/tickets/1283
 2390 
 2391 In "explicit commit mode" the immcfg command allows a user to build
 2392 up multi-operation CCBs. The IMM API in general allows a CCB to have
 2393 an object create operation later followed by an object modify operation
 2394 on the object that was created in the same CCB. But due to a limitation
 2395 in the implementation of the immcfg command/tool this sequence was not
 2396 supported by immcfg. This enhancement removed that limitation.
 2397 
 2398 
 2399 Abort non-critical CCBs when implementer is disconnected (4.7)
 2400 ==============================================================
 2401 http://sourceforge.net/p/opensaf/tickets/1391
 2402 
 2403 During the buildup of a CCB, i.e. a CCB where the OM client has successfully
 2404 added one or more operations to the CCB, but then the OM client lingers or
 2405 performs some other task not related to the open CCB; if an OI involved in
 2406 the open and idle CCB detaches, then the CCB is doomed to be aborted. Only
 2407 when the OM client invoked the next request was the abort of the CCB processed.
 2408 
 2409 One problem with this was that a restart of the detached OI would have to wait
 2410 for the CCB to get aborted before it could attach, which could mean waiting
 2411 indefinitely on some oblivious om-client to act.
 2412 
 2413 This enhancement fixes so that instead of having to wait for the OM client to
 2414 trigger the abort and cleanup of the CCB when it invokes the next operation,
 2415 the imm server can immediately process the abort as triggered by the OI detach.
 2416 
 2417 
 2418 Periodically audit the PBE imm.db file (4.7)
 2419 ============================================
 2420 http://sourceforge.net/p/opensaf/tickets/19/
 2421 
 2422 A first and very limited form of PBE file audit has been implemented in 
 2423 OpenSAF 4.7. This due to a serious incident with reference attributes that
 2424 have the NO_DANGLING flag set. See critical ticket #1377.
 2425 
 2426 In principle the audit function can and should be extended to run periodically
 2427 as a background job and to cover more consistency checks. In the lack of any
 2428 pro-active driver, it should at least be extended as a side effect of any future
 2429 actual incidents/cases with inconsistency in the persistent imm data that
 2430 actually does occur.
 2431 
 2432 
 2433 Don't check for pending fevs when only updating pure runtime attributes (4.7)
 2434 =============================================================================
 2435 http://sourceforge.net/p/opensaf/tickets/1445
 2436 
 2437 The IMMND server process has a flow control mechanism to prevent the
 2438 IMMNDs (one at each processor of the cluster) to overload the single active
 2439 IMMD at the active SC with fevs requests. The message type used when an OI
 2440 updates its runtime attributes is conditionally a fevs message. It will become
 2441 a fevs message for the normal case of the OI updating a cached runtime 
 2442 attribute. But when the OI is updating a pure (non cached) runtime attribute
 2443 as a side effect of a processor local OM read request on this attribute, then
 2444 the update of the attribute is actually only done locally and is not sent
 2445 over fevs. Despite this the update of a pure and local runtime attribute 
 2446 followed the same code path and ended up being pushed back with TRY_AGAIN
 2447 towards the OI, if fevs traffic was heavy. This push back (flow control) was
 2448 in this case totally unnecessary since this variant of the runtime attribute
 2449 update message would not be sent over fevs. This potential and unnecessary
 2450 delay of an update of a pure runtime attribute has been removed by this
 2451 enhancement.
 2452 
 2453 
 2454 Provide an admin-operation for aborting all non-critical CCBs (4.7)
 2455 ===================================================================
 2456 http://sourceforge.net/p/opensaf/tickets/1107
 2457 
 2458 There may arise situations where an open CCB that is not in critical,
 2459 i.e. has not entered the commit protocol yet, is blocking an involved
 2460 service/OI from performing some other task that is more urgent and more
 2461 important than completing that CCB. If the OM client is idling and the 
 2462 built up CCB is so far well formed (accepted by involved OIs) then the
 2463 OM client can linger indefinitely, particularly a human operator.
 2464 
 2465 A good example where a more urgent task may be blocked is the AMF, where
 2466 an si-swap will fail and cause the standby to reboot if it was involved
 2467 in an open CCB when the si-swap order was issued (see ticket #1105).
 2468 Ticket #1105 can be fixed by the AMF (active or standby) sending this
 2469 admin-operation directed at the IMM service requesting it to abort non
 2470 critical CCBs. The AMF can either use a synchronous admin-op or an 
 2471 asynchronous admin-op. After the admin-operation has been invoked the AMF 
 2472 should allow a few seconds for the CCB to get aborted and the AMF OI to
 2473 get the abort callback for the CCB. That should then clear the path for
 2474 the AMF standby to succeed with the si-swap. 
 2475 
 2476 The admin-operation for aborting non critical CCBs involves requesting the
 2477 operation id '202' directed at the IMM SF service object:
 2478 
 2479 	immadm -o 202 safRdn=immManagement,safApp=safImmService
 2480 
 2481 
 2482 Use error string to classify cause for aborted CCB.(4.7)
 2483 ========================================================
 2484 http://sourceforge.net/p/opensaf/tickets/744/
 2485 
 2486 For the case of a ccb-operation resulting in the return of:
 2487 
 2488     SA_AIS_ERR_FAILED_OPERATION
 2489 
 2490 which means the CCB has been aborted, there is a need for some clients
 2491 to discriminate between the two possible generic categories of abort:
 2492 
 2493 	Validation-abort.
 2494 or
 2495 	Resource-abort.
 2496 
 2497 A validation-abort is the result when the operations added by the OM-client to
 2498 the CCB constitute an *invalid* set of operations according to either
 2499 fundamental rules enforced by the IMM service or model specific rules enforced
 2500 by the involved Ois relative to the *current* state of the database.
 2501 
 2502 A resource-abort is the result if there is a resource problem somewhere in the 
 2503 system such that the CCB can not be committed or further processed at this time.
 2504 There are many kinds of resource aborts. A few examples would be:
 2505 
 2506       Ccb is aborted because it interferes with another operation in the system.
 2507       Ccb is aborted due to loss of contact with some involved OI.
 2508       Ccb is aborted due to loss of contact with PBE before apply request.
 2509       Ccb is aborted because an OI reports that it has insufficient resources.
 2510 
 2511 Note that SA_AIS_ERR_FAILED_OPERATION is the *only* error code with the meaning
 2512 that the CCB has been aborted. The fact that the CCB has been aborted is the
 2513 dominating issue/fact that must be understood and coped with by any and all
 2514 applications using the CCB interface. The further discrimination into validation
 2515 abort or resource abort is only relevant for *some* applications that are capable
 2516 of handling the abort differently depending on this discrimination.
 2517 
 2518 The fundamental logical difference between validation abort and resource abort
 2519 is that a validation abort is due to the CCB being incorrectly formed by the
 2520 OM-client; while a resource abort is due to problems in the server or OIs
 2521 making it "physically" impossible to successfully commit this CCB even though
 2522 it *may* be a correctly formed CCB.
 2523 
 2524 The possible actionable difference for the OM client between the two abort
 2525 categories is that a replay by the OM client of a resource aborted CCB could
 2526 succeed. This would be the case if the resource problem has been resolved. But
 2527 a replay by the client of a validation aborted CCB will never succeed, except
 2528 for some rare cases where another CCB was interleaved and altered the database
 2529 in such a way as to *make* the replay suddenly become a valid change relative
 2530 to a *new* state. This latter case should be so rare as to be ignored.
 2531 
 2532 So the bottom line is that some applications may wish to determine if an
 2533 aborted CCB that is replayed will have some chance of successfully comitting.
 2534 The application can do this by obtaining the abort category.
 2535 
 2536 This enhancement makes it possible for an application or end-user to obtain the
 2537 abort category. The abort category is prepended as a string prefix to one or more
 2538 error strings returned by saImmOmCcbGetErrorStrings().
 2539 
 2540 A validation aborted CCB will have an error string prefixed with:
 2541 
 2542              "IMM: Validation abort:"
 2543 
 2544 A resource aborted CCB will have an error string prefixed with:
 2545 
 2546               "IMM: Resource abort:"
 2547 
 2548 The prefix provides both a readable tag for human end-users and a standardized
 2549 prefix that may be string-matched in a program or script.
 2550 
 2551 Finally there are some subtle issues that should be understood by applications
 2552 intending to discriminate between resource abort and validation abort.
 2553 
 2554 a) A Ccb may be aborted for more than one reason. Thus a Ccb that is actually
 2555 not a valid Ccb (absolutely or relative to current IMM state) may run into a
 2556 resource problem during buildup and thus be aborted as a resource abort before
 2557 it had the chance to be evaluated for validity.
 2558 
 2559 b) There may be more than one OI involved in a Ccb. Some OIs may reply OK
 2560 on validation (completed callback), some OIs may reply with validation 
 2561 error, and some OIs may reply with resource error or default on timeout
 2562 resulting in resource error. The end result is that if there is one validation
 2563 error "vote" then that will dominate the resulting abort category.
 2564 
 2565 c) Validation success or failure often depends on the current state. i.e. 
 2566 the state of the set of involved objects *before* the attempted Ccb. Such a
 2567 state may be changed by some other client applying its own Ccb. So a Ccb
 2568 that is aborted with validation error, may later succeed due to the changed
 2569 prior state. But the assumption here is that the two Ccb clients are not
 2570 communicating with each other and so this possible case should be ignored
 2571 by the application. 
 2572 
 2573 d) There is never any general guarantee that a Ccb that is replayed after a
 2574 resource abort will "sooner or later" be validated successfully and commit.
 2575 Thus any loop that replays a Ccb on resource abort should still limit the
 2576 number of retries.
 2577 
 2578 e) All error strings generated by the IMM service are prefixed with "IMM:"
 2579 The prefix means that the OM client can recognize error strings generated by
 2580 the IMM service as distinct from error strings generated by an OI.
 2581 
 2582 f) For the case where there is no error string or no prefixed error string,
 2583 the application may assume that it is a resource abort. 
 2584 
 2585 g) If the error strings contain prefixes for both validationabort and resource
 2586 abort, then validation abort dominates.
 2587 
 2588 
 2589 Add attribute definition flag SA_IMM_ATTR_DEFAULT_REMOVED (4.7)
 2590 ===============================================================
 2591 http://sourceforge.net/p/opensaf/tickets/1471
 2592 
 2593 Support for removing a default value from an attribute definition in a class
 2594 definition has now been added. This kind of upgrade was not allowed previously
 2595 since it is inherently a case of non backwards compatibility and can cause
 2596 problems for legacy applications/users expecting and relying on the default.
 2597 That is after all the entire point of having a default.
 2598 
 2599 This enhancement removes the restriction of not allowing the removal of a
 2600 default value if this new flag is set. The effect of this flag is that
 2601 when/if an object of the class is created with no value assigned to the
 2602 attribute that used to have a default but no longer has a default; then a
 2603 trace message is generated noting that this attribute used to have a default
 2604 but no longer has a default and will in this case have no value.
 2605 
 2606 This is to assist users or troubleshooters if they get some form of problem
 2607 by the removal of the default. The trace message should speed up
 2608 troubleshooting and prevent the creation of unnecessary tickets or trouble
 2609 reports.
 2610 
 2611 
 2612 Sync data Mbcsv check pointing can be optimized (4.7)
 2613 =====================================================
 2614 http://sourceforge.net/p/opensaf/tickets/952/
 2615 
 2616 Imm sync messages can be large. They are sent over fevs. Fevs messages are
 2617 handled by the active IMMD by MDS broadcasting them to all IMMNDs. Before
 2618 doing the broadcast, the active IMMD uses the message based checkpoint 
 2619 service Mbcsv, to checkpoint each fevs message to the standby IMMD. This is
 2620 to secure that a failover or switchover of SC/IMMD will not cause any gap
 2621 in the fevs count for the fevs broadcasts. But if a failover actually happens
 2622 during a sync, then the new active IMMD will abort the sync. There is then
 2623 no point in ever re-broadcasting the contents of a sync message. The only
 2624 thing that needs to be re-broadcast is the fevs message header to close any
 2625 gap in the fevs count. This enhancement truncates the sync mesasages to only
 2626 contain the fevs header, before they are checkpointed to the standby IMMD.
 2627 This speeds up the sync and removes some communication load.
 2628 
 2629 
 2630 PBE: imm.db.XXXXXX temp files should managed in pbe subdirectory (4.7)
 2631 ======================================================================
 2632 http://sourceforge.net/p/opensaf/tickets/896/
 2633 
 2634 The PBE generates the imm.db file in what should be a local tmp directory.
 2635 This is set by the configuration variable IMMSV_PBE_TMP_DIR in the immnd.conf
 2636 configuration file. By default it is /tmp.
 2637 
 2638 This enhancement of PBE creates a sub-directory to IMMSV_PBE_TMP_DIR where all
 2639 temporary PBE files are created. This will reduce the risk of interference with
 2640 other services and applications sharing the tmp directory. It also facilitates
 2641 the safe cleanup of all such temporary files by a PBE that is restarted.
 2642 
 2643 
 2644 PBE: Detach of PBE should abort all non-critical and non-empty CCBs (4.7)
 2645 =========================================================================
 2646 http://sourceforge.net/p/opensaf/tickets/1261/
 2647 
 2648 This enhancement is related to defect ticket [#1260].
 2649 
 2650 If the PBE detaches while there are any active non-critical and non-empty CCBs,
 2651 then such CCBs should be ABORTED, i.e. prevented from being further processed.
 2652 
 2653 The abort must be done when the detach arrives over fevs. It must NOT be done
 2654 when the initial IMMND local PBE detach occurs, since that would make the ccb
 2655 state deviate locally.
 2656 
 2657 
 2658 Make it possible to run valgrind on osafimmnd when PBE is enabled (4.7)
 2659 =======================================================================
 2660 http://sourceforge.net/p/opensaf/tickets/1496/
 2661 
 2662 A minor enhancement removing an obstacle to executing the IMMND process under
 2663 valgrind when PBE is enabled.
 2664 
 2665 Notes on upgrading from OpenSAF 4.[1,2,3,4,5,6] to OpenSAF (4.7)
 2666 ================================================================
 2667 OpenSAF4.7 adds new attribute flag allowing the removal of a default value
 2668 definitions (#1471). During a rolling upgrade from an earlier OpenSAF release
 2669 to the 4.7 release there will be nodes executing the older release concurrently
 2670 with nodes executing OpenSAF 4.7. Nodes executing the earlier release will not
 2671 recognize the new attribute flag originating from nodes executing 4.7.
 2672 
 2673 Because of this upgrade issue, the new attribute flag added in OpenSAF 4.7 is 
 2674 not allowed unless a flag is toggled on in the opensafImmNostdFlags runtime
 2675 attribute in the object:
 2676 
 2677    opensafImm=opensafImm,safApp=safImmService.
 2678 
 2679 The following is the shell command:
 2680 
 2681         immadm -o 1 -p opensafImmNostdFlags:SA_UINT32_T:64 \
 2682            opensafImm=opensafImm,safApp=safImmService
 2683 
 2684 This will set bit 7 of the 'opensafImmNostdFlags' runtime attribute inside the immsv.
 2685 Operation-id '1' invoked on the object:
 2686 
 2687  'opensafImm=opensafImm,safApp=safImmService'
 2688 
 2689 has the meaning of 'flags-ON'. Operation-id '2' has the meaning of 'flags-OFF'.
 2690 This flag (and possibly other relevant flags) needs to be toggled ON when the upgrade
 2691 to OpenSAF 4.7 has been successfully completed. This would be in some final step of
 2692 the upgrade. Any cluster start/restart of an OpenSAF4.7 system will always
 2693 automatically toggle on relevant flags. 
 2694 
 2695 In summary:
 2696 
 2697 Bit 1 controls schema (imm class) changes allowed or not (normally off/0).
 2698 Bit 2 controls OpenSAF4.1 protocols allowed or not (normally on/1).
 2699 Bit 3 controls OpenSAF4.3 protocols allowed or not (normally on/1).
 2700 Bit 4 controls 2PBE oneSafe2PBE, see 2PBE feature in OpenSAF4.4 above (normally off/0).
 2701 Bit 5 controls OpenSAF4.5 protocols allowed or not (normally on/1).
 2702 Bit 6 controls OpenSAF4.6 protocols allowed or not (normally on/1).
 2703 Bit 7 controls OpenSAF4.7 protocols allowed or not (normally on/1).
 2704 
 2705 
 2706 Safe-read (transactional read) (5.0)
 2707 =============================================
 2708 http://sourceforge.net/p/opensaf/tickets/48/
 2709 
 2710 Adds support for an OM CCB client to read-access a config object transactionally.
 2711 The API works exactly the same way as saImmOmAccessorGet, except that
 2712 
 2713    a) The API takes a SaImmCcbHandleT instead of a SaImmAccessorHandleT
 2714 
 2715    b) The values returned for the objects config attributes are from a version
 2716       of the object consistent with the CCB/transaction. This means either the
 2717       latest applied version or a newer version created in the same CCB but
 2718       not yet applied. 
 2719 
 2720    c) Access to an object that has been deleted in the same CCB but not applied
 2721       is rejected with ERR_NOT_EXIST. 
 2722 
 2723    d) Access to an object that has been created in the same CCB but not applied
 2724       is allowed, providing the latest version of the config attributes in that CCB.
 2725 
 2726    e) Safe read is not allowed using a runtime object as target.
 2727 
 2728 Runtime attributes residing in a config object are handled exactly the same as for
 2729 saImmOmAccessorGet. The reason a safe-read call is not allowed on a runtime *object*
 2730 is that a runtime object *only* contains runtime attributes. Performing a safe-read
 2731 on a runtime object makes no sense. 
 2732 
 2733 
 2734 saImmOmCcbObjectRead(SaImmCcbHandleT ccbHandle, /* in */
 2735                      SaConstStringT objectName, /* in */
 2736                      const SaImmAttrNameT *attributeNames, /* in */
 2737                      SaImmAttrValuesT_2 ***attributes); /* out */
 2738 
 2739 
 2740 Return Values :   SA_AIS_ERR_BUSY - The object targeted by the request is already
 2741                   the target of conflicting/exclusive operation in another CCB.
 2742                   The conflicting operation would be a modify or delete.
 2743 
 2744                   SA_AIS_ERR_INVALID_PARAM - The objectName parameter identifies a
 2745                   runtime object.
 2746 
 2747                   Returncodes otherwise identical to saImmOmAccessorGet.
 2748 
 2749 
 2750 SC Absence (5.0)
 2751 ===============================================
 2752 https://sourceforge.net/p/opensaf/tickets/1625
 2753 
 2754 SC absence enhancement has the goal of increasing OpenSAFs resilience in 
 2755 the face of both active and standby SC going down. Both SCs being absent 
 2756 implies that all OpenSAF director services are absent indefinitely, until 
 2757 an SC is re-established. Prior to the SC absence enhancement, departure 
 2758 of both SCs always resulted in a cluster restart. With the SC Absence 
 2759 enhancement, payloads continue to provide reduced and limited service 
 2760 until an SC-active is re-established. For the IMM service, SC absence is 
 2761 configured by uncommenting the environment variable IMMSV_SC_ABSENCE_ALLOWED.
 2762 
 2763 export IMMSV_SC_ABSENCE_ALLOWED=900
 2764 
 2765 Value of environment variable IMMSV_SC_ABSENCE_ALLOWED is stored in 
 2766 scAbsenceAllowed attribute in opensafImm=opensafImm,safApp=safImmService
 2767 object. This value is used by AMF to restart the cluster if SC absence
 2768 is longer than the value in scAbsenceAllowed attribute in seconds.
 2769 
 2770 Support for SC absence is incompatible with 2PBE. If both are configured
 2771 then 2PBE will win and the SC absence feature will be ignored. An error
 2772 message is printed in this case to the syslog at startup. Allowing SC absence
 2773 feature is a configuration choice impacting all OpenSAF services, not just 
 2774 the IMM service. If it is to be allowed then it is not sufficient to only 
 2775 configure the IMM service for SC absence. The level of service that is 
 2776 provided during absent SC depends on the particular service. In the case 
 2777 of the IMM service, the service provided during SC absence is in essence 
 2778 only the reading of config data.
 2779 
 2780 See: osaf/services/saf/immsv/README.SC_ABSENCE for details.
 2781 
 2782 
 2783 Add attribute definition flag SA_IMM_ATTR_STRONG_DEFAULT (5.0)
 2784 ===============================================================
 2785 https://sourceforge.net/p/opensaf/tickets/1425
 2786 
 2787 The default value is only effective for object creation, and not later
 2788 in the life cycle of the object. This makes the default attribute value
 2789 mechanism weaker than some users would like.
 2790 
 2791 When there is an attempt to set empty value to or to delete all values from
 2792 attributes with SA_IMM_ATTR_STRONG_DEFAULT flag, default value will be set.
 2793 The SaImmAttrModificationT_2 will be changed to SA_IMM_ATTR_VALUES_REPLACE
 2794 with default value and sent to OI.
 2795 
 2796 SA_IMM_ATTR_STRONG_DEFAULT flag can only be set on an attribute definition
 2797 that includes default value.
 2798 
 2799 SA_IMM_ATTR_STRONG_DEFAULT flag is supported when OM API registers with
 2800 version A.02.17 or higher.
 2801 
 2802 
 2803 Canonicalize attributes presented by saImmOiCcbObjectModifyCallbackT_2 (5.0)
 2804 ============================================================================
 2805 https://sourceforge.net/p/opensaf/tickets/801
 2806 https://sourceforge.net/p/opensaf/tickets/1651
 2807 
 2808 For OIs or appliers receiving ccb-callbacks, the handling of the modify-callback
 2809 can be particularly complex. This is due to the modify callback being defined
 2810 by SAF as faithfully presenting the same parameters as the originating om-client 
 2811 parameters provided over the om-api.
 2812 
 2813 The object modify operation allows for modifications expressed as changes
 2814 relative to the current state of attributes. This is of course acted on by
 2815 the imm-server resulting in an after-image for the modified object.
 2816 
 2817 The callback to the OI or applier is currently of the same form, in general
 2818 requiring the OI/applier to:
 2819  a) know the current state of the attributes of the object or do an for 
 2820     appliers unsafe read 
 2821  b) to correctly compute the transformation as defined by the imm-spec. 
 2822 Particularly the later is asking quite a lot for the average OI/applier 
 2823 implementer.
 2824 
 2825 Only for the "special applier" is the modify callback currently canonicalized 
 2826 to contain simply the replacement values, i.e. the after operation image.
 2827 
 2828 To simplify the task for OI's and appliers in handling modify callbacks,
 2829 the modify-callback instead provides the after-modify-operation-image of 
 2830 the attributes. The new callback format containing just the replacement value
 2831 resulting from the operation must be logically equivalent to the set of all
 2832 before-image and operation variants resulting in this replacement value.
 2833 
 2834 Canonicalize attributes are presented by saImmOiCcbObjectModifyCallbackT_2
 2835 when IMM OI registers with version A.02.17 or higher.
 2836 
 2837 
 2838 Class and object applier set are on local node (5.0)
 2839 ====================================================
 2840 https://sourceforge.net/p/opensaf/tickets/1535
 2841 
 2842 The enhancement makes object and class applier set info consistent per node.
 2843 This means that applier set will bind objects and classes that have already 
 2844 been bound on the local node by the same applier name.
 2845 
 2846 Applier name continues to be shared between nodes, while object and class
 2847 applier bindings are kept on the originating node.
 2848 
 2849 IMM Hardcoded limits as config parameters (5.1)
 2850 ===============================================================
 2851 http://sourceforge.net/p/opensaf/tickets/195/
 2852 
 2853 The enhancemnet makes IMM limits configurable. The limits like maxClasses,
 2854 maxImplementers, maxAdminowners and maxCcbs are allowed as configurable.
 2855 The limits are added as attributes to the imm object 
 2856 "opensafImm=opensafImm,safApp=safImmService".
 2857 
 2858 The default values configured for these attributes are :
 2859 
 2860 IMMSV_MAX_CLASSES 1000
 2861 IMMSV_MAX_IMPLEMENTERS 3000
 2862 IMMSV_MAX_ADMINOWNERS 2000
 2863 IMMSV_MAX_CCBS 10000
 2864 
 2865 The user can not configure the values less than the default values.
 2866 
 2867 The value for maxCcbs are the active ccbs present in the cluster.
 2868 The terminated CCBs will be removed from the cluster after 5 minutes.
 2869 If the number of terminated CCBs are greater than "2 * maxCcbs" then 
 2870 CCBs will be removed from the cluster after 2 minutes.
 2871 
 2872 Veteran max timeout is made configurable in case of SC Absence (5.1)
 2873 ===================================================================
 2874 http://sourceforge.net/p/opensaf/tickets/1974/
 2875 
 2876 IMMSV_SC_ABSENCE_VETERAN_MAX_WAIT is the IMMD waits for veteran IMMNDs after 
 2877 SC absence. If IMMD processes the intro messages from new IMMNDs before the intro
 2878 messages from veteran IMMNDs, IMM service will start to load from file instead of 
 2879 syncing data from veteran nodes. The IMMSV_SC_ABSENCE_VETERAN_MAX_WAIT will 
 2880 prevent that from happening.
 2881 
 2882 Note that during a fresh cluster startup, IMMD also waits for veteran IMMNDs.
 2883 Setting this value to large will slow down the cluster startup. This timeout is 
 2884 only used with SC absence enabled. When SC absence is disabled, IMMD will not wait 
 2885 for the veteran IMMNDs.
 2886 
 2887 The IMMSV_SC_ABSENCE_VETERAN_MAX_WAIT is present in immd.conf.
 2888 The default value (when the environment variable is not set) is 3 seconds.
 2889 
 2890 export IMMSV_SC_ABSENCE_VETERAN_MAX_WAIT=3
 2891 
 2892 immcfg allows admin-owner in explicit commit mode (5.1)
 2893 ===============================================================
 2894 https://sourceforge.net/p/opensaf/tickets/1383/
 2895 
 2896 immcfg in explicit commit mode, will not be able to set adminOwnerName.
 2897 This enhancement removes the restriction.
 2898 
 2899 immcfg -o adminOwnerName.
 2900 
 2901 --ccb-apply and --ccb-abort will finalize the admin owner.
 2902 if -o option is not specified default admin-owner is considered.
 2903 
 2904 immadm can explicitly set ReleaseOnFinalize (5.1)
 2905 =======================================================
 2906 https://sourceforge.net/p/opensaf/tickets/1937/
 2907 
 2908 with this enhancement immadm introduces a new flag '-r' for setting ROF (Release On Finalize) flag.
 2909 The default value for ROF (when -r option is not specified) is SA_FALSE.
 2910 
 2911 Notes on upgrading to OpenSAF 5.1
 2912 ================================================================
 2913 OpenSAF5.1 makes the IMM attributes as configurable (#195).
 2914 During a rolling upgrade from an earlier OpenSAF release to the 5.1 
 2915 release there will be nodes executing the older release concurrently.
 2916 Nodes executing the earlier release will not recognize the new attributes 
 2917 introduced in imm config object from nodes executing 5.0.
 2918 
 2919 Because of this upgrade issue, the new attribute flag added in OpenSAF 5.1 is
 2920 not allowed unless a flag is toggled on in the opensafImmNostdFlags runtime
 2921 attribute in the object:
 2922 
 2923    opensafImm=opensafImm,safApp=safImmService.
 2924 
 2925 The following shell command must be used once all nodes are upgraded :
 2926 
 2927         immadm -o 1 -p opensafImmNostdFlags:SA_UINT32_T:256 \
 2928            opensafImm=opensafImm,safApp=safImmService
 2929 
 2930 This will set bit 9 of the 'opensafImmNostdFlags' runtime attribute inside the immsv.
 2931 Operation-id '1' invoked on the object:
 2932 
 2933  'opensafImm=opensafImm,safApp=safImmService'
 2934 
 2935 Integrate IMM service with CLM (5.2)
 2936 =======================================================
 2937 https://sourceforge.net/p/opensaf/tickets/1640/
 2938 
 2939 with this enhancement IMM is integrated with CLM.
 2940 
 2941 The IMM agent has to register with A.02.18 to integrate with CLM and receive 
 2942 CLM memebership information of the node. 
 2943 
 2944 The IMMND registers with CLMS and receives notification about the node leaves 
 2945 and joins the CLM membership. The CLM membership information will be sent to 
 2946 IMM agents which registers with A.02.18.
 2947 
 2948 IMMD is not yet integrated, as the IMMND has to services to clients with lesser versions.
 2949 IMMND has to be UP even if the CLM node is down. The IMMD integration has to be done along 
 2950 with Enhanced cluster management(#439).
 2951 
 2952 In case of CLM node lock, and HEADLESS is triggered after CLM node lock, then CLM locked 
 2953 node is nullified. The node is considered as HEADLESS and all other agent functions 
 2954 supported in HEADLESS are supported.
 2955 
 2956 Notes on upgrading to OpenSAF 5.17.11
 2957 =====================================
 2958 OpenSAF 5.17.11 adds new attribute flag allowing the removal of disconnected
 2959 appliers (#2579). During a rolling upgrade from an earlier OpenSAF release
 2960 to the 5.17.11 release there will be nodes executing the older release concurrently
 2961 with nodes executing OpenSAF 5.17.11. Nodes executing the earlier release will not
 2962 recognize the new attribute flag originating from nodes executing 5.17.11.
 2963 
 2964 Because of this upgrade issue, the new attribute flag added in OpenSAF 5.17.11 is
 2965 not allowed unless a flag is toggled on in the opensafImmNostdFlags runtime
 2966 attribute in the object:
 2967 
 2968    opensafImm=opensafImm,safApp=safImmService.
 2969 
 2970 The following is the shell command:
 2971 
 2972         immadm -o 1 -p opensafImmNostdFlags:SA_UINT32_T:1024 \
 2973            opensafImm=opensafImm,safApp=safImmService
 2974 
 2975 This will set bit 10 of the 'opensafImmNostdFlags' runtime attribute inside the immsv.
 2976 Operation-id '1' invoked on the object:
 2977 
 2978  'opensafImm=opensafImm,safApp=safImmService'
 2979 
 2980 has the meaning of 'flags-ON'. Operation-id '2' has the meaning of 'flags-OFF'.
 2981 This flag (and possibly other relevant flags) needs to be toggled ON when the upgrade
 2982 to OpenSAF 5.17.11 has been successfully completed. This would be in some final step of
 2983 the upgrade. Any cluster start/restart of an OpenSAF 5.17.11 system will always
 2984 automatically toggle on relevant flags.
 2985 
 2986 An imm xml file containing the extended class definition for OpensafImm has
 2987 been added at samples/immsv/OpensafImm_Upgrade_5.17.11.xml.
 2988 
 2989 In summary:
 2990 
 2991 Bit 1 controls schema (imm class) changes allowed or not (normally off/0).
 2992 Bit 2 controls OpenSAF4.1 protocols allowed or not (normally on/1).
 2993 Bit 3 controls OpenSAF4.3 protocols allowed or not (normally on/1).
 2994 Bit 4 controls 2PBE oneSafe2PBE, see 2PBE feature in OpenSAF4.4 above (normally off/0).
 2995 Bit 5 controls OpenSAF4.5 protocols allowed or not (normally on/1).
 2996 Bit 6 controls OpenSAF4.6 protocols allowed or not (normally on/1).
 2997 Bit 7 controls OpenSAF4.7 protocols allowed or not (normally on/1).
 2998 Bit 8 controls OpenSAF5.0 protocols allowed or not (normally on/1).
 2999 Bit 9 controls OpenSAF5.1 protocols allowed or not (normally on/1).
 3000 Bit 10 controls OpenSAF5.17.11 protocols allowed or not (normally on/1).
 3001 
 3002 Removal of disconnected applier (5.17.11)
 3003 =================================================
 3004 Once an implementer is created, it resides in the system to the cluster
 3005 restart. There is no way to remove implementers from the system. Applier is
 3006 a special type of implementers and the same feature applies to appliers.
 3007 
 3008 Applier names usually contain the node name where they are created to avoid
 3009 the collision with the same appliers on other nodes. In systems where new
 3010 nodes come with new names every time they join the cluster, this can be
 3011 a problem and the limit of 3000 implementer and appliers can be reached.
 3012 
 3013 The new feature is introduced in 5.17.11 which will handle the time for
 3014 keeping disconnected appliers in the system. When the time expires,
 3015 disconnected appliers will be removed from the system.
 3016 
 3017 The applier timeout is configurable, and it is set in the new attribute
 3018 minApplierTimeout in IMM object. If minApplierTimeout attribute is set to 0,
 3019 the removal of disconnected appliers is disabled. The unit in counting
 3020 the timeout is in seconds.
 3021 
 3022 With enabling the removal of disconnected appliers, there might be a risk that
 3023 some applications in the cluster rely on reattaching appliers. This will not
 3024 work if an application tries to reattach appliers after the applier timeout
 3025 expires.
 3026 
 3027 To be possible to use this new feature, bit 10 must be set in
 3028 opensafImmNostdFlags attribute in IMM object.
 3029 
 3030 
 3031 Provide an admin-operation for re-generating backend database from one in RAM
 3032 =============================================================================
 3033 https://sourceforge.net/p/opensaf/tickets/2940/
 3034 
 3035 After split-brain recovery, there is possibility of having inconsistencies
 3036 between IMM data model in memory held by IMMND and one in the back-end
 3037 database (sqlite).
 3038 
 3039 That could happen as we might have 02 active IMMDs, 02 IMMND coordinators
 3040 and more than one PBE processes accessing a shared pbe database during
 3041 split-brain. Change to such database from one therefore might not get
 3042 noticed by the other.
 3043 
 3044 By using this admin operation ID and targeting to IMM, IMM will regenerate the
 3045 back-end database from one in memory to keep them both consistent.
 3046 
 3047 immadm -o 303 safRdn=immManagement,safApp=safImmService
 3048 
 3049 Return try-again on requests that imply fs-access when fs is configured
 3050 to be administratively unavailable.
 3051 ===================================================================
 3052 https://sourceforge.net/p/opensaf/tickets/3019
 3053 
 3054 When underlying file system is unresponsive to pbe write request, all IMM
 3055 write requests that need their changes to be persistent such as apply of ccb or
 3056 updates to persistent runtime attributes or creation/deletion of classes
 3057 or creation/deletion of persistent runtime objects likely gets SA_AIS_ERR_TIMEOUT.
 3058 
 3059 In case of putting the pbe database on a network file server, there is possibility
 3060 that network file server is not responding to FS client and that worse situation
 3061 may last for minutes.
 3062 
 3063 This enhancement introduces two administrative operations to let user inform IMM
 3064 about the availability of the file system. If the server (IMMND) detects the logical
 3065 unavailability of file system through this variable, such write requests will get
 3066 the more honest code SA_AIS_ERR_TRY_AGAIN rather than SA_AIS_ERR_TIMEOUT.
 3067 
 3068 Operation ID 400 is used to inform IMM that the file system is unavailable:
 3069  immadm -o 400 safRdn=immManagement,safApp=safImmService
 3070 
 3071 and operation ID 401 is used to inform IMM the file system is back:
 3072  immadm -o 401 safRdn=immManagement,safApp=safImmService
 3073 
 3074 Besides, a new runtime IMM attribute, saImmFileSystemStatus, is added to
 3075 SaImmMngt class; the value shows the current logical status of the file system:
 3076 saImmFileSystemStatus = 0 means the file system is unavailable and
 3077 saImmFileSystemStatus = 1 means the file system is available.
 3078 
 3079 Fetching the value of that attribute, other services or applications that
 3080 have activities to read/write data from/to the file system may benefit as well.
 3081 
 3082 Note that, to use this new feature, bit 11 must be set in opensafImmNostdFlags.
 3083 
 3084 The following is the shell command to set the 11st bit:
 3085  immadm -o 1 -p opensafImmNostdFlags:SA_UINT32_T:1024 \
 3086 		opensafImm=opensafImm,safApp=safImmService
 3087 
 3088 In summary:
 3089  Bit 1 controls schema (imm class) changes allowed or not (normally off/0).
 3090  Bit 2 controls OpenSAF4.1 protocols allowed or not (normally on/1).
 3091  Bit 3 controls OpenSAF4.3 protocols allowed or not (normally on/1).
 3092  Bit 4 controls 2PBE oneSafe2PBE, see 2PBE feature in OpenSAF4.4 above (normally off/0).
 3093  Bit 5 controls OpenSAF4.5 protocols allowed or not (normally on/1).
 3094  Bit 6 controls OpenSAF4.6 protocols allowed or not (normally on/1).
 3095  Bit 7 controls OpenSAF4.7 protocols allowed or not (normally on/1).
 3096  Bit 8 controls OpenSAF5.0 protocols allowed or not (normally on/1).
 3097  Bit 9 controls OpenSAF5.1 protocols allowed or not (normally on/1).
 3098  Bit 10 controls OpenSAF5.17.11 protocols allowed or not (normally on/1).
 3099  Bit 11 controls OpenSAF5.19.07 protocols allowed or not (normally on/1).
 3100 ===================================================================
 3101 https://sourceforge.net/p/opensaf/tickets/3268/
 3102 
 3103 In a large cluster (e.g, 24 nodes), the components have very often been
 3104 timeout in the single step upgrade. To prevent timeout in this case,
 3105 Operator need to increase the MDS wait time for all clients in cluster.
 3106 Technically the IMMA_SYNC_TIMEOUT environment variable can be exported with
 3107 appropriate value in each components on every nodes. However, the cluster
 3108 has several nodes, each of which has hundreds of components.
 3109 
 3110 This enhancement introduce a configurable attribute `saImmSyncrTimeout`
 3111 to `SaImmMngt` class. This attribute represents the current MDS wait
 3112 time of IMM clients. It has 0 (disable this feature) by default and
 3113 a valid value is within the range [NCS_SAF_MIN_ACCEPT_TIME(10) - INT64_MAX].
 3114 The unit of value is 10 milliseconds.
 3115 
 3116 IMMND will distribute new timeout to all IMM clients when there is a change.
 3117 When deleting value of attribute `saImmSyncrTimeout`, it will go back
 3118 to default value (0) then force all IMM clients restore to old value.
 3119 
 3120 ----------------------------------------
 3121 DEPENDENCIES
 3122 ============
 3123 immsv depends of the following other services:
 3124 - NID
 3125 - MBCSV
 3126 - MDS
 3127 - BASE
 3128 - AMF
 3129 - logtrace
 3130 - libxml2
 3131 - sqlite3
 3132 
 3133 
 3134 DATA STRUCTURES
 3135 
 3136 The most important data structures for the IMMD process type are:
 3137 
 3138  IMMD_CB:               The "control block" for the IMMD, a singleton
 3139  IMMD_IMMND_INFO_NODE:  Information about one IMMND
 3140  IMMD_CB.immnd_tree:    Collection of IMMD_IMMND_INFO_NODEs
 3141 
 3142 
 3143 The most important data structures for the IMMND process type are:
 3144 
 3145  IMMND_CB:                              The "control block" for the IMMND, a
 3146                                         singleton
 3147  IMMND_IMM_CLIENT_NODE:                 Information about one client connection
 3148  IMMND_IMM_CLIENT_NODE.client_info_db:  Collection of IMMND_IMM_CLIENT_NODEs
 3149  IMMND_IMM_CLIENT_NDOE.immModel:        Pointer to the core Imm model,
 3150                                         a singleton. The ImmModel code is found
 3151                                         in ImmModel.[hh|cc]
 3152 
 3153 The most important data structures for the IMMA library are:
 3154 
 3155  IMMA_CB:                       The "control block" for the IMMA, a singleton
 3156  IMMA_CLIENT_NODE:              Information about one OM or OI connection
 3157  IMMA_CB.client_tree:           Collection of IMMA_CLIENT_NODEs
 3158  IMMA_ADMIN_OWNER_NODE:         Information about one admin-owner registration
 3159  IMMA_CB.admin_owner_tree:      Collection of IMMA_ADMIN_OWNER_NODEs
 3160  IMMA_CCB_NODE:                 Information about one initialised CCB
 3161  IMMA_CB.ccb_tree:              Collection of IMMA_CCB_NODEs
 3162  IMMA_SEARCH_NODE:              Information about one initialised search
 3163  IMMA_CB.search_tree:           Collection of IMMA_SEARCH_NODEs
 3164  IMMA_CONTINUATION_RECORD:      Information about one invoked asynchronous
 3165                                 admin-op
 3166  IMMA_CB.imma_continuations:    Collection of IMMA_CONTINUATION_RECORDs
 3167 
 3168 Common message types and message layouts are found under
 3169 osaf/libs/common/immsv/include in the files immsv_evt.h and immsv_evt_model.h
 3170 
 3171 
 3172 STATE MACHINES
 3173 
 3174 There are two state machines that control important aspects of the IMMND
 3175 behaviour.
 3176 
 3177 The ImmNodeState in immModel.cc governs the accessibility of the data model of
 3178 the IMM at each node.
 3179 
 3180     typedef enum {
 3181 
 3182         IMM_NODE_UNKNOWN = 0,         /*Initial state */
 3183         IMM_NODE_LOADING = 1,         /* Participating in a cluster restart */
 3184         IMM_NODE_FULLY_AVAILABLE = 2, /* Normal fully available state */
 3185         IMM_NODE_ISOLATED = 3,        /* Trying to join an established cluster*/
 3186         IMM_NODE_W_AVAILABLE = 4,     /* We are being synced, No model reads
 3187                                          allowed */
 3188         IMM_NODE_R_AVAILABLE = 5      /* Write locked while other nodes are
 3189                                          being synced. No model writes allowed*/
 3190     } ImmNodeState;
 3191 
 3192   Transitions:
 3193 
 3194     IMM_NODE_UNKNOWN -(StartLoading)-> IMM_NODE_LOADING
 3195     IMM_NODE_LOADING -(LoadingSuccess)->IMM_NODE_FULLY_AVAILABLE
 3196     IMM_NODE_UNKNOWN -(MissedLoading)-> IMM_NODE_ISOLATED
 3197     IMM_NODE_ISOLATED-(StartSync)->IMM_NODE_W_AVAILABLE
 3198     IMM_NODE_W_AVAILABLE-(SyncSuccess)->IMM_NODE_FULLY_AVAILABLE
 3199     IMM_NODE_FULLY_AVAILABLE->(StartSync)->IMM_NODE_R_AVAILABLE
 3200     IMM_NODE_R_AVAILABLE->(SyncTerminate)->IMM_NODE_FULLY_AVAILABLE
 3201 
 3202 
 3203 The IMMND_SERVER_STATE in immnd_cb.h and immnd_proc.c governs the progress of
 3204 cluster level tasks. In particular the behaviour of the coordinator, and
 3205 non-coordinators during loading and sync.
 3206 
 3207     typedef enum immnd_server_state {
 3208 
 3209         IMM_SERVER_UNKNOWN         = 0, /* Not allowed. Not used */
 3210         IMM_SERVER_ANONYMOUS       = 1, /* Initial state */
 3211         IMM_SERVER_CLUSTER_WAITING = 2, /* Waiting for expected #nodes */
 3212         IMM_SERVER_LOADING_PENDING = 3, /* Waiting for loading to start */
 3213         IMM_SERVER_LOADING_SERVER  = 4, /* Coord is executing loading */
 3214         IMM_SERVER_LOADING_CLIENT  = 5, /* Not coord, receive loading */
 3215         IMM_SERVER_SYNC_PENDING    = 6, /* Not coord, load impossible, wait for
 3216                                            sync*/
 3217         IMM_SERVER_SYNC_CLIENT     = 7, /* Not coord, this node is being
 3218                                            synced */
 3219         IMM_SERVER_SYNC_SERVER     = 8, /* Coord, executing sync */
 3220         IMM_SERVER_DUMP            = 9, /* Not used */
 3221         IMM_SERVER_READY           = 10 /* Coord & Not coord. Normal idle
 3222                                            state.*/
 3223 
 3224     }IMMND_SERVER_STATE;
 3225 
 3226   Transitions:
 3227 
 3228     IMM_SERVER_ANONYMOUS-(IMMD is up, intro sent)->IMM_SERVER_CLUSTER_WAITING
 3229     IMM_SERVER_CLUSTER_WAITING-(requisite nodes, timeout)->
 3230                                                       IMM_SERVER_LOADING_PENDING
 3231     IMM_SERVER_LOADING_PENDING-(coord and loading possible)->
 3232                                                        IMM_SERVER_LOADING_SERVER
 3233     IMM_SERVER_LOADING_PENDING-(non coord and loading possible)->
 3234                                                        IMM_SERVER_LOADING_CLIENT
 3235     IMM_SERVER_LOADING_PENDING-(non coord and loading impossible)->
 3236                                                          IMM_SERVER_SYNC_PENDING
 3237     IMM_SERVER_LOADING_SERVER-(loading succeeded)->IMM_SERVER_READY
 3238     IMM_SERVER_LOADING_CLIENT-(loading succeeded)->IMM_SERVER_READY
 3239     IMM_SERVER_SYNC_PENDING-(sync request accepted)->IMM_SERVER_SYNC_CLIENT
 3240     IMM_SERVER_READY-(coord and accept sync request)->IMM_SERVER_SYNC_SERVER
 3241     IMM_SERVER_SYNC_CLIENT-(sync succeeded)->IMM_SERVER_READY
 3242     IMM_SERVER_SYNC_SERVER-(sync terminated)->IMM_SERVER_READY
 3243 
 3244 
 3245 OTHER IMPLEMENTATION POINTS
 3246 
 3247 There is an epoch-count maintained by the IMMD. The epoch count is incremented
 3248 cluster-wide by the coordinator after important events such as: loading
 3249 completed, sync completed and dump completed. The epoch count is stored
 3250 persistently in every dump so that a re-load using that dump will be starting
 3251 from the saved epoch.
 3252 
 3253 The central data model of the IMMND is implemented in C++ and uses STL (Standard
 3254 Template Library) collections to represent the data model.
 3255 Other parts of IMMND are implemented in C.
 3256 
 3257 The immload and immdump programs are implemented in C++. These programs
 3258 communicate with the IMMSv using the IMM-OM API. They use libxml2 to parse and
 3259 generate the imm.xml format.
 3260 
 3261 The IMMA library is implemented in C.
 3262 
 3263 The IMMD is implemented in C.
 3264 
 3265 
 3266 
 3267 CONFIGURATION
 3268 
 3269 See OpenSAF_IMMSv_PR.
 3270 
 3271 COMMAND LINE INTERFACE
 3272 
 3273 See OpenSAF_IMMSv_PR.
 3274 
 3275 DEBUG
 3276 
 3277 To enable/disable immd/immnd trace in a running system, send signal USR2 to the
 3278 immd/immnd process. Each signal toggles the trace.
 3279 Trace default is disabled.
 3280 
 3281 Traces are written to the file configured in immd.conf and immnd.conf.
 3282 Traces are always stored in $PKGLOGDIR directory and the directory component
 3283 of the path name (if any) is ignored.
 3284 
 3285 To enable traces from the very start, uncomment:
 3286 
 3287         #args="--tracemask=0xffffffff"
 3288 
 3289 in immd.conf/immnd.conf and restart the cluster.
 3290 
 3291 Errors, warnings and notice level messages are logged to the syslog.
 3292 
 3293 To enable traces in the IMM library, export the variable IMMA_TRACE_PATHNAME
 3294 with a valid pathname before starting the application using the IMM library.
 3295 
 3296 For example:
 3297 
 3298 $ export IMMA_TRACE_PATHNAME=imm.trace
 3299 $ ./immomtest
 3300 $ cat $pkglogdir/imm.trace
 3301 
 3302 
 3303 It is also possible to trace slave processes forked by the IMMND.
 3304 This would be processes for loading, sync and dump/pbe.
 3305 To enable such trace uncomment:
 3306 
 3307 #export IMMSV_TRACE_PATHNAME=osafimmnd
 3308 
 3309 It is recommended to use osaflog command as it takes care of flushing
 3310 unwritten trace messages from memory to disk, as well as concatenating
 3311 the pieces that may have resulted from log rotation of the trace stream.
 3312 
 3313 TEST
 3314 
 3315 Currently a "tetware like" test suite can be found in 'tests/immsv/'.
 3316 See tests/immsv/README for instructions.
 3317 
 3318 
 3319 TODO
 3320 
 3321 - Revisit MDS reliability, implement FEVS re-send by IMMD.
 3322 - Implement version handling for MDS messages.
 3323 - Revisit escalation problem.
 3324 - Cleanup "TODO" comments
 3325 - Revisit trace statements, cleanup
 3326 See:
 3327 	 http://devel.opensaf.org/report
 3328 
 3329 
 3330 CONTRIBUTORS/MAINTAINERS
 3331 
 3332 Anders Bjornerstedt <Anders.Bjornerstedt@ericsson.com>
 3333 Hans Feldt <Hans.Feldt@ericsson.com>
 3334 Peter Strand <peter.strand@ericsson.com>
 3335 Mahesh Alla <mahesh.valla@oracle.com>
 3336 Neelakanta Reddy <reddy.neelakanta@oracle.com>
 3337 Zoran Millinkovic <zoran.milinkovic@ericsson.com>
 3338 
 3339 The IMM service OpenSAF framework was originally cloned from the OpenSAF
 3340 check-point service.