"Fossies" - the Fresh Open Source Software Archive

Member "openssl-1.0.2q/crypto/engine/README" (20 Nov 2018, 13957 Bytes) of package /linux/misc/openssl-1.0.2q.tar.gz:


As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file. See also the last Fossies "Diffs" side-by-side code changes report for "README": 1.1.0g_vs_1.1.1-pre2.

    1 Notes: 2001-09-24
    2 -----------------
    3 
    4 This "description" (if one chooses to call it that) needed some major updating
    5 so here goes. This update addresses a change being made at the same time to
    6 OpenSSL, and it pretty much completely restructures the underlying mechanics of
    7 the "ENGINE" code. So it serves a double purpose of being a "ENGINE internals
    8 for masochists" document *and* a rather extensive commit log message. (I'd get
    9 lynched for sticking all this in CHANGES or the commit mails :-).
   10 
   11 ENGINE_TABLE underlies this restructuring, as described in the internal header
   12 "eng_int.h", implemented in eng_table.c, and used in each of the "class" files;
   13 tb_rsa.c, tb_dsa.c, etc.
   14 
   15 However, "EVP_CIPHER" underlies the motivation and design of ENGINE_TABLE so
   16 I'll mention a bit about that first. EVP_CIPHER (and most of this applies
   17 equally to EVP_MD for digests) is both a "method" and a algorithm/mode
   18 identifier that, in the current API, "lingers". These cipher description +
   19 implementation structures can be defined or obtained directly by applications,
   20 or can be loaded "en masse" into EVP storage so that they can be catalogued and
   21 searched in various ways, ie. two ways of encrypting with the "des_cbc"
   22 algorithm/mode pair are;
   23 
   24 (i) directly;
   25      const EVP_CIPHER *cipher = EVP_des_cbc();
   26      EVP_EncryptInit(&ctx, cipher, key, iv);
   27      [ ... use EVP_EncryptUpdate() and EVP_EncryptFinal() ...]
   28 
   29 (ii) indirectly; 
   30      OpenSSL_add_all_ciphers();
   31      cipher = EVP_get_cipherbyname("des_cbc");
   32      EVP_EncryptInit(&ctx, cipher, key, iv);
   33      [ ... etc ... ]
   34 
   35 The latter is more generally used because it also allows ciphers/digests to be
   36 looked up based on other identifiers which can be useful for automatic cipher
   37 selection, eg. in SSL/TLS, or by user-controllable configuration.
   38 
   39 The important point about this is that EVP_CIPHER definitions and structures are
   40 passed around with impunity and there is no safe way, without requiring massive
   41 rewrites of many applications, to assume that EVP_CIPHERs can be reference
   42 counted. One an EVP_CIPHER is exposed to the caller, neither it nor anything it
   43 comes from can "safely" be destroyed. Unless of course the way of getting to
   44 such ciphers is via entirely distinct API calls that didn't exist before.
   45 However existing API usage cannot be made to understand when an EVP_CIPHER
   46 pointer, that has been passed to the caller, is no longer being used.
   47 
   48 The other problem with the existing API w.r.t. to hooking EVP_CIPHER support
   49 into ENGINE is storage - the OBJ_NAME-based storage used by EVP to register
   50 ciphers simultaneously registers cipher *types* and cipher *implementations* -
   51 they are effectively the same thing, an "EVP_CIPHER" pointer. The problem with
   52 hooking in ENGINEs is that multiple ENGINEs may implement the same ciphers. The
   53 solution is necessarily that ENGINE-provided ciphers simply are not registered,
   54 stored, or exposed to the caller in the same manner as existing ciphers. This is
   55 especially necessary considering the fact ENGINE uses reference counts to allow
   56 for cleanup, modularity, and DSO support - yet EVP_CIPHERs, as exposed to
   57 callers in the current API, support no such controls.
   58 
   59 Another sticking point for integrating cipher support into ENGINE is linkage.
   60 Already there is a problem with the way ENGINE supports RSA, DSA, etc whereby
   61 they are available *because* they're part of a giant ENGINE called "openssl".
   62 Ie. all implementations *have* to come from an ENGINE, but we get round that by
   63 having a giant ENGINE with all the software support encapsulated. This creates
   64 linker hassles if nothing else - linking a 1-line application that calls 2 basic
   65 RSA functions (eg. "RSA_free(RSA_new());") will result in large quantities of
   66 ENGINE code being linked in *and* because of that DSA, DH, and RAND also. If we
   67 continue with this approach for EVP_CIPHER support (even if it *was* possible)
   68 we would lose our ability to link selectively by selectively loading certain
   69 implementations of certain functionality. Touching any part of any kind of
   70 crypto would result in massive static linkage of everything else. So the
   71 solution is to change the way ENGINE feeds existing "classes", ie. how the
   72 hooking to ENGINE works from RSA, DSA, DH, RAND, as well as adding new hooking
   73 for EVP_CIPHER, and EVP_MD.
   74 
   75 The way this is now being done is by mostly reverting back to how things used to
   76 work prior to ENGINE :-). Ie. RSA now has a "RSA_METHOD" pointer again - this
   77 was previously replaced by an "ENGINE" pointer and all RSA code that required
   78 the RSA_METHOD would call ENGINE_get_RSA() each time on its ENGINE handle to
   79 temporarily get and use the ENGINE's RSA implementation. Apart from being more
   80 efficient, switching back to each RSA having an RSA_METHOD pointer also allows
   81 us to conceivably operate with *no* ENGINE. As we'll see, this removes any need
   82 for a fallback ENGINE that encapsulates default implementations - we can simply
   83 have our RSA structure pointing its RSA_METHOD pointer to the software
   84 implementation and have its ENGINE pointer set to NULL.
   85 
   86 A look at the EVP_CIPHER hooking is most explanatory, the RSA, DSA (etc) cases
   87 turn out to be degenerate forms of the same thing. The EVP storage of ciphers,
   88 and the existing EVP API functions that return "software" implementations and
   89 descriptions remain untouched. However, the storage takes more meaning in terms
   90 of "cipher description" and less meaning in terms of "implementation". When an
   91 EVP_CIPHER_CTX is actually initialised with an EVP_CIPHER method and is about to
   92 begin en/decryption, the hooking to ENGINE comes into play. What happens is that
   93 cipher-specific ENGINE code is asked for an ENGINE pointer (a functional
   94 reference) for any ENGINE that is registered to perform the algo/mode that the
   95 provided EVP_CIPHER structure represents. Under normal circumstances, that
   96 ENGINE code will return NULL because no ENGINEs will have had any cipher
   97 implementations *registered*. As such, a NULL ENGINE pointer is stored in the
   98 EVP_CIPHER_CTX context, and the EVP_CIPHER structure is left hooked into the
   99 context and so is used as the implementation. Pretty much how things work now
  100 except we'd have a redundant ENGINE pointer set to NULL and doing nothing.
  101 
  102 Conversely, if an ENGINE *has* been registered to perform the algorithm/mode
  103 combination represented by the provided EVP_CIPHER, then a functional reference
  104 to that ENGINE will be returned to the EVP_CIPHER_CTX during initialisation.
  105 That functional reference will be stored in the context (and released on
  106 cleanup) - and having that reference provides a *safe* way to use an EVP_CIPHER
  107 definition that is private to the ENGINE. Ie. the EVP_CIPHER provided by the
  108 application will actually be replaced by an EVP_CIPHER from the registered
  109 ENGINE - it will support the same algorithm/mode as the original but will be a
  110 completely different implementation. Because this EVP_CIPHER isn't stored in the
  111 EVP storage, nor is it returned to applications from traditional API functions,
  112 there is no associated problem with it not having reference counts. And of
  113 course, when one of these "private" cipher implementations is hooked into
  114 EVP_CIPHER_CTX, it is done whilst the EVP_CIPHER_CTX holds a functional
  115 reference to the ENGINE that owns it, thus the use of the ENGINE's EVP_CIPHER is
  116 safe.
  117 
  118 The "cipher-specific ENGINE code" I mentioned is implemented in tb_cipher.c but
  119 in essence it is simply an instantiation of "ENGINE_TABLE" code for use by
  120 EVP_CIPHER code. tb_digest.c is virtually identical but, of course, it is for
  121 use by EVP_MD code. Ditto for tb_rsa.c, tb_dsa.c, etc. These instantiations of
  122 ENGINE_TABLE essentially provide linker-separation of the classes so that even
  123 if ENGINEs implement *all* possible algorithms, an application using only
  124 EVP_CIPHER code will link at most code relating to EVP_CIPHER, tb_cipher.c, core
  125 ENGINE code that is independant of class, and of course the ENGINE
  126 implementation that the application loaded. It will *not* however link any
  127 class-specific ENGINE code for digests, RSA, etc nor will it bleed over into
  128 other APIs, such as the RSA/DSA/etc library code.
  129 
  130 ENGINE_TABLE is a little more complicated than may seem necessary but this is
  131 mostly to avoid a lot of "init()"-thrashing on ENGINEs (that may have to load
  132 DSOs, and other expensive setup that shouldn't be thrashed unnecessarily) *and*
  133 to duplicate "default" behaviour. Basically an ENGINE_TABLE instantiation, for
  134 example tb_cipher.c, implements a hash-table keyed by integer "nid" values.
  135 These nids provide the uniquenness of an algorithm/mode - and each nid will hash
  136 to a potentially NULL "ENGINE_PILE". An ENGINE_PILE is essentially a list of
  137 pointers to ENGINEs that implement that particular 'nid'. Each "pile" uses some
  138 caching tricks such that requests on that 'nid' will be cached and all future
  139 requests will return immediately (well, at least with minimal operation) unless
  140 a change is made to the pile, eg. perhaps an ENGINE was unloaded. The reason is
  141 that an application could have support for 10 ENGINEs statically linked
  142 in, and the machine in question may not have any of the hardware those 10
  143 ENGINEs support. If each of those ENGINEs has a "des_cbc" implementation, we
  144 want to avoid every EVP_CIPHER_CTX setup from trying (and failing) to initialise
  145 each of those 10 ENGINEs. Instead, the first such request will try to do that
  146 and will either return (and cache) a NULL ENGINE pointer or will return a
  147 functional reference to the first that successfully initialised. In the latter
  148 case it will also cache an extra functional reference to the ENGINE as a
  149 "default" for that 'nid'. The caching is acknowledged by a 'uptodate' variable
  150 that is unset only if un/registration takes place on that pile. Ie. if
  151 implementations of "des_cbc" are added or removed. This behaviour can be
  152 tweaked; the ENGINE_TABLE_FLAG_NOINIT value can be passed to
  153 ENGINE_set_table_flags(), in which case the only ENGINEs that tb_cipher.c will
  154 try to initialise from the "pile" will be those that are already initialised
  155 (ie. it's simply an increment of the functional reference count, and no real
  156 "initialisation" will take place).
  157 
  158 RSA, DSA, DH, and RAND all have their own ENGINE_TABLE code as well, and the
  159 difference is that they all use an implicit 'nid' of 1. Whereas EVP_CIPHERs are
  160 actually qualitatively different depending on 'nid' (the "des_cbc" EVP_CIPHER is
  161 not an interoperable implementation of "aes_256_cbc"), RSA_METHODs are
  162 necessarily interoperable and don't have different flavours, only different
  163 implementations. In other words, the ENGINE_TABLE for RSA will either be empty,
  164 or will have a single ENGING_PILE hashed to by the 'nid' 1 and that pile
  165 represents ENGINEs that implement the single "type" of RSA there is.
  166 
  167 Cleanup - the registration and unregistration may pose questions about how
  168 cleanup works with the ENGINE_PILE doing all this caching nonsense (ie. when the
  169 application or EVP_CIPHER code releases its last reference to an ENGINE, the
  170 ENGINE_PILE code may still have references and thus those ENGINEs will stay
  171 hooked in forever). The way this is handled is via "unregistration". With these
  172 new ENGINE changes, an abstract ENGINE can be loaded and initialised, but that
  173 is an algorithm-agnostic process. Even if initialised, it will not have
  174 registered any of its implementations (to do so would link all class "table"
  175 code despite the fact the application may use only ciphers, for example). This
  176 is deliberately a distinct step. Moreover, registration and unregistration has
  177 nothing to do with whether an ENGINE is *functional* or not (ie. you can even
  178 register an ENGINE and its implementations without it being operational, you may
  179 not even have the drivers to make it operate). What actually happens with
  180 respect to cleanup is managed inside eng_lib.c with the "engine_cleanup_***"
  181 functions. These functions are internal-only and each part of ENGINE code that
  182 could require cleanup will, upon performing its first allocation, register a
  183 callback with the "engine_cleanup" code. The other part of this that makes it
  184 tick is that the ENGINE_TABLE instantiations (tb_***.c) use NULL as their
  185 initialised state. So if RSA code asks for an ENGINE and no ENGINE has
  186 registered an implementation, the code will simply return NULL and the tb_rsa.c
  187 state will be unchanged. Thus, no cleanup is required unless registration takes
  188 place. ENGINE_cleanup() will simply iterate across a list of registered cleanup
  189 callbacks calling each in turn, and will then internally delete its own storage
  190 (a STACK). When a cleanup callback is next registered (eg. if the cleanup() is
  191 part of a gracefull restart and the application wants to cleanup all state then
  192 start again), the internal STACK storage will be freshly allocated. This is much
  193 the same as the situation in the ENGINE_TABLE instantiations ... NULL is the
  194 initialised state, so only modification operations (not queries) will cause that
  195 code to have to register a cleanup.
  196 
  197 What else? The bignum callbacks and associated ENGINE functions have been
  198 removed for two obvious reasons; (i) there was no way to generalise them to the
  199 mechanism now used by RSA/DSA/..., because there's no such thing as a BIGNUM
  200 method, and (ii) because of (i), there was no meaningful way for library or
  201 application code to automatically hook and use ENGINE supplied bignum functions
  202 anyway. Also, ENGINE_cpy() has been removed (although an internal-only version
  203 exists) - the idea of providing an ENGINE_cpy() function probably wasn't a good
  204 one and now certainly doesn't make sense in any generalised way. Some of the
  205 RSA, DSA, DH, and RAND functions that were fiddled during the original ENGINE
  206 changes have now, as a consequence, been reverted back. This is because the
  207 hooking of ENGINE is now automatic (and passive, it can interally use a NULL
  208 ENGINE pointer to simply ignore ENGINE from then on).
  209 
  210 Hell, that should be enough for now ... comments welcome: geoff@openssl.org
  211