"Fossies" - the Fresh Open Source Software Archive

Member "freeipa-4.8.8/doc/guide/guide.org" (15 Jun 2020, 53029 Bytes) of package /linux/misc/freeipa-4.8.8.tar.gz:

As a special service "Fossies" has tried to format the requested text file into HTML format (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file.

    1 #+OPTIONS: ^:{}
    2 #+EMAIL: abokovoy@redhat.com
    3 #+AUTHOR: Alexander Bokovoy
    4 #+STYLE: <style type="text/css">
    5 #+STYLE:  pre {
    6 #+STYLE:     border: 1pt solid #000000;
    7 #+STYLE:     background-color: #404040;
    8 #+STYLE:     color: white;
    9 #+STYLE:   }
   10 #+STYLE: .src {width: 940px;}
   11 #+STYLE: dt {width: 400px; margin 25px auto;}
   12 #+STYLE: dd {width: 940px;}
   13 #+STYLE: p {text-align:justify;}
   14 #+STYLE: body {width: 960px;
   15 #+STYLE:       margin: 0 auto;
   16 #+STYLE:      }
   17 #+STYLE: div#content {margin: 0 10px 0 10px;
   18 #+STYLE:          display: inline;
   19 #+STYLE:          float: left;
   20 #+STYLE:          width: 940px;
   21 #+STYLE:          overflow: hidden;}
   22 #+STYLE: </style>
   23 Extending FreeIPA
   24 * Introduction
   25 FreeIPA is an integrated security information management solution. There is a common
   26 framework written in Python to command LDAP server provided by a 389-ds project, certificate
   27 services of a Dogtag project, and a MIT Kerberos server, as well as configuring various other
   28 services typically used to maintain integrity of an enterprise environment, like DNS and
   29 time management (NTP). The framework is written in Python, runs at a server side, and
   30 provides access via command line tools or web-based user interface.
   32 As core parts of the framework are implemented as pluggable modules, it is possible to
   33 extend FreeIPA on multiple levels. This document attempts to present general ideas and
   34 ways to make use of most of extensibility points in FreeIPA.
   36 For information management solutions extensibility could mean multiple things. Information
   37 objects that are managed could be extended themselves or new objects could be added. New
   38 operations on existing objects might become needed or certain aspects of an object should
   39 be hidden in a specific environment. All these tasks may require quite different approaches
   40 to implement.
   42 Following chapters will cover high-level design of FreeIPA and dive into details of its core
   43 framework. Knowledge of Python programming language basics is required. Understanding
   44 LDAP concepts is desirable, though it is not required for simple
   45 extensions as FreeIPA attempts to provide sufficient mapping of LDAP concepts onto less
   46 complex structures and Python objects, lowering a barrier to fine tune FreeIPA for
   47 the specific use cases.
   48 * High level design
   49 FreeIPA core is written in Python programming language. The data is stored in LDAP
   50 database, and client-server paradigm is used for managing it. A FreeIPA server instance
   51 runs its own LDAP database, provided by 389-ds project (formerly Fedora Directory
   52 Server). A single instance of LDAP database corresponds to the single FreeIPA
   53 domain. Access to all information stored in the database is provided via FreeIPA server
   54 core which is run as a simple WSGI application which uses XML-RPC and JSON to exchange
   55 requests with its own clients.
   57 Multiple replicas of the FreeIPA instance can be created on different servers, they are
   58 managed with the help of replication mechanisms of 389-ds directory server.
   60 As LDAP database is used for data storage, LDAP's Access Control Model is used to provide
   61 privilege separation and Kerberos tickets are used to pass-through assertion of
   62 authenticity. As Kerberos server is using the same LDAP database instance, use of Kerberos
   63 tickets allows to perform operations against the database on the server if a client is
   64 capable to forward such tickets via communication channels selected for the operation.
   66 When FreeIPA client connects to FreeIPA server, a Kerberos ticket is forwarded
   67 to the server and operations against LDAP database are performed under identity
   68 authenticated when the ticket was issued. As LDAP database also uses Kerberos to establish
   69 identity of a client, Access Control Information attributes can be used to limit what
   70 entries could be accessed and what operations could be performed.
   72 The approach allows to delegate operations from a FreeIPA client to the FreeIPA server
   73 and in general gives FreeIPA server ability to interact with any Kerberos-aware service on
   74 behalf of the client. It also allows to keep FreeIPA client side implementation relatively
   75 light-weight: all it needs to do is to be able to forward Kerberos ticket, process XML-RPC or
   76 JSON, and present resulting responses to the user.
   78 Besides run-time core, FreeIPA includes few configuration tools. These tools
   79 are split between server and client. Server-side tools are used when an instance of
   80 FreeIPA server is set up and configured, while client-side tools are used to configure client
   81 systems. While the server tools are used to configure LDAP database, put proper schema
   82 definitions in use, create Kerberos domain, Certificate Authority and configure all
   83 corresponding services, client side is more limited to configure PAM/NSS modules to work
   84 against FreeIPA server, and make sure that appropriate information about the client host
   85 is recorded in FreeIPA databases.
   86 * Core plug-in framework
   87 FreeIPA core defines few fundamentals. These are managed objects, their properties, and
   88 methods to apply actions to the objects. Methods, in turn, are commands that are
   89 associated with a specific object. Additionally, there are commands that do not have
   90 directly associated objects and may perform actions over few of those. Objects are stored
   91 using data store represented by a back end, and one of most useful back ends is LDAP store
   92 back end.
   94 Altogether, set of =Object=, =Method=, =Command=, and =Backend= instances
   95 represent application programming interface, API, of FreeIPA core framework.
   97 In Python programming language object oriented support is implemented using a fairly
   98 simple concept that allows to modify instances in place, extending or removing their
   99 properties and methods. While this concept is highly useful, in security-oriented
  100 frameworks ability to lock down and trace origins of changes is also important. FreeIPA core
  101 attempts to implement locking down feature by artificially making instances of foundation
  102 classes read-only after their initialization has happened. If an attempt to modify object
  103 happens after it was locked down, an exception is thrown. There are many classes
  104 following this pattern.
  106 For example, =ipalib.frontend.Command= class is derived from =ipalib.frontend.HasParam= class
  107 that derives from =ipalib.plugable.Plugin= class which, in turn, is derived from
  108 =ipalib.base.ReadOnly= class.
  110 As result, every command has typed parameters and can dynamically be added to the
  111 framework. At the same time, one cannot modify the properties of the command accidentally
  112 once it is instantiated. This protects from modifications and enforces true nature of the
  113 commands: they cannot have state that is carried over across multiple calls to the same
  114 command unless the state is changing globally the whole environment around.
  116 Environment also holds information about the context of execution. The /context/ is
  117 important part of the FreeIPA framework as it also defines which methods of
  118 the command instance are called in order to perform action. /Context/ in itself is defined
  119 by the /environment/ which gives means to catch and store certain information about execution.
  120 As with commands themselves, once instantiated, environment cannot be changed.
  122 By default, for primary FreeIPA use, there are three major contexts defined: server,
  123 client, and installer/updates.
  125 - /server context/ :: plugins are registered and communicate with clients via XML-RPC and JSON
  126      listeners. They validate any arguments and options defined and then execute whatever
  127      action they supposed to perform
  128 - /client context/ :: plugins are used to validate any arguments and options they take and
  129      then forward the request to the FreeIPA server.
  130 - /installer context/, /updates context/ :: plugins specific to installation and update
  131      are loaded and registered. This context can be used to extend possible operations
  132      during set up of FreeIPA server.
  134 A user may define any context they want. FreeIPA names server context as '~server~'. When
  135 using the ~ipa~ command line tool the context is '~cli~'. Server installation tools, in
  136 particular, '~ipa-ldap-updater~', use special '~updates~' context to load specialized
  137 plugins useful during update of the installed FreeIPA server.
  139 Because these utilities use the same framework they will do the same validation, set default
  140 values, and perform other basic actions in all contexts. This can help to save a
  141 round-trip when testing for invalid data. However, for client-server communication, the
  142 server is always authoritative and can re-define what the client has sent.
  144 ** Name space
  145 FreeIPA has one special type of read-only objects: =NameSpace=. =NameSpace= class gives an
  146 ordered, immutable mapping object whose values can also be accessed as attributes. A
  147 =NameSpace= instance is constructed from iterable providing its members, which are simply
  148 arbitrary objects with =name= attribute. This attribute must conform to two following
  149 rules:
  150 - Its value must be unique among the members of the name space
  151 - Its value must pass the =check_name()= function =ipalib.base= module.
  153 =check_name()= function encodes a simple rule of a lower-case Python identifier that
  154 neither starts nor ends with an underscore. Actual regular expression that codifies this
  155 rule is =NAME_REGEX= within =ipalib.constants= module.
  157 Once name space is created, it locks itself down and becomes read-only. It means that
  158 while original objects accessed through the name space might change, the references to
  159 them via name space will stay intact. They cannot be removed or changed to point to other
  160 objects.
  162 The name spaces are used widely in FreeIPA core framework. As mentioned earlier, API
  163 includes set of objects, commands, and methods. Objects include properties that are
  164 defined before lock-down. At object's lock-down parameters are placed into a name space
  165 and that locks them down so that no parameter specification can change. Command's
  166 parameters and options also locked down and cannot change once command instance is
  167 instantiated.
  169 ** Parameters
  170 =Param= class is used to define attributes, arguments, or options throughout FreeIPA core
  171 framework. The =Param= base class is not used directly but rather sub-classed to define
  172 properties like passwords or specific data types like =Str= or =Int=.
  174 Instances of classes inherited from =Param= base class give uniform access to the
  175 properties required to command line interface, Web UI, and internally to FreeIPA
  176 code. Following properties are most important:
  177  - /name/ :: name of the parameter used internally to address the parameter in Python
  178              code. The /name/ could include special characters to designate a =Param= spec.
  179  - /cli_name/ :: optional name of the parameter to use in command line
  180                   interface. FreeIPA's CLI sets a mechanism to automatically translate
  181                   from a command line option name to a parameter's /name/ if /cli_name/
  182                   is specified.
  183  - /label/ :: A short phrase describing the parameter. It is used on the CLI when
  184               interactively prompting for the values, and as a label for the form inputs
  185               in the Web UI. The /label/ should start with an initial capital letter.
  186  - /doc/ :: A long description of the parameter. It is used by the CLI when displaying the
  187             help information for a command, and as an extra instruction for the form input
  188             on the Web UI. By default the /doc/ is the same as the /label/ but can be
  189             overridden when a =Param= instance is created. As with /label/, /doc/ should
  190             start with an initial capital letter and additionally should not end with any
  191             punctuation.
  192  - /required/ ::  If set to =True=, means this parameter is required to supply. All
  193                  parameters are required by default and that means that /required/
  194                  property should only be specified when parameter *is not required*.
  195  - /multivalue/ :: if set to =True=, means this parameter can accept a Python's tuple of
  196                    values. By default all parameters are *single-valued*.
  198 When parameter /name/ has any of ~?~, ~*~, or ~+~ characters, it is treated as parameter
  199 spec and is used to specify whether parameter is required, and should it be
  200 multivalued. Following syntax is used:
  202 | Spec   | Name  | Required | Multivalue |
  203 |--------+-------+----------+------------|
  204 | 'var'  | 'var' | True     | False      |
  205 | 'var?' | 'var' | False    | False      |
  206 | 'var*' | 'var' | False    | True       |
  207 | 'var+' | 'var' | True     | True       |
  209 Access to the value stored by the =Param= class is given through a callable interface:
  211 #+BEGIN_SRC python
  212 age = Int('age', label='Age', default=100)
  213 print age(10)
  214 #+END_SRC
  216 Following parameter classes are defined and used throughout FreeIPA framework:
  217 - /Bool/ :: boolean parameters that are stored in Python's ~bool~ type, therefore, they
  218             return either ~True~ or ~False~ value. However, they accept ~1~, ~True~
  219             (Python boolean), or Unicode strings '~1~', '~true~' and '~TRUE~' as truth value, and ~0~,
  220             ~False~ (Python boolean), or Unicode strings '~0~', '~false~', and '~FALSE~' as false.
  221 - /Flag/ :: boolean parameters which always have default value. Property /default/ can be
  222             used to set the value. Defaults to ~False~:
  223 #+BEGIN_SRC python
  224 verbose = Flag('verbose', default=True)
  225 #+END_SRC
  226 - /Int/ :: integer parameters that are stored in Python's int type. Two additional properties can be
  227            specified when constructing =Int= parameter:
  228            - /minvalue/ :: minimal value that this parameter accepts, defaults to =MININT=
  229            - /maxvalue/ :: maximum value this parameter can accept, defaults to =MAXINT=
  230 - /Decimal/ :: floating point parameters that are stored in Python's Decimal type. =Decimal= has
  231              the same two additional properties as =Int=. Unlike =Int=, there are no
  232              default values for the minimal and maximum boundaries.
  233 - /Bytes/ :: a parameter to represent binary data.
  234 - /Str/ :: parameter representing a Unicode text. Both /Bytes/ and /Str/ parameters accept
  235            following additional properties:
  236            - /minlength/ :: minimal length of the parameter
  237            - /maxlength/ :: maximum length of the parameter
  238            - /length/ :: length of the parameters
  239            - /pattern/ :: regular expression applied to the parameter's value to check its
  240                           validness
  241            - /pattern_errmsg/ :: an error message to show when regular expression check fails
  242 - /IA5Str/ :: string parameter as defined by RFC 4517. It means all characters of the
  243               string must be ASCII characters (7-bit).
  244 - /Password/ :: parameter to store passwords in Python =unicode= type. /Password/ has one
  245                 additional property:
  246                 - /confirm/ :: boolean specifying whether password should be confirmed
  247                                when entered. The confirmation is enabled by default.
  248 - /Enum/ :: parameter can have one of predefined values that are specified with /values/
  249             property which is a Python's =tuple=.
  251 For most common case of enumerable strings there are two parameters:
  252 - /BytesEnum/ :: parameter value should be one of predefined =unicode= strings
  253 - /StrEnum/ :: equivalent to /BytesEnum/. Originally /BytesEnum/ was stored in Python's
  254                =str= class instances but to be aligned with Python 3.0 changes both
  255                classes moved to store as =unicode=.
  257 When more than one value should be accepted, there is /List/ parameter that allows to
  258 provide list of strings separated by a separator, default to ','. Also, the /List/
  259 parameter skips spaces before the next item in the list unless property /skipspace/ is set to False:
  260 #+BEGIN_SRC python
  261 names = List('names', separator=',', skipspace=True)
  262 names_list = names(u'John Doe, John Lee, Brad Moe')
  263 # names_list is (u'John Doe', u'John Lee', u'Brad Moe')
  264 names = List('names', separator=',', skipspace=False)
  265 names_list = names(u'John Doe, John Lee, Brad Moe')
  266 # names_list is (u'John Doe', u' John Lee', u' Brad Moe')
  267 #+END_SRC
  269 ** Objects
  270 The data manipulated by FreeIPA is represented by an Object class instances. Instance of
  271 an Object class is a collection of properties, accepted parameters, action methods, and a
  272 reference to where this object's data is preserved. Each object also has a reference to a
  273 property that represents a primary key for retrieving the object.
  275 In addition to properties and parameters, Object class instances hold their labels to use
  276 in user interfaces. In practice, there are few differences in how labels are presented
  277 depending on whether it is command line interface or a Web UI, but they can be ignored at
  278 this point.
  280 To be useful, all Object sub-classes need to override =takes_param= property. This is
  281 where most of flexibility of FreeIPA comes from.
  283 *** takes_param attribute
  284 Properties of every object derived from Object class can be specified manually but FreeIPA
  285 gives a handy mechanism to perform descriptive specification. Each =Object= class has
  286 =Object.takes_param= attribute which defines a specification of all parameters this object
  287 type is accepting. 
  289 Next example shows how to create new object type. We create an aquarium tank by defining
  290 its dimensions and specifying which fish is living there.
  291 #+BEGIN_SRC python -n -r -l '(%s)'
  292 from ipalib import api, Object
  293 class tank(Object):
  294     takes_params = (
  295         StrEnum('species*', label=u'Species', doc=u'Fish species',
  296                  values=(u'Angelfish', u'Betta', u'Cichlid', u'Firemouth')),
  297         Decimal('height', label=u'Height', doc=u'height in mm', default='400.0'),
  298         Decimal('width', label=u'Width', doc=u'width in mm', default='400.0'),
  299         Decimal('depth', label=u'Depth', doc=u'Depth in mm', default='300.0')
  300     )
  302 api.register(tank) (ref:register)
  303 api.finalize()     (ref:finalize)
  304 print list(api.Object.tank.params)
  305 # ['species', 'height', 'width', 'depth']
  306 #+END_SRC
  308 First we define new class, =tank=, that takes four parameters. On line [[(register)]] we register the class
  309 in FreeIPA's API instance, api. This creates =tank= object in =api.Object= name
  310 space. Many objects can be added into the API up until =api.finalize()= is called as we do
  311 on line [[(finalize)]].
  313 When =api.finalize()= is called, all name spaces are locked down and all registered Python
  314 objects in those name spaces are also finalized which in turn locks their structure down
  315 as well.
  317 As result, once we have finalized our API instance, every registered Object can be
  318 accessed through =api.Object.<name>=. Our aquarium tank object now has defined =params=
  319 attribute which is a name space holding all =Param= instances. Thus we can introspect and
  320 see which parameters this object has.
  322 At this point we can't do anything reasonable with our aquarium tank yet because we
  323 haven't defined methods to handle it. In addition, our object isn't very useful as it does
  324 not know how to store the information about aquarium's dimensions and species living in
  325 it.
  327 *** Object methods
  328 Methods perform actions on the associated objects. The association of methods and objects
  329 is done through naming convention rather than using programming language features. FreeIPA
  330 expects methods operating on an object =<name>= to be named =<name>_<action>=:
  331 #+BEGIN_SRC python
  332 class tank_create(Method):
  333     def execute(self, **options):
  334         # create new aquarium tank
  336 api.register(tank_create)
  338 class tank_populate(Method):
  339     def execute(self, **options):
  340         # populate the aquarium tank with fish
  342 api.register(tank_populate)
  343 #+END_SRC
  345 As can be seen, each method is a separate Python class. This approach allows to maintain
  346 complexity of methods isolated from each other and from the complexity of the objects and
  347 their storage which is probably most important aspect due to LDAP complexity overall.
  349 The linking between objects and their methods goes further. All parameters defined for an
  350 object, may be used as arguments of the methods without explicit declaration. This means
  351 =api.Method.tank_populate= will accept ~species~ argument.
  353 *** Methods with storage back ends
  354 In order to store the information, =Object= class instances require a back end. FreeIPA
  355 defines several back ends but the ones that could store data are derived of
  356 =ipalib.CrudBackend=. CRUD, or /Create/, /Retrieve/, /Update/, and /Delete/, are basic
  357 operations that could be performed with corresponding objects. =ipalib.crud.CrudBackend=
  358 is an abstract class, it only defines functions that should be overridden in classes that
  359 actually implement the back end operations.
  361 As back end is not used directly, FreeIPA defines methods that could use back end and
  362 operate on object's defined by certain criteria. Each method is defined as a separate
  363 Python class. As CRUD acronym suggests, there are four base operations:
  364 =ipalib.crud.Create=, =ipalib.crud.Retrieve=, =ipalib.crud.Update=,
  365 =ipalib.crud.Delete=. In addition, method =ipalib.crud.Search= allows to retrieve all
  366 entries that match a given search criteria.
  368 When objects are defined and the back end is known, methods can be used to manipulate
  369 information stored by the back end. Most of useful operations combine some of CRUD base
  370 operations to perform their tasks.
  372 In order to support flexible way to extend methods, FreeIPA gives special treatment for
  373 the LDAP back end. Methods using LDAP back end hide complexity of handling LDAP queries and
  374 allow to register user-provided functions that are called before or after method. This
  375 mechanism is defined by ipalib.plugins.baseldap.CallbackInterface and used by LDAP-aware
  376 CRUD classes, =LDAPCreate=, =LDAPRetrieve=, =LDAPUpdate=, =LDAPDelete=, and an analogue to
  377 =ipalib.crud.Search=, =LDAPSearch=. There are also classes that define methods to operate
  378 on reverse relationships between objects in LDAP to allow addition or removal of
  379 membership information both in forward and reverse directions: =LDAPAddMember=,
  380 =LDAPModMember=, =LDAPRemoveMember=, =LDAPAddReverseMember=, =LDAPModReverseMember=, =LDAPRemoveReverseMember=.
  382 Most of CRUD classes are based on a =LDAPQuery= class which generalizes concept of
  383 querying a record addressed with a primary key and supports JSON marshalling of the
  384 queried attributes and their values.
  386 Base LDAP operation classes implement everything needed to create typical methods to
  387 work with self-contained objects stored in LDAP. 
  389 *** LDAPObject class
  390 A large class of objects is LDAPObject. LDAPObject instances represent entries stored in
  391 FreeIPA LDAP database instance. They are referenced by their distinguished name, DN, and
  392 able to represent complex relationships between entries in LDAP like direct and indirect
  393 membership. 
  395 Any class derived from LDAPObject needs to re-define few properties so that base class can
  396 properly function for the specific object that is defined by the class. Below are commonly
  397 redefined properties:
  398  - /container_dn/ :: DN of the container for this object entries in LDAP. This one
  399       usually comes from the environment associated with the API and by default is populated
  400       from the =DEFAULT_CONFIG= of =ipalibs.constants=. For example, all accounts are
  401       stored under =cn=accounts=, with users are under =cn=users,cn=accounts= and groups
  402       are under =cn=groups,cn=accounts=. In case of a new object added, it
  403       is reasonable to select its container coordinated to default configuration.
  404  - /object_class/ :: list of LDAP object classes associated with the object
  405  - /search_attributes/ :: list of attributes that will be used for search
  406  - /default_attributes/ :: list of attributes that are always returned by searches
  407  - /uuid_attribute/ :: an attribute that defines uniqueness of the entry
  408  - /attribute_members/ :: a dict defining relations between other objects and this
  409       one. Key is the name of attribute and value is a list of objects this attribute may
  410       refer to. For example, =host= object defines that  =memberof= attribute of a
  411       host may refer to a =hostgroup=, =netgroup=, =role=, =hbacrule=, or =sudorule=
  412       object. In other words, it means that =host= could be a member of any of those
  413       objects.
  414  - /reverse_members/ :: a dict defining reverse relations between this object and other
  415       objects. Key is the name of attribute and value is the name of an object that refers
  416       to this object with the attribute. For example, =role= object defines that =member=
  417       attribute of a =privilege= refers to a =role= object.
  418  - /password_attributes/ :: list of pairs defining an attribute in LDAP and a property of
  419       a Python dictionary representing the LDAP object attributes that will be set
  420       accordingly if such attribute exists in the LDAP entry. As passwords have restricted
  421       access, often one needs only to know that there is a password set on the entry to
  422       perform additional processing.
  423  - /relationships/ :: a dict defining existing relationship criteria associated with the
  424       object. These are used in Web UI to allow filtering of objects by the criteria. The
  425       value is defined as a tuple of an UI label and two prefixes: inclusive and exclusive
  426       that are prepended to the attribute parameter when options are generated by the
  427       framework. LDAPObject defines few default criteria: /member/, /memberof/,
  428       /memberindirect/, /memberofindirect/, and objects can redefine or append more. Due
  429       to regularity of the design of LDAP objects, default criteria already makes it
  430       possible to apply searches almost uniformly: one can ask for membership of a user in
  431       a group, as well as for a membership of a role in a privilege without explicitly
  432       defining those relationships.
  435 These properties define how translation would go from Python side to and from an LDAP
  436 backend.
  438 As an example, let's see how role is defined. This is fully functioning plugin that
  439 provides operations on roles:
  440 #+INCLUDE "role.py.txt" src python -n
  442 * Extending existing object
  443 As said earlier, until API instance is finalized, objects, methods, and commands can be
  444 added, removed, or modified freely. This allows to extend existing objects. Before API is
  445 finalized, we cannot address objects through the unified interface as =api.Object.foo=,
  446 but for almost all cases an object named =foo= is defined in a plugin
  447 =ipalib.plugins.foo=.
  449 1. Add new parameter:
  450   #+BEGIN_SRC python -n
  451 from ipalib.plugins.user import user
  452 from ipalib import Str, _
  453 user.takes_params += (
  454        Str('foo',
  455             cli_name='foo',
  456             label=_('Foo'),
  457        ),
  458     )
  459   #+END_SRC
  460 2. Re-define User object label to use organisation-specific terminology in Web UI:
  461   #+BEGIN_SRC python -n
  462 from ipalib.plugins.user import user
  463 from ipalib import text
  465 _ = text.GettextFactory(domain='extend-ipa')
  466 user.label = _('Staff')
  467 user.label_singular = _('Engineer')
  468   #+END_SRC
  469   Note that we re-defined locally =_= method to use different ~GettextFactory~. As
  470   GettextFactory is supporting a single translation domain, all new translation terms need
  471   to be placed in a separate translation domain and referred accordingly. Python rules for
  472   scoping will keep this symbol as ~<package>._~ and as nobody imports it explicitly, it
  473   will not interfere with the framework's provided ~text._~.
  474 3. Assume =/dev/null= as default shell for all new users:
  475   #+BEGIN_SRC python -n -r
  476 from ipalib.plugins.user import user_add
  478 def override_default_shell_cb(self, ldap, dn. 
  479                               entry_attrs, attrs_list,
  480                               *keys, **options):
  481     if 'loginshell' in entry_attrs:
  482         default_shell = [self.api.Object.user.params['loginshell'].default]
  483         if entry_attrs['loginshell'] == default_shell:
  484             entry_attrs['loginshell'] = [u'/dev/null']
  486 user_add.register_pre_callback(override_default_shell_cb)
  487   #+END_SRC
  489 The last example exploits a powerful feature available for every method of LDAPObject:
  490 registered callbacks.
  491 * Extending existing method
  492 For objects stored in LDAP database instance all methods support adding callbacks. A
  493 /callback/ is a user-provided function that is called at certain point of execution of a
  494 method.
  496 There are four types of callbacks:
  497 - /PRE callback/ :: called before executing the method's action. Allows to modify passed
  498                     arguments, do additional validation or data transformation and
  499                     specific access control beyond what is provided by the framework.
  500 - /POST callback/ :: called after executing the method's action. Allows to analyze results
  501      of the action and perform additional actions or modify output.
  502 - /EXC callback/  :: called in case execution of the method's action caused an execution
  503      error. These callbacks provide means to recover from an erroneous execution.
  504 - /INTERACTIVE callback/ :: called at a client context to allow a command to decide if
  505      additional parameters should be requested from an user. This mechanism especially
  506      useful to simplify complex interaction when there are several levels of possible
  507      scenarios depending on what was provided at a client side.
  509 All callback types are available to any class derived from =CallbackInterface=
  510 class. These include all LDAP-based CRUD methods.
  512 Callback registration methods accept a reference to callable and optionally ordering
  513 argument =first= (~False~ by default) to allow the callback be executed before previously
  514 registered callbacks of this type.
  516 =CallbackInterface= class provides following class methods:
  517 - =register_pre_callback= :: registers /PRE/ callback
  518 - =register_post_callback= :: registers /POST/ callback
  519 - =register_exc_callback= :: registers /EXC/ callback for purpose of recovering from
  520      execution errors
  521 - =register_interactive_prompt_callback= :: registers callbacks called by the client
  522      context.
  524 Let's look again at the last example:
  525 #+BEGIN_SRC python -n -r
  526 from ipalib.plugins.user import user_add
  528 def override_default_shell_cb(self, ldap, dn. 
  529                               entry_attrs, attrs_list, 
  530                               *keys, **options):
  531     if 'loginshell' in entry_attrs:
  532         default_shell = [self.api.Object.user.params['loginshell'].default]
  533         if entry_attrs['loginshell'] == default_shell:
  534             entry_attrs['loginshell'] = [u'/dev/null']
  536 user_add.register_pre_callback(override_default_shell_cb)
  537 #+END_SRC
  539 This extension defines a pre-processing callback that accepts number of arguments:
  540 - /ldap/ :: reference to the back end to store and retrieve the object's data
  541 - /dn/ :: reference to the object data in LDAP
  542 - /entry_attrs/ :: arguments and options of the command and their values as a
  543                    dictionary. All values in /entry_attrs/ will be used for communicating
  544                    with LDAP store, thus replacing values should be done with care. For
  545                    details please see Python LDAP module documentation
  546 - /attrs_list/ :: list of all attributes we intend to fetch from the back end
  547 - /keys/ :: arguments of the command
  548 - /options/ :: all other unidentified parameters passed to the method
  550 Arguments of a post-processing callback, /POST/, are slightly different. As action is
  551 already performed and the attributes of the entry are fetched back from the back end,
  552 there is no need to provide =attrs_list=:
  553 #+BEGIN_SRC python -n -r
  554 from ipalib.plugins.user import user_add
  555 def verify_shell_cb(self, ldap, dn. entry_attrs, 
  556                     *keys, **options):
  557     if 'loginshell' in entry_attrs:
  558         default_shell = [self.api.Object.user.params['loginshell'].default]
  559         if entry_attrs['loginshell'] == default_shell:
  560             # report that default shell is assigned
  562 user_add.register_post_callback(verify_shell_cb)
  563 #+END_SRC
  565 Execution error callback, /EXC/, has following signature:
  566 #+BEGIN_SRC python -n
  567 def user_add_error_cb(self, args, options, exc,
  568                       call_func, *call_args, **call_kwargs):
  569     return
  570 #+END_SRC
  572 where arguments have following meaning:
  573 - /args/ :: arguments of the original method
  574 - /options/ :: options of the original method
  575 - /exc/ :: exception object thrown by a /call_func/
  576 - /call_func/ :: function that was called by the method and caused the error of
  577                  execution. In case of LDAP-based methods this is often =ldap.add_entry()=
  578                  or =ldap.modify_entry()=, or a similar function
  579 - /call_args/ :: first argument passed to the /call_func/
  580 - /call_kwargs/ :: remaining arguments of /call_func/
  582 Finally, interactive prompt callback receives /kw/ argument which is a dictionary of all
  583 arguments of the command.
  585 All callbacks are supplied with a reference to the method instance, ~self~, unless the
  586 callback itself has an attribute called '~im_self~'. As can be seen in callback examples,
  587 self reference recursively provides access to the whole FreeIPA API structure.
  589 This approach gives complete control of existing FreeIPA methods without
  590 deep dive into details of LDAP programming even if the framework allows such a deep dive.
  592 * Web UI
  593 FreeIPA framework has two major client applications: Web UI and command line-based client
  594 tool, ~ipa~. Web UI communicates with a FreeIPA server running WSGI application that
  595 accepts JSON-formatted requests and translates them to calls to FreeIPA plugins.
  597 A following code in ~install/share/ui/wsgi.py~ defines FreeIPA web application:
  598 #+INCLUDE "wsgi.py.txt" src python -n -r
  600 At line [[(wsgi-app-bootstrap)]] we set up FreeIPA framework with server context. This means
  601 plugins are loaded and initialized from following locations:
  602 - ~ipalib/plugins/~ -- general FreeIPA plugins, available for all contexts
  603 - ~ipaserver/plugins/~ -- server-specific plugins, available in '~server~' context
  605 With =api.finalize()= call at line [[(wsgi-app-finalize)]] FreeIPA framework is locked down and all
  606 components provided by plugins are registered at ~api~ name spaces: =api.Object=,
  607 =api.Method=, =api.Command=, =api.Backend=.
  609 At this point, ~api~ name spaces become usable and our WSGI entry point, defined on lines
  610 [[(wsgi-app-start)]] to [[(wsgi-app-end)]] can access =api.Backend.session()= to generate
  611 response for WSGI request.
  613 Web UI itself is written in JavaScript and utilizes JQuery framework. It can be split into
  614 three major parts:
  615 - /communication/ :: tools defined in ~ipa.js~ to allow talking with FreeIPA server using
  616      AJAX requests and JSON formatting
  617 - /presentation/ :: tools in ~facet.js~, ~entity.js~, ~search.js~, ~widget.js~, ~add.js~,
  618                     and ~details.js~ to give basic building blocks of Web UI
  619 - /objects/ :: actual implementation of Web UI for FreeIPA objects (user, group, host,
  620                rule, and other available objects registered at =api.Object= by the server
  621                side)
  623 The code of these JavaScript files is loaded in ~index.html~ and kicked into work by
  624 ~webui.js~ where main navigation and document's ~onready~ event handler are defined. In
  625 addition, ~index.html~ imports ~extension.js~ file where all extensions to Web UI can be
  626 registered or referenced. As ~extension.js~ is loaded after all other Web UI JavaScript
  627 files but before ~webui.js~, it can already use all tools of the Web UI.
  629 The execution of Web UI starts with the call of =IPA.init()= function which does
  630 following:
  631 1. Set up AJAX asynchronous communication via POST method using JSON format.
  632 2. Fetches meta-data about FreeIPA methods available on the server using JSON format and
  633    makes them available as =IPA.methods=.
  634 3. Fetches meta-data about FreeIPA objects available on the server using JSON format and
  635    makes them available as =IPA.objects=.
  636 4. Fetches translations of messages used in the Web UI and makes them available as
  637    =IPA.messages=.
  638 5. Fetches identity of the user running the Web UI, accessible as =IPA.whoami=.
  639 6. Fetches FreeIPA environment specific for Web UI, accessible as =IPA.env=.
  641 The communication with FreeIPA server is done using =IPA.command()= function. Commands
  642 created with =IPA.command()= can later be executed with =execute()= method. This
  643 separation of construction and actual execution allows to create multiple commands and
  644 combine them together in a single request. Batch requests are created with
  645 =IPA.batch_command()= function and command are added to them with =add_command()=
  646 method. In addition, FreeIPA Web UI allows to run commands concurrently with
  647 =IPA.concurrent_command()= function.
  649 Web UI has following DOM structure:
  650 |-----------------------+-----------------------------------+------------+-----------|
  651 |                       | Container                         |            |           |
  652 |-----------------------+-----------------------------------+------------+-----------|
  653 | background            | header                            | navigation | content   |
  654 | background-header     | header-logo                       |            |           |
  655 | background-navigation | header-network-activity-indicator |            |           |
  656 | background-left       | loggedinas                        |            |           |
  657 | background-right      |                                   |            |           |
  658 |-----------------------+-----------------------------------+------------+-----------|
  660 ~Container~ div is a top-level one, it includes background, header, navigation, content
  661 divs. These divs and their parts can be manipulated from the JavaScript code to represent
  662 the UI. However, FreeIPA gives an easier way to accomplish this.
  664 ** Facets
  665 Facet is a smallest block of FreeIPA Web UI. When facet is defined, it has name, label,
  666 link to an entity it is part of, and methods to create, show, load, and hide itself.
  668 ** Entities
  669 Entity is addressable group of facets. FreeIPA Web UI provides a declarative way of
  670 creating entities and defining their facets based on JavaScript's syntax. Following
  671 example is a complete definition of a netgroup facet:
  672 #+INCLUDE "netgroup.js" src js2-mode -n
  674 This definition of a netgroup facet describes:
  675 - /details facet/ :: a facet named '~identity~' and three fields, ~cn~, ~description~,
  676      and ~nisdomainname~. In addition, ~description~ field is a text area widget. This
  677      facet is used to display existing netgroup information.
  678 - /association facets/ :: number of facets, linking this one with others. In case of a
  679      netgroup, netgroups are linked to facet group ~member~ via different attributes. The
  680      definition also adds standard association facets defined in ~entity.js~.
  681 - /adder dialog/ :: a dialog to create a new netgroup. The dialog has two fields: ~cn~ and
  682                     ~description~ where ~description~ is again a text area widget.
  684 Similarly to FreeIPA core framework, created entity needs to be registered to the Web UI
  685 via =IPA.register()= method.
  687 In order to add new entity to the Web UI, one can use ~extension.js~. This file in
  688 ~/usr/share/ipa/html~ is empty and provided specifically for this purpose.
  690 As an example, let's define an entity 'Tank' corresponding to our aquarium tank:
  691 #+BEGIN_SRC js2-mode -n
  692 IPA.tank = {};
  693 IPA.tank.entity = function(spec) {
  694     var that = IPA.entity(spec);
  695     that.init = function(params) {
  696         details_facet({
  697             sections: [
  698                 {
  699                      name: 'identity',
  700                      fields: [
  701                          'species', 'height', 'width', 'depth'
  702                      ]
  703                 }
  704             ]
  705         }).
  706         standard_association_facets().
  707         adder_dialog({
  708             fields: [
  709                 'species', 'height', 'width', 'depth'
  710             ]
  711         });
  712     };
  713 };
  715 IPA.register('tank', IPA.tank.entity);
  716 #+END_SRC
  718 * Command line tools
  719 As an alternative to Web UI, FreeIPA server can be controlled via command-line interface
  720 provided by the ~ipa~ utility. This utility is operating under '~client~' context and
  721 looks even simpler than Web UI's ~wsgi.py~:
  722 #+BEGIN_SRC python -n
  723 import sys
  724 from ipalib import api, cli
  726 if __name__ == '__main__':
  727     cli.run(api)
  728 #+END_SRC
  730 =cli.run()= is the central running point defined in ~ipalib/cli.py~:
  731 #+BEGIN_SRC python -n
  732 # <cli.py code> ....
  733 cli_plugins = (
  734     cli,
  735     textui,
  736     console,
  737     help,
  738     show_mappings,
  739 )
  742 def run(api):
  743     error = None
  744     try:
  745         (_options, argv) = api.bootstrap_with_global_options(context='cli')
  746         for klass in cli_plugins:
  747             api.add_plugin(klass)
  748         api.finalize()
  749         if not 'config_loaded' in api.env and not 'help' in argv:
  750             raise NotConfiguredError()
  751         sys.exit(api.Backend.cli.run(argv))
  752     except KeyboardInterrupt:
  753         print('')
  754         logger.info('operation aborted')
  755     except PublicError as e:
  756         error = e
  757     except Exception as e:
  758         logger.exception('%s: %s', e.__class__.__name__, str(e))
  759         error = InternalError()
  760     if error is not None:
  761         assert isinstance(error, PublicError)
  762         logger.error(error.strerror)
  763         sys.exit(error.rval)
  764 #+END_SRC
  766 As with WSGI, =api= is bootstraped, though with a client context and using global options
  767 from ~/etc/ipa/default.conf~, and command line arguments. In addition to common plugins
  768 available in ~ipalib/plugins~, ~cli.py~ adds few command-line specific classes defined in
  769 the module itself:
  770 - ~cli~ :: a backend for executing from command line interface which does translation of
  771            command line option names, basic verification of commands and fallback to show
  772            help messages with ~help~ command, execution of the command, and translation of
  773            the output to command-line friendly format if this is defined for the command.
  774 - ~textui~ :: a backend to nicely format output to stdout which handles conversion from
  775               binary to base64, prints text word-wrapped to the terminal width, formats
  776               returned complex values so that they can be easily understood by a human
  777               being.
  778    #+BEGIN_EXAMPLE
  779 >>> entry = {'name' : u'Test example', 'age' : u'100'}
  780 >>> api.Backend.textui.print_entry(entry)
  781   age: 100
  782   name: Test example
  783    #+END_EXAMPLE
  784 - ~console~ :: starts interactive Python console with FreeIPA commands
  785 - ~help~ :: generates help for every command and method of FreeIPA and structures it into
  786             sections according to the registered FreeIPA objects.
  787    #+BEGIN_EXAMPLE
  788 >>> api.Command.help(u'user-show')
  789 Purpose: Display information about a user.
  790 Usage: ipa [global-options] user-show LOGIN [options]
  792 Options:
  793 -h, --help  show this help message and exit
  794 --rights    Display the access rights of this entry (requires --all). See 
  795             ipa man page for details.
  796 --all       Retrieve and print all attributes from the server. Affects 
  797             command output.
  798 --raw       Print entries as stored on the server. Only affects output
  799             format.
  800    #+END_EXAMPLE
  801 - ~show_mappings~ ::  displays mappings between command's parameters and LDAP attributes:
  802    #+BEGIN_EXAMPLE
  803 >>> api.Command.show_mappings(command_name=u"role-find")
  804 Parameter : LDAP attribute
  805 ========= : ==============
  806 name      : cn
  807 desc      : description
  808 timelimit : timelimit?
  809 sizelimit : sizelimit?
  810    #+END_EXAMPLE
  812 ** Extending command line utility
  813 Since ~ipa~ utility operates under client context, it loads all command plugins from
  814 ~ipalib/plugins~. A simple way to extend command line is to drop its plugin file into
  815 ~ipalib/plugins~ on the machine where ~ipa~ utility is executed. Next time ~ipa~ is
  816 started, new plugin will be loaded together with all other plugins from ~ipalib/plugins~
  817 and commands provided by it will be added to the =api=.
  819 Let's add a command line plugin that allows to ping a server and measures round trip time:
  820 #+BEGIN_SRC python -n
  821 from ipalib import frontend
  822 from ipalib import output
  823 from ipalib import _, ngettext
  824 from ipalib import api
  825 import time
  827 __doc__ = _("""
  828 Local extensions to FreeIPA commands
  829 """)
  831 class timed_ping(frontend.Command):
  832     __doc__ = _('Ping remote FreeIPA server and measure round-trip')
  834     has_output = (
  835 			  output.summary,
  836 			  )
  837     def run(self):
  838         t1 = time.time()
  839         result = self.api.Command.ping()
  840         t2 = time.time()
  841         summary = u"""Round-trip to the server is %f ms.
  842 Server response is %s"""
  843         return dict(summary=summary % ((t2-t1)*1000.0, result['summary']))
  845 api.register(timed_ping)
  846 #+END_SRC
  848 When this plugin code is placed into ~ipalib/plugins/extend-cli.py~ (name of the plugin
  849 file can be set arbitrarily), ~ipa timed-ping~ will produce following output:
  851 $ ipa timed-ping
  852 -----------------------------------------------------------------------------
  853 Round-trip to the server is 286.306143 ms.
  854 Server response is IPA server version 2.1.3GIT8a254ca. API version 2.13
  855 -----------------------------------------------------------------------------
  858 In this example we have created ~timed-ping~ command and overrode its =run()=
  859 method. Effectively, this command will only work properly on the client. If the client is
  860 also FreeIPA server (all FreeIPA servers are enrolled as FreeIPA clients), the same code
  861 will also be loaded by the server context and will be accessible to the Web UI as well,
  862 albeit its usefulness will be questionable as it will be measuring the round-trip to the
  863 server from the server itself.
  865 * File paths
  866 Finally, it should be noted that depending on installed Python version and operating
  867 system, paths where plugins are loaded from may differ. Usually Python extensions are
  868 placed in ~site-packages~ Python sub-directory. In Fedora and RHEL distributions, this is
  869 ~/usr/lib/python<version>/site-packages~. Thus, full path to ~extend-cli.py~ would be
  870 ~/usr/lib/python<version>/site-packages/ipalib/plugins/extend-cli.py~.
  872 On recent Fedora distribution, following paths are used:
  873 |--------------------+---------------------------+------------------------------------------------------------|
  874 | Plugins            | Python module prefix      | File path                                                  |
  875 |--------------------+---------------------------+------------------------------------------------------------|
  876 | common             | ipalib/plugins            | /usr/lib/python2.7/site-packages/ipalib/plugins            |
  877 | server             | ipaserver/plugins         | /usr/lib/python2.7/site-packages/ipaserver/plugins         |
  878 | installer, updates | ipaserver/install/plugins | /usr/lib/python2.7/site-packages/ipaserver/install/plugins |
  879 |--------------------+---------------------------+------------------------------------------------------------|
  881 Next table explains use of contexts in FreeIPA applications:
  882 |---------+------------------+-------------------------+----------------------------------------|
  883 | Context | Application      | Plugins                 | Description                            |
  884 |---------+------------------+-------------------------+----------------------------------------|
  885 | server  | wsgi.py          | common, server          | Main FreeIPA server, server context    |
  886 | cli     | ipa              | common                  | Command line interface, client context |
  887 | updates | ipa-ldap-updater | common, server, updates | LDAP schema updater                    |
  888 |---------+------------------+-------------------------+----------------------------------------|
  891 * Platform portability
  892 Originally FreeIPA was created utilizing packages available in Fedora and RHEL
  893 distributions. During configuration stages multiple system services need to be stopped
  894 and started again, scheduled to start after reboot and re-configured. In addition, when
  895 operating system utilizing security measures to harden the server setup, appropriate
  896 activities need to be done as well for preserving proper security contexts. As
  897 configuration details, service names, security features and management tools differ
  898 substantially between various GNU/Linux distributions and other operating systems, porting
  899 FreeIPA project's code to other environment has proven to be problematic.
  901 When Fedora project has decided to migrate to systemd for services management, FreeIPA
  902 packages for Fedora needed to be updated as well, at the same time preserving support for
  903 older SystemV initialization scheme used in older releases. This prompted to develop a
  904 'platformization' support allowing to abstract services management between different
  905 platforms.
  907 FreeIPA 2.1.3 includes first cut of platformization work to support Fedora 16 distribution
  908 based on systemd. At the same time, there is an effort to port FreeIPA client side code to
  909 Ubuntu distributions.
  911 Platform portability in FreeIPA means centralization of code to manage system-provided
  912 services, authentication setup, and means to manage security context and host names. It is
  913 going to be extended in future to cover other areas as well, both client- and server-side.
  915 The code that implements platform-specific adaptation is placed under
  916 ~ipaplatform~. As of FreeIPA 4.4.2, there are two major "platforms" supported:
  917 - /rhel/ :: Red Hat Enterprise Linux 7-based distributions utilizing Systemd
  918             such as CentOS 7 and Scientific Linux 7.
  919 - /fedora/ :: Fedora distribution version 23 above are supported by this platform
  920                 module. It is based on ~systemd~ system management tool and utilizes
  921                 common code in ~ipaplatform/base/services.py~. ~fedora~ contains
  922                 only differentiation required to cover Fedora 23-specific implementation
  923                 of systemd use, depending on changes to Dogtag, Tomcat6, and 389-ds
  924                 packages.
  926 Each platform-specific adaptation should provide few basic building blocks:
  928 *** AuthConfig class and tasks module
  930 =ipaplatform.tasks= module implements system-independent interface to configure system
  931 resources. In Red Hat systems some of these tasks are done with authconfig(8) utility.
  933 =AuthConfig= class is nothing more than a tool to gather configuration options and execute
  934 their processing. These options then converted by an actual implementation to series of a
  935 system calls to appropriate utilities performing real configuration.
  937 From FreeIPA code perspective, the system configuration should be done with
  938 use of ~ipaplatform.tasks.tasks~:
  940 #+BEGIN_SRC python -n
  941 from ipaplatform.tasks import tasks
  943 tasks.set_nisdomain('nisdomain.example')
  944 #+END_SRC
  946 The actual implementation can differ. ~redhat~ platform module builds up arguments to
  947 authconfig(8) tool and on =execute()= method runs it with those arguments. Other systems
  948 will need to have processing based on their respective tools.
  950 *** PlatformService class
  951 =PlatformService= class abstracts out an external process running on the system which is
  952 possible to administer: start, stop, check its status, schedule for automatic startup,
  953 etc.
  955 Services are used thoroughly through FreeIPA server and client install tools. There are
  956 several services that are used especially often and they are selected to be accessible via
  957 Python properties of =ipaplatform.services.knownservices= instance.
  959 To facilitate more expressive way of working with often used services, ipaplatform.services
  960 module provides a shortcut to access them by name via
  961 ipaplatform.services.knownservices.<service>. A typical code change looks like this:
  963 import ipaplatform.services.knownservices
  964 ....
  965 -    service.restart("dirsrv")
  966 -    service.restart("krb5kdc")
  967 -    service.restart("httpd")
  968 +    ipaplatform.services.knownservices.dirsrv.restart()
  969 +    ipaplatform.services.knownservices.krb5kdc.restart()
  970 +    ipaplatform.services.knownservices.httpd.restart()
  973 Besides expression change this also makes more explicit to platform providers access to
  974 what services they have to implement. Service names are defined in
  975 ipaplatform.platform.base.wellknownservices and represent definitive names to access these
  976 services from FreeIPA code. Of course, platform provider should remap those names to
  977 platform-specific ones -- for ipaplatform.redhat provider mapping is identity.
  979 Porting to a new platform may be hard as can be witnessed by this example:
  980 https://www.redhat.com/archives/freeipa-devel/2011-September/msg00408.html
  982 If there is doubt, always consult existing providers. ~redhat/services.py~ is canonical -- it
  983 represents the code which was used throughout FreeIPA v2 development.
  985 *** Enabling new platform provider
  986 When support for new platform is implemented and appropriate provider is placed to
  987 ~ipaplatform/platform/~, it is time to enable its use by the FreeIPA. Since FreeIPA is
  988 supposed to be rolled out uniformly on multiple clients and servers, best approach is to
  989 build and distribute software packages using platform-provided package management tools.
  991 With this in mind, platform code selection in FreeIPA is static and run at package
  992 production time. In order to select proper platform provider, one needs to pass
  993 ~--with-ipaplatform~ argument to FreeIPA's configure process:
  996 ./configure --with-ipaplatform=fedora