As of the Newton release, keystone supports two different approaches to upgrading across releases. The traditional approach requires a significant outage to be scheduled for the entire duration of the upgrade process. The more modern approach results in zero downtime, but is more complicated due to a longer upgrade procedure.
The details of these steps are entirely dependent on the details of your specific deployment, such as your chosen application server and database management system. Use it only as a guide when implementing your own upgrade process.
Plan your upgrade:
keystone-paste.ini, and anything else in
/etc/keystone/, by customizing the corresponding files from the next release.
This is a high-level description of our upgrade strategy built around
keystone-manage db_sync. It assumes that you are willing to have downtime of your control plane during the upgrade process and presents minimal risk. With keystone unavailable, no other OpenStack services will be able to authenticate requests, effectively preventing the rest of the control plane from functioning normally.
/etc/keystone/) with those corresponding from the latest release.
keystone-manage db_syncfrom any single node to upgrade both the database schema and run any corresponding database migrations.
keystone-manage doctorto diagnose symptoms of common deployment issues and receive instructions for resolving them.
If you run a multi-node keystone cluster that uses a replicated database, like a Galera cluster, it is possible to upgrade with minimal downtime. This method also optimizes recovery time from a failed upgrade. This section assumes familiarity with the base case (Upgrading with downtime) outlined above. In these steps the nodes will be divided into
firstnode. This can be done via a variety of mechanisms that will depend on the deployment. If you are unable to disable a service or place a service into maintenance mode in your load balancer, you can stop the keystone processes.
othernodes in the cluster. This will isolate the old dataset on a single node in the cluster. In the event of a failed update this data can be used to rebuild the cluster without having to restore from backup.
firstnode. keystone is now down for your cloud.
keystone-manage db_syncon the
firstnode. As soon as this finishes, keystone is now working again on a single node in the cluster.
othernodes. This entails updating configuration files and upgrading the code. The
db_syncdoes not need to be run again.
Using this model, the outage window is minimized because the only time when your cluster is totally offline is between loading the newer version of keystone and running the
db_sync command. Typically the outage with this method can be measured in tens of seconds especially if automation is used.
If the upgrade fails, only a single node has been affected. This makes the recovery simpler and quicker. If issues are not discovered until the entire cluster is upgraded, a full shutdown and restore from backup will be required. That will take much longer than just fixing a single node with an old copy of the database still available. This process will be dependent on your architecture and it is highly recommended that you've practiced this in a development environment before trying to use it for the first time.
This process should be doable in a matter of minutes and will minimize cloud downtime if it is required.
Upgrading without downtime is only supported in deployments upgrading from Newton or a newer release.
If upgrading a Mitaka deployment to Newton, the commands described here will be available as described below, but the
keystone-manage db_sync --expandcommand will incur downtime (similar to running
keystone-manage db_sync), as it runs legacy (downtime-incurring) migrations prior to running schema expansions.
This is a high-level description of our upgrade strategy built around additional options in
keystone-manage db_sync. Although it is much more complex than the upgrade process described above, it assumes that you are not willing to have downtime of your control plane during the upgrade process. With this upgrade process, end users will still be able to authenticate to receive tokens normally, and other OpenStack services will still be able to authenticate requests normally.
Make a backup of your database. keystone does not support downgrading the database, so restoring from a full backup is your only option for recovery in the event of an upgrade failure.
Stop the keystone processes on the first node (or really, any arbitrary node). This node will serve to orchestrate database upgrades.
Upgrade your first node to the next release, but do not start any keystone processes.
Update your configuration files on the first node (
/etc/keystone/) with those corresponding to the latest release.
(New in Newton) Run
keystone-manage doctor on the first node to diagnose symptoms of common deployment issues and receive instructions for resolving them.
(New in Newton) Run
keystone-manage db_sync --expand on the first node to expand the database schema to a superset of what both the previous and next release can utilize, and create triggers to facilitate the live migration process.
For MySQL, using the
keystone-manage db_sync --expand command requires that you either grant your keystone user
SUPER privileges, or run
set global log_bin_trust_function_creators=1; in mysql beforehand.
At this point, new columns and tables may exist in the database, but will not all be populated in such a way that the next release will be able to function normally.
As the previous release continues to write to the old schema, database triggers will live migrate the data to the new schema so it can be read by the next release.
(New in Newton) Run
keystone-manage db_sync --migrate on the first node to forcefully perform data migrations. This process will migrate all data from the old schema to the new schema while the previous release continues to operate normally.
When this process completes, all data will be available in both the new schema and the old schema, so both the previous release and the next release will be capable of operating normally.
Update your configuration files (
/etc/keystone/) on all nodes (except the first node, which you've already done) with those corresponding to the latest release.
Upgrade all keystone nodes to the next release, and restart them one at a time. During this step, you'll have a mix of releases operating side by side, both writing to the database.
As the next release begins writing to the new schema, database triggers will also migrate the data to the old schema, keeping both data schemas in sync.
(New in Newton) Run
keystone-manage db_sync --contract to remove the old schema and all data migration triggers.
When this process completes, the database will no longer be able to support the previous release.
(New in Pike) In order to check the current state of your rolling upgrades, you may run the command
keystone-manage db_sync --check. This will inform you of any outstanding actions you have left to take as well as any possible upgrades you can make from your current version. Here are a list of possible return codes.
0means you are currently up to date with the latest migration script version and all
db_synccommands are complete.
1generally means something serious is wrong with your database and operator intervention will be required.
2means that an upgrade from your current database version is available, your database is not currently under version control, or the database is already under control. Your first step is to run
keystone-manage db_sync --expand.
3means that the expansion stage is complete, and the next step is to run
keystone-manage db_sync --migrate.
4means that the expansion and data migration stages are complete, and the next step is to run
keystone-manage db_sync --contract.