Red Hat Ceph Storage-3-Installation Guide For Red Hat Enterprise Linux-En-US
Red Hat Ceph Storage-3-Installation Guide For Red Hat Enterprise Linux-En-US
Red Hat Ceph Storage-3-Installation Guide For Red Hat Enterprise Linux-En-US
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons
Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is
available at
http://creativecommons.org/licenses/by-sa/3.0/
. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must
provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert,
Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift,
Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States
and other countries.
Linux ® is the registered trademark of Linus Torvalds in the United States and other countries.
XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States
and/or other countries.
MySQL ® is a registered trademark of MySQL AB in the United States, the European Union and
other countries.
Node.js ® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the
official Joyent Node.js open source or commercial project.
The OpenStack ® Word Mark and OpenStack logo are either registered trademarks/service marks
or trademarks/service marks of the OpenStack Foundation, in the United States and other
countries and are used with the OpenStack Foundation's permission. We are not affiliated with,
endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
Abstract
This document provides instructions on installing Red Hat Ceph Storage on Red Hat Enterprise
Linux 7 running on AMD64 and Intel 64 architectures.
Table of Contents
Table of Contents
. . . . . . . . . . . 1.. .WHAT
CHAPTER . . . . . . .IS
. . RED
. . . . . HAT
. . . . .CEPH
. . . . . . STORAGE?
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4. . . . . . . . . . . . .
.CHAPTER
. . . . . . . . . . 2.
. . REQUIREMENTS
. . . . . . . . . . . . . . . . . .FOR
. . . . .INSTALLING
. . . . . . . . . . . . . RED
. . . . . HAT
. . . . . CEPH
. . . . . . .STORAGE
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6. . . . . . . . . . . . .
2.1. PREREQUISITES 6
2.2. REQUIREMENTS CHECKLIST FOR INSTALLING RED HAT CEPH STORAGE 6
2.3. OPERATING SYSTEM REQUIREMENTS FOR RED HAT CEPH STORAGE 7
2.4. REGISTERING RED HAT CEPH STORAGE NODES TO THE CDN AND ATTACHING SUBSCRIPTIONS 8
Prerequisites 8
Procedure 8
Additional Resources 9
2.5. ENABLING THE RED HAT CEPH STORAGE REPOSITORIES 9
Prerequisites 9
Procedure 9
Additional Resources 10
2.6. CONSIDERATIONS FOR USING A RAID CONTROLLER WITH OSD NODES (OPTIONAL) 10
2.7. CONSIDERATIONS FOR USING NVME WITH OBJECT GATEWAY (OPTIONAL) 10
2.8. VERIFYING THE NETWORK CONFIGURATION FOR RED HAT CEPH STORAGE 11
Prerequisites 11
Procedure 11
Additional Resources 11
2.9. CONFIGURING A FIREWALL FOR RED HAT CEPH STORAGE 11
2.10. CREATING AN ANSIBLE USER WITH SUDO ACCESS 15
2.11. ENABLING PASSWORD-LESS SSH FOR ANSIBLE 17
Prerequisites 17
Procedure 17
Additional Resources 18
.CHAPTER
. . . . . . . . . . 3.
. . DEPLOYING
. . . . . . . . . . . . . .RED
. . . . .HAT
. . . . .CEPH
. . . . . .STORAGE
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
..............
3.1. PREREQUISITES 19
3.2. INSTALLING A RED HAT CEPH STORAGE CLUSTER 19
Prerequisites 19
Procedure 20
3.3. CONFIGURING OSD ANSIBLE SETTINGS FOR ALL NVME STORAGE 31
3.4. INSTALLING METADATA SERVERS 32
3.5. INSTALLING THE CEPH CLIENT ROLE 33
Prerequisites 33
Procedure 34
Additional Resources 35
3.6. INSTALLING THE CEPH OBJECT GATEWAY 35
Prerequisites 35
Procedure 35
Additional Resources 37
3.6.1. Configuring a multisite Ceph Object Gateway 37
3.7. INSTALLING THE NFS-GANESHA GATEWAY 39
Prerequisites 39
Procedure 39
Additional Resources 40
3.8. UNDERSTANDING THE LIMIT OPTION 41
3.9. ADDITIONAL RESOURCES 41
.CHAPTER
. . . . . . . . . . 4.
. . .UPGRADING
. . . . . . . . . . . . .A
. . RED
. . . . .HAT
. . . . .CEPH
. . . . . . STORAGE
. . . . . . . . . . . CLUSTER
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42
..............
Prerequisites 43
1
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
. . . . . . . . . . . 5.
CHAPTER . . WHAT
. . . . . . . TO
. . . .DO
. . . .NEXT?
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49
..............
.APPENDIX
. . . . . . . . . . .A.
. . TROUBLESHOOTING
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .50
..............
A.1. ANSIBLE STOPS INSTALLATION BECAUSE IT DETECTS LESS DEVICES THAN IT EXPECTED 50
.APPENDIX
. . . . . . . . . . .B.
. . MANUALLY
. . . . . . . . . . . . . INSTALLING
. . . . . . . . . . . . . .RED
. . . . .HAT
. . . . CEPH
. . . . . . .STORAGE
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
..............
B.1. PREREQUISITES 51
Configuring the Network Time Protocol for Red Hat Ceph Storage 51
Prerequisites 51
Procedure: Configuring the Network Time Protocol for RHCS 51
Additional Resources 52
Monitor Bootstrapping 52
B.2. MANUALLY INSTALLING CEPH MANAGER 58
OSD Bootstrapping 59
.APPENDIX
. . . . . . . . . . .C.
. . .INSTALLING
. . . . . . . . . . . . .THE
. . . . .CEPH
. . . . . . COMMAND
. . . . . . . . . . . . LINE
. . . . . .INTERFACE
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .65
..............
Prerequisites 65
Procedure 65
. . . . . . . . . . . .D.
APPENDIX . . .MANUALLY
. . . . . . . . . . . . INSTALLING
. . . . . . . . . . . . . .CEPH
. . . . . . BLOCK
. . . . . . . . DEVICE
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .66
..............
Prerequisites 66
Procedure 66
. . . . . . . . . . . .E.
APPENDIX . . MANUALLY
. . . . . . . . . . . . .INSTALLING
. . . . . . . . . . . . . .CEPH
. . . . . .OBJECT
. . . . . . . . . GATEWAY
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .69
..............
Prerequisites 69
Procedure 69
Additional Details 71
. . . . . . . . . . . .F.
APPENDIX . . OVERRIDING
. . . . . . . . . . . . . . CEPH
. . . . . . .DEFAULT
. . . . . . . . . . SETTINGS
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72
..............
. . . . . . . . . . . .G.
APPENDIX . . .MANUALLY
. . . . . . . . . . . . UPGRADING
. . . . . . . . . . . . . .FROM
. . . . . . .RED
. . . . .HAT
. . . . .CEPH
. . . . . . STORAGE
. . . . . . . . . . .2. .TO
. . . 3. . . . . . . . . . . . . . . . . . . . . . . . . . . .73
..............
Upgrading Monitor Nodes 74
Procedure 74
G.1. MANUALLY INSTALLING CEPH MANAGER 75
Upgrading OSD Nodes 77
Prerequisites 77
Procedure 77
Additional Resources 79
Upgrading the Ceph Object Gateway Nodes 79
Prerequisites 79
Procedure 80
See Also 81
Upgrading a Ceph Client Node 81
Prerequisites 81
Procedure 81
. . . . . . . . . . . .H.
APPENDIX . . .CHANGES
. . . . . . . . . . .IN
. . ANSIBLE
. . . . . . . . . . VARIABLES
. . . . . . . . . . . . .BETWEEN
. . . . . . . . . . .VERSION
. . . . . . . . . .2. AND
. . . . . .3. . . . . . . . . . . . . . . . . . . . . . . . . . . . .83
..............
. . . . . . . . . . . .I.. .IMPORTING
APPENDIX . . . . . . . . . . . . AN
. . . .EXISTING
. . . . . . . . . . .CEPH
. . . . . .CLUSTER
. . . . . . . . . . TO
. . . .ANSIBLE
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .84
..............
. . . . . . . . . . . .J.
APPENDIX . . PURGING
. . . . . . . . . . .A. .CEPH
. . . . . . CLUSTER
. . . . . . . . . . .BY
. . . USING
. . . . . . . ANSIBLE
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85
..............
2
Table of Contents
3
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
Red Hat Ceph Storage is designed for cloud infrastructure and web-scale object storage. Red Hat Ceph
Storage clusters consist of the following types of nodes:
Optionally, local repositories for installing Ceph on nodes that cannot access the Internet for
security reasons
Monitor nodes
Each monitor node runs the monitor daemon (ceph-mon), which maintains a master copy of the
cluster map. The cluster map includes the cluster topology. A client connecting to the Ceph cluster
retrieves the current copy of the cluster map from the monitor which enables the client to read from
and write data to the cluster.
IMPORTANT
Ceph can run with one monitor; however, to ensure high availability in a production
cluster, Red Hat will only support deployments with at least three monitor nodes. Red Hat
recommends deploying a total of 5 Ceph Monitors for storage clusters exceeding 750
OSDs.
OSD nodes
Each Object Storage Device (OSD) node runs the Ceph OSD daemon (ceph-osd), which interacts
with logical disks attached to the node. Ceph stores data on these OSD nodes.
Ceph can run with very few OSD nodes, which the default is three, but production clusters realize
better performance beginning at modest scales, for example 50 OSDs in a storage cluster. Ideally, a
Ceph cluster has multiple OSD nodes, allowing isolated failure domains by creating the CRUSH map.
MDS nodes
Each Metadata Server (MDS) node runs the MDS daemon (ceph-mds), which manages metadata
related to files stored on the Ceph File System (CephFS). The MDS daemon also coordinates access
to the shared cluster.
Object Gateway node
Ceph Object Gateway node runs the Ceph RADOS Gateway daemon (ceph-radosgw), and is an
object storage interface built on top of librados to provide applications with a RESTful gateway to
Ceph Storage Clusters. The Ceph Object Gateway supports two interfaces:
S3
Provides object storage functionality with an interface that is compatible with a large subset of the
Amazon S3 RESTful API.
4
CHAPTER 1. WHAT IS RED HAT CEPH STORAGE?
Swift
Provides object storage functionality with an interface that is compatible with a large subset of the
OpenStack Swift API.
For details on the Ceph architecture, see the Architecture Guide for Red Hat Ceph Storage 3.
For minimum recommended hardware, see the Red Hat Ceph Storage Hardware Selection Guide 3.
5
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
Before installing Red Hat Ceph Storage (RHCS), review the following requirements and prepare each
Monitor, OSD, Metadata Server, and client nodes accordingly.
2.1. PREREQUISITES
Verify the hardware meets the minimum requirements. For details, see the Hardware Guide for
Red Hat Ceph Storage 3.
6
CHAPTER 2. REQUIREMENTS FOR INSTALLING RED HAT CEPH STORAGE
Creating an Ansible Yes Section 2.10, Creating the Ansible user is required
user “Creating an Ansible on all Ceph nodes.
user with sudo
access”
NOTE
IMPORTANT
Red Hat Ceph Storage 3 is not supported on Red Hat Enterprise Linux 8.
IMPORTANT
Red Hat does not support clusters with heterogeneous operating systems or versions.
Additional Resources
7
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
NOTE
For RHCS nodes that cannot access the Internet during the installation, provide the
software content by using the Red Hat Satellite server. Alternatively, mount a local Red
Hat Enterprise Linux 7 Server ISO image and point the RHCS nodes to the ISO image.
For additional details, contact Red Hat Support .
For more information on registering Ceph nodes with the Red Hat Satellite server, see
the How to Register Ceph with Satellite 6 and How to Register Ceph with Satellite 5
articles on the Red Hat Customer Portal.
Prerequisites
Procedure
Perform the following steps on all nodes in the storage cluster as the root user.
1. Register the node. When prompted, enter your Red Hat Customer Portal credentials:
# subscription-manager register
# subscription-manager refresh
Replace
5. Disable the default software repositories. Then, enable the Red Hat Enterprise Linux 7 Server,
Red Hat Enterprise Linux 7 Server Extras, and RHCS repositories:
8
CHAPTER 2. REQUIREMENTS FOR INSTALLING RED HAT CEPH STORAGE
# yum update
Additional Resources
See the Registering a System and Managing Subscriptions chapter in the System
Administrator’s Guide for Red Hat Enterprise Linux 7.
Local Repository
For Ceph Storage clusters where security measures preclude nodes from accessing the
internet, install Red Hat Ceph Storage 3.3 from a single software build delivered as an ISO
image, which will allow you to install local repositories.
Prerequisites
For CDN installations, RHCS nodes must be able to connect to the internet.
Procedure
For CDN installations:
On the Ansible administration node, enable the Red Hat Ceph Storage 3 Tools repository and Ansible
repository:
9
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
3. In the Red Hat Ceph Storage area, click Download Software to download the latest version of
the software.
Additional Resources
The Registering and Managing Subscriptions chapter in the System Administrator’s Guide for
Red Hat Enterprise Linux.
Modern RAID controllers usually have super capacitors that provide enough power to drain volatile
memory to non-volatile NAND memory during a power loss event. It is important to understand how a
particular controller and its firmware behave after power is restored.
Some RAID controllers require manual intervention. Hard drives typically advertise to the operating
system whether their disk caches should be enabled or disabled by default. However, certain RAID
controllers and some firmware do not provide such information. Verify that disk level caches are
disabled to avoid file system corruption.
Create a single RAID 0 volume with write-back for each Ceph OSD data drive with write-back cache
enabled.
If Serial Attached SCSI (SAS) or SATA connected Solid-state Drive (SSD) disks are also present on the
RAID controller, then investigate whether the controller and firmware support pass-through mode.
Enabling pass-through mode helps avoid caching logic, and generally results in much lower latency for
fast media.
You might have a network interface card for a cluster network so that Ceph can conduct heart-beating,
peering, replication, and recovery on a network separate from the public network.
Configure the network interface settings and ensure to make the changes persistent.
IMPORTANT
Red Hat does not recommend using a single network interface card for both a public and
private network.
Prerequisites
Procedure
Do the following steps on all RHCS nodes in the storage cluster, as the root user.
c. If you intend to use IPv6 addressing, you must set the IPv6 parameters such as IPV6INIT to
yes, except the IPV6_FAILURE_FATAL parameter.
Also, edit the Ceph configuration file, /etc/ceph/ceph.conf, to instruct Ceph to use IPv6,
otherwise, Ceph will use IPv4.
Additional Resources
For details on configuring network interface scripts for Red Hat Enterprise Linux 7, see the
Configuring a Network Interface Using ifcfg Files chapter in the Networking Guide for Red Hat
Enterprise Linux 7.
For more information on network configuration see the Network Configuration Reference
chapter in the Configuration Guide for Red Hat Ceph Storage 3.
The Monitor daemons use port 6789 for communication within the Ceph storage cluster.
On each Ceph OSD node, the OSD daemons use several ports in the range 6800-7300:
11
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
One for communicating with clients and monitors over the public network
One for sending data to other OSDs over a cluster network, if available; otherwise, over the
public network
One for exchanging heartbeat packets over a cluster network, if available; otherwise, over the
public network
The Ceph Manager (ceph-mgr) daemons use ports in range 6800-7300. Consider colocating the ceph-
mgr daemons with Ceph Monitors on same nodes.
The Ceph Metadata Server nodes (ceph-mds) use ports in the range 6800-7300.
The Ceph Object Gateway nodes are configured by Ansible to use port 8080 by default. However, you
can change the default port, for example to port 80.
Prerequisite
Procedure
Run the following commands as the root user.
1. On all RHCS nodes, start the firewalld service. Enable it to run on boot, and ensure that it is
running:
Replace
Example
12
CHAPTER 2. REQUIREMENTS FOR INSTALLING RED HAT CEPH STORAGE
If you have a separate cluster network, repeat the commands with the appropriate zone.
4. On all Ceph Manager (ceph-mgr) nodes (usually the same nodes as Monitor ones), open ports
6800-7300 on the public network:
If you have a separate cluster network, repeat the commands with the appropriate zone.
5. On all Ceph Metadata Server (ceph-mds) nodes, open port 6800 on the public network:
If you have a separate cluster network, repeat the commands with the appropriate zone.
6. On all Ceph Object Gateway nodes, open the relevant port or ports on the public network.
Replace
13
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
Example
b. Optional. If you installed Ceph Object Gateway using Ansible and changed the default port
that Ansible configures Ceph Object Gateway to use from 8080, for example, to port 80,
open this port:
To limit access based on the source address, run the following commands:
Replace
Example
To limit access based on the source address, run the following commands:
14
CHAPTER 2. REQUIREMENTS FOR INSTALLING RED HAT CEPH STORAGE
Replace
Example
Additional Resources
For more information about public and cluster network, see Verifying the Network
Configuration for Red Hat Ceph Storage.
For additional details on firewalld, see the Using Firewalls chapter in the Security Guide for Red
Hat Enterprise Linux 7.
Prerequisite
Procedure
ssh root@$HOST_NAME
Replace
15
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
Example
# ssh root@mon01
adduser $USER_NAME
Replace
$USER_NAME with the new user name for the Ansible user.
Example
# adduser admin
IMPORTANT
Do not use ceph as the user name. The ceph user name is reserved for the Ceph
daemons. A uniform user name across the cluster can improve ease of use, but
avoid using obvious user names, because intruders typically use them for brute-
force attacks.
# passwd $USER_NAME
Replace
$USER_NAME with the new user name for the Ansible user.
Example
# passwd admin
Replace
$USER_NAME with the new user name for the Ansible user.
Example
16
CHAPTER 2. REQUIREMENTS FOR INSTALLING RED HAT CEPH STORAGE
Replace
$USER_NAME with the new user name for the Ansible user.
Example
Additional Resources
The Adding a New User section in the System Administrator’s Guide for Red Hat Enterprise
Linux 7.
Prerequisites
Procedure
Do the following steps from the Ansible administration node, and as the Ansible user.
1. Generate the SSH key pair, accept the default file name and leave the passphrase empty:
ssh-copy-id $USER_NAME@$HOST_NAME
Replace
$USER_NAME with the new user name for the Ansible user.
Example
17
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
IMPORTANT
By creating and editing the ~/.ssh/config file you do not have to specify the -u
$USER_NAME option each time you execute the ansible-playbook command.
b. Open the config file for editing. Set the Hostname and User options for each node in the
storage cluster:
Host node1
Hostname $HOST_NAME
User $USER_NAME
Host node2
Hostname $HOST_NAME
User $USER_NAME
...
Replace
$USER_NAME with the new user name for the Ansible user.
Example
Host node1
Hostname monitor
User admin
Host node2
Hostname osd
User admin
Host node3
Hostname gateway
User admin
Additional Resources
The OpenSSH chapter in the System Administrator’s Guide for Red Hat Enterprise Linux 7
18
CHAPTER 3. DEPLOYING RED HAT CEPH STORAGE
To install a Red Hat Ceph Storage cluster, see Section 3.2, “Installing a Red Hat Ceph Storage
Cluster”.
To install the ceph-client role, see Section 3.5, “Installing the Ceph Client Role” .
To install the Ceph Object Gateway, see Section 3.6, “Installing the Ceph Object Gateway” .
To configure a multisite Ceph Object Gateway, see Section 3.6.1, “Configuring a multisite Ceph
Object Gateway”.
To learn about the Ansible --limit option, see Section 3.8, “Understanding the limit option”.
3.1. PREREQUISITES
Obtain a valid customer subscription.
Register the node to the Content Delivery Network (CDN) and attach subscriptions .
Production Ceph storage clusters start with a minimum of three monitor hosts and three OSD nodes
containing multiple OSD daemons.
Prerequisites
Using the root account on the Ansible administration node, install the ceph-ansible package:
19
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
Using the root account on the Ansible administration node, install the ceph-ansible package:
Procedure
Run the following commands from the Ansible administration node unless instructed otherwise.
1. As the Ansible user, create the ceph-ansible-keys directory where Ansible stores temporary
values generated by the ceph-ansible playbook.
a. Edit the group_vars/all.yml file. See the table below for the most common required and
optional parameters to uncomment. Note that the table does not include all parameters.
IMPORTANT
Do not set the cluster: ceph parameter to any value other than ceph
because using custom cluster names is not supported.
20
CHAPTER 3. DEPLOYING RED HAT CEPH STORAGE
ceph_rhcs_versio 3 Yes
n
21
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
ceph_origin: distro
ceph_repository: rhcs
ceph_repository_type: cdn
ceph_rhcs_version: 3
monitor_interface: eth0
public_network: 192.168.0.0/24
NOTE
Be sure to set ceph_origin to distro in the all.yml file. This ensures that the
installation process uses the correct download repository.
NOTE
Having the ceph_rhcs_version option set to 3 will pull in the latest version
of Red Hat Ceph Storage 3.
WARNING
22
CHAPTER 3. DEPLOYING RED HAT CEPH STORAGE
b. Edit the group_vars/osds.yml file. See the table below for the most common required and
optional parameters to uncomment. Note that the table does not include all parameters.
IMPORTANT
Use a different physical device to install an OSD than the device where the
operating system is installed. Sharing the same device between the
operating system and OSDs causes performance issues.
devices List of devices where Yes to specify the list Cannot be used
ceph data is stored of devices when
osd_auto_discov
ery setting is used.
When using lvm as
the osd_scenario
and setting the
devices option,
ceph-volume lvm
batch mode creates
the optimized OSD
configuration.
23
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
The following are examples of the osds.yml file when using the three OSD scenarios:
collocated, non-collocated, and lvm. The default OSD object store format is BlueStore, if
not specified.
24
CHAPTER 3. DEPLOYING RED HAT CEPH STORAGE
Collocated
osd_objectstore: filestore
osd_scenario: collocated
devices:
- /dev/sda
- /dev/sdb
Non-collocated - BlueStore
osd_objectstore: bluestore
osd_scenario: non-collocated
devices:
- /dev/sda
- /dev/sdb
- /dev/sdc
- /dev/sdd
dedicated_devices:
- /dev/nvme0n1
- /dev/nvme0n1
- /dev/nvme1n1
- /dev/nvme1n1
This non-collocated example will create four BlueStore OSDs, one per device. In this
example, the traditional hard drives (sda, sdb, sdc, sdd) are used for object data, and the
solid state drives (SSDs) (/dev/nvme0n1, /dev/nvme1n1) are used for the BlueStore
databases and write-ahead logs. This configuration pairs the /dev/sda and /dev/sdb
devices with the /dev/nvme0n1 device, and pairs the /dev/sdc and /dev/sdd devices with
the /dev/nvme1n1 device.
Non-collocated - FileStore
osd_objectstore: filestore
osd_scenario: non-collocated
devices:
- /dev/sda
- /dev/sdb
- /dev/sdc
- /dev/sdd
dedicated_devices:
- /dev/nvme0n1
- /dev/nvme0n1
- /dev/nvme1n1
- /dev/nvme1n1
LVM simple
osd_objectstore: bluestore
osd_scenario: lvm
devices:
- /dev/sda
- /dev/sdb
or
25
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
osd_objectstore: bluestore
osd_scenario: lvm
devices:
- /dev/sda
- /dev/sdb
- /dev/nvme0n1
With these simple configurations ceph-ansible uses batch mode (ceph-volume lvm batch)
to create the OSDs.
In the first scenario, if the devices are traditional hard drives or SSDs, then one OSD per
device is created.
In the second scenario, when there is a mix of traditional hard drives and SSDs, the data is
placed on the traditional hard drives (sda, sdb) and the BlueStore database (block.db) is
created as large as possible on the SSD (nvme0n1).
LVM advance
osd_objectstore: filestore
osd_scenario: lvm
lvm_volumes:
- data: data-lv1
data_vg: vg1
journal: journal-lv1
journal_vg: vg2
- data: data-lv2
journal: /dev/sda
data_vg: vg1
or
osd_objectstore: bluestore
osd_scenario: lvm
lvm_volumes:
- data: data-lv1
data_vg: data-vg1
db: db-lv1
db_vg: db-vg1
wal: wal-lv1
wal_vg: wal-vg1
- data: data-lv2
data_vg: data-vg2
db: db-lv2
db_vg: db-vg2
wal: wal-lv2
wal_vg: wal-vg2
With these advance scenario examples, the volume groups and logical volumes must be
created beforehand. They will not be created by ceph-ansible.
NOTE
26
CHAPTER 3. DEPLOYING RED HAT CEPH STORAGE
NOTE
If using all NVMe SSDs set the osd_scenario: lvm and osds_per_device: 4
options. For more information, see Configuring OSD Ansible settings for all
NVMe Storage for Red Hat Enterprise Linux or Configuring OSD Ansible
settings for all NVMe Storage for Ubuntu in the Red Hat Ceph Storage
Installation Guides.
6. Edit the Ansible inventory file located by default at /etc/ansible/hosts. Remember to comment
out example hosts.
[mons]
<monitor-host-name>
<monitor-host-name>
<monitor-host-name>
b. Add OSD nodes under the [osds] section. If the nodes have sequential naming, consider
using a range:
[osds]
<osd-host-name[1:10]>
NOTE
For OSDs in a new installation, the default object store format is BlueStore.
Optionally, use the devices parameter to specify devices that the OSD nodes will use. Use
a comma-separated list to list multiple devices.
[osds]
<ceph-host-name> devices="[ '<device_1>', '<device_2>' ]"
For example:
[osds]
ceph-osd-01 devices="[ '/dev/sdb', '/dev/sdc' ]"
ceph-osd-02 devices="[ '/dev/sdb', '/dev/sdc', '/dev/sdd' ]"
When specifying no devices, set the osd_auto_discovery option to true in the osds.yml
file.
NOTE
Using the devices parameter is useful when OSDs use devices with different
names or when one of the devices failed on one of the OSDs.
7. Optionally, for all deployments, bare-metal or in containers, you can create a custom CRUSH
hierarchy using ansible-playbook:
27
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
a. Setup your Ansible inventory file. Specify where you want the OSD hosts to be in the
CRUSH map’s hierarchy by using the osd_crush_location parameter. You must specify at
least two CRUSH bucket types to specify the location of the OSD, and one bucket type
must be host. By default, these include root, datacenter, room, row, pod, pdu, rack,
chassis and host.
Syntax
[osds]
CEPH_OSD_NAME osd_crush_location="{ 'root': ROOT_BUCKET_', 'rack':
'RACK_BUCKET', 'pod': 'POD_BUCKET', 'host': 'CEPH_HOST_NAME' }"
Example
[osds]
ceph-osd-01 osd_crush_location="{ 'root': 'default', 'rack': 'rack1', 'pod': 'monpod', 'host':
'ceph-osd-01' }"
b. Set the crush_rule_config and create_crush_tree parameters to True, and create at least
one CRUSH rule if you do not want to use the default CRUSH rules. For example, if you are
using HDD devices, edit the paramters as follows:
crush_rule_config: True
crush_rule_hdd:
name: replicated_hdd_rule
root: root-hdd
type: host
class: hdd
default: True
crush_rules:
- "{{ crush_rule_hdd }}"
create_crush_tree: True
If you are using SSD devices, then edit the parameters as follows:
crush_rule_config: True
crush_rule_ssd:
name: replicated_ssd_rule
root: root-ssd
type: host
class: ssd
default: True
crush_rules:
- "{{ crush_rule_ssd }}"
create_crush_tree: True
NOTE
The default CRUSH rules fail if both ssd and hdd OSDs are not deployed
because the default rules now include the class parameter, which must be
defined.
NOTE
28
CHAPTER 3. DEPLOYING RED HAT CEPH STORAGE
NOTE
Add the custom CRUSH hierarchy also to the OSD files in the host_vars
directory as described in a step below to make this configuration work.
Example
>>>>>>> 3993c70c7f25ab628cbfd9c8e27623403ca18c99
copy_admin_key: True
user_config: True
pool1:
name: "pool1"
pg_num: 128
pgp_num: 128
rule_name: "HDD"
type: "replicated"
device_class: "hdd"
pools:
- "{{ pool1 }}"
# for i in $(rados lspools);do echo "pool: $i"; ceph osd pool get $i crush_rule;done
pool: pool1
crush_rule: HDD
1. For all deployments, bare-metal or in containers, open for editing the Ansible inventory
file, by default the /etc/ansible/hosts file. Comment out the example hosts.
c. Add a node under [grafana-server]. This role installs Grafana and Prometheus to provide real-
time insights into the performance of the Ceph cluster. These metrics are presented in the
Ceph Dashboard, which allows you to monitor and manage the cluster. The installation of the
dashboard, Grafana, and Prometheus are required. You can colocate the metrics functions on
the Ansible Administration node. If you do, ensure the system resources of the node are greater
than than what is required for a stand alone metrics node .
[grafana-server]
GRAFANA-SERVER_NODE_NAME
d. Add the Ceph Manager (ceph-mgr) nodes under the [mgrs] section. Colocate the Ceph
Manager daemon with Monitor nodes.
[mgrs]
<monitor-host-name>
<monitor-host-name>
<monitor-host-name>
29
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
1. As the Ansible user, ensure that Ansible can reach the Ceph hosts:
retry_files_save_path = ~/
3. As root, create the /var/log/ansible/ directory and assign the appropriate permissions for
the ansible user:
log_path = /var/log/ansible/ansible.log
NOTE
1. Using the root account on a Monitor node, verify the status of the Ceph cluster:
f. From a monitor node, create a test pool with eight placement groups:
Syntax
Example
30
CHAPTER 3. DEPLOYING RED HAT CEPH STORAGE
Example
h. Upload hello-world.txt to the test pool using the object name hello-world:
Syntax
Example
Example
"Hello World!"
NOTE
In addition to verifying the cluster status, you can use the ceph-medic utility to
overall diagnose the Ceph Storage Cluster. See the Installing and Using ceph-
medic to Diagnose a Ceph Storage Cluster chapter in the Red Hat Ceph Storage
3 Troubleshooting Guide.
NOTE
31
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
NOTE
If you mix SSDs and HDDs, then SSDs will be used for either journals or block.db, not
OSDs.
NOTE
In testing, configuring four OSDs on each NVMe device was found to provide optimal
performance. It is recommended to set osds_per_device: 4, but it is not required. Other
values may provide better performance in your environment.
Prerequisites
Procedure
osd_scenario: lvm
osds_per_device: 4
devices:
- /dev/nvme0n1
- /dev/nvme1n1
- /dev/nvme2n1
- /dev/nvme3n1
osd_scenario: lvm
osds_per_device: 4
devices:
- /dev/nvme0n1
- /dev/nvme1n1
- /dev/nvme2n1
- /dev/nvme3n1
NOTE
You must use devices with this configuration, not lvm_volumes. This is because
lvm_volumes is generally used with pre-created logical volumes and osds_per_device
implies automatic logical volume creation by Ceph.
Additional Resources
Installing a Red Hat Ceph Storage Cluster on Red Hat Enterprise Linux
32
CHAPTER 3. DEPLOYING RED HAT CEPH STORAGE
Use the Ansible automation application to install a Ceph Metadata Server (MDS). Metadata Server
daemons are necessary for deploying a Ceph File System.
Prerequisites
Procedure
Perform the following steps on the Ansible administration node.
[mdss]
hostname
hostname
hostname
Replace hostname with the host names of the nodes where you want to install the Ceph
Metadata Servers.
5. After installing Metadata Servers, configure them. For details, see the Configuring Metadata
Server Daemons chapter in the Ceph File System Guide for Red Hat Ceph Storage 3.
Additional Resources
The Ceph File System Guide for Red Hat Ceph Storage 3
Prerequisites
33
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
Perform the tasks listed in Chapter 2, Requirements for Installing Red Hat Ceph Storage .
Procedure
Perform the following tasks on the Ansible administration node.
[clients]
<client-hostname>
Replace <client-hostname> with the host name of the node where you want to install the ceph-
client role.
keys:
- { name: client.test, caps: { mon: "allow r", osd: "allow class-read object_prefix
rbd_children, allow rwx pool=test" }, mode: "{{ ceph_keyring_permissions }}" }
a. Replace client.test with the real client name, and add the client key to the client definition
line, for example:
key: "ADD-KEYRING-HERE=="
NOTE
a. Update clients.yml.
Uncomment the pools and keys sections and update them as required. You can define
custom pools and client names altogether with the cephx capabilities.
34
CHAPTER 3. DEPLOYING RED HAT CEPH STORAGE
ceph_conf_overrides:
global:
osd_pool_default_pg_num: <number>
Additional Resources
Prerequisites
A running Red Hat Ceph Storage cluster, preferably in the active + clean state.
On the Ceph Object Gateway node, perform the tasks listed in Chapter 2, Requirements for
Installing Red Hat Ceph Storage.
Procedure
Perform the following tasks on the Ansible administration node.
1. Add gateway hosts to the /etc/ansible/hosts file under the [rgws] section to identify their roles
to Ansible. If the hosts have sequential naming, use a range, for example:
[rgws]
<rgw_host_name_1>
<rgw_host_name_2>
<rgw_host_name[3..10]>
4. Open and edit the group_vars/rgws.yml file. To copy the administrator key to the Ceph
Object Gateway node, uncomment the copy_admin_key option:
copy_admin_key: true
5. The rgws.yml file may specify a different default port than the default port 7480. For example:
ceph_rgw_civetweb_port: 80
35
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
radosgw_interface: eth0
Specifying the interface prevents Civetweb from binding to the same IP address as another
Civetweb instance when running multiple instances on the same host.
7. Generally, to change default settings, uncomment the settings in the rgw.yml file, and make
changes accordingly. To make additional changes to settings that are not in the rgw.yml file,
use ceph_conf_overrides: in the all.yml file. For example, set the rgw_dns_name: with the
host of the DNS server and ensure the cluster’s DNS server to configure it for wild cards to
enable S3 subdomains.
ceph_conf_overrides:
client.rgw.rgw1:
rgw_dns_name: <host_name>
rgw_override_bucket_index_max_shards: 16
rgw_bucket_default_quota_max_objects: 1638400
For advanced configuration details, see the Red Hat Ceph Storage 3 Ceph Object Gateway for
Production guide. Advanced topics include:
Developing Storage Strategies . See the Creating the Root Pool , Creating System Pools, and
Creating Data Placement Strategies sections for additional details on how create and
configure the pools.
See Bucket Sharding for configuration details on bucket sharding.
radosgw_interface: <interface>
Replace:
<interface> with the interface that the Ceph Object Gateway nodes listen to
NOTE
For a single site configuration, add Ceph Object Gateways to the Ansible configuration.
For multi-site deployments, you should have an Ansible configuration for each zone. That is, Ansible will
create a Ceph storage cluster and gateway instances for that zone.
After installation for a multi-site cluster is complete, proceed to the Multi-site chapter in the Object
Gateway Guide for Red Hat Enterprise Linux for details on configuring a cluster for multi-site.
36
CHAPTER 3. DEPLOYING RED HAT CEPH STORAGE
Additional Resources
Prerequisites
On the Ceph Object Gateway node, perform the tasks listed in the Requirements for Installing
Red Hat Ceph Storage found in the Red Hat Ceph Storage Installation Guide.
Install and configure one Ceph Object Gateway per storage cluster.
Procedure
1. Do the following steps on Ansible node for the primary storage cluster:
a. Generate the system keys and capture their output in the multi-site-keys.txt file:
c. Open and edit the group_vars/all.yml file. Enable multisite support by adding the following
options, along with updating the $ZONE_NAME, $ZONE_GROUP_NAME,
$REALM_NAME, $ACCESS_KEY, and $SECRET_KEY values accordingly.
When more than one Ceph Object Gateway is in the master zone, then the
rgw_multisite_endpoints option needs to be set. The value for the
rgw_multisite_endpoints option is a comma separated list, with no spaces.
Example
rgw_multisite: true
rgw_zone: $ZONE_NAME
rgw_zonemaster: true
rgw_zonesecondary: false
rgw_multisite_endpoint_addr: "{{ ansible_fqdn }}"
rgw_multisite_endpoints:
http://foo.example.com:8080,http://bar.example.com:8080,http://baz.example.com:8080
rgw_zonegroup: $ZONE_GROUP_NAME
rgw_zone_user: zone.user
37
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
rgw_realm: $REALM_NAME
system_access_key: $ACCESS_KEY
system_secret_key: $SECRET_KEY
NOTE
NOTE
2. Do the following steps on Ansible node for the secondary storage cluster:
b. Open and edit the group_vars/all.yml file. Enable multisite support by adding the following
options, along with updating the $ZONE_NAME, $ZONE_GROUP_NAME,
$REALM_NAME, $ACCESS_KEY, and $SECRET_KEY values accordingly: The
rgw_zone_user, system_access_key, and system_secret_key must be the same value as
used in the master zone configuration. The rgw_pullhost option must be the Ceph Object
Gateway for the master zone.
When more than one Ceph Object Gateway is in the secondary zone, then the
rgw_multisite_endpoints option needs to be set. The value for the
rgw_multisite_endpoints option is a comma separated list, with no spaces.
Example
rgw_multisite: true
rgw_zone: $ZONE_NAME
rgw_zonemaster: false
rgw_zonesecondary: true
rgw_multisite_endpoint_addr: "{{ ansible_fqdn }}"
rgw_multisite_endpoints:
http://foo.example.com:8080,http://bar.example.com:8080,http://baz.example.com:8080
rgw_zonegroup: $ZONE_GROUP_NAME
rgw_zone_user: zone.user
rgw_realm: $REALM_NAME
system_access_key: $ACCESS_KEY
system_secret_key: $SECRET_KEY
38
CHAPTER 3. DEPLOYING RED HAT CEPH STORAGE
rgw_pull_proto: http
rgw_pull_port: 8080
rgw_pullhost: $MASTER_RGW_NODE_NAME
NOTE
NOTE
3. After running the Ansible playbook on the master and secondary storage clusters, you will have a
running active-active Ceph Object Gateway configuration.
a. From the Ceph Monitor and Object Gateway nodes at each site, primary and secondary,
must be able to curl the other site.
Prerequisites
Procedure
Perform the following tasks on the Ansible administration node.
39
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
2. Add gateway hosts to the /etc/ansible/hosts file under an [nfss] group to identify their group
membership to Ansible. If the hosts have sequential naming, use a range. For example:
[nfss]
<nfs_host_name_1>
<nfs_host_name_2>
<nfs_host_name[3..10]>
4. To copy the administrator key to the Ceph Object Gateway node, uncomment the
copy_admin_key setting in the /usr/share/ceph-ansible/group_vars/nfss.yml file:
copy_admin_key: true
5. Configure the FSAL (File System Abstraction Layer) sections of the /usr/share/ceph-
ansible/group_vars/nfss.yml file. Provide an ID, S3 user ID, S3 access key and secret. For
NFSv4, it should look something like this:
###################
# FSAL RGW Config #
###################
#ceph_nfs_rgw_export_id: <replace-w-numeric-export-id>
#ceph_nfs_rgw_pseudo_path: "/"
#ceph_nfs_rgw_protocols: "3,4"
#ceph_nfs_rgw_access_type: "RW"
#ceph_nfs_rgw_user: "cephnfs"
# Note: keys are optional and can be generated, but not on containerized, where
# they must be configered.
#ceph_nfs_rgw_access_key: "<replace-w-access-key>"
#ceph_nfs_rgw_secret_key: "<replace-w-secret-key>"
WARNING
Additional Resources
40
CHAPTER 3. DEPLOYING RED HAT CEPH STORAGE
Ansible supports the --limit option that enables you to use the site, site-docker, and rolling_upgrade
Ansible playbooks for a particular section of the inventory file.
For example, to redeploy only OSDs on bare metal, run the following command as the Ansible user:
IMPORTANT
If you colocate Ceph components on one node, Ansible applies a playbook to all
components on the node despite that only one component type was specified with the
limit option. For example, if you run the rolling_update playbook with the --limit osds
option on a node that contains OSDs and Metadata Servers (MDS), Ansible will upgrade
both components, OSDs and MDSs.
41
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
To upgrade a storage cluster, see Section 4.1, “Upgrading the Storage Cluster” .
To upgrade Red Hat Ceph Storage Dashboard, see Section 4.2, “Upgrading Red Hat Ceph
Storage Dashboard”.
Monitor nodes
MGR nodes
OSD nodes
MDS nodes
NOTE
Red Hat Ceph Storage 3 introduces several changes in Ansible configuration files located
in the /usr/share/ceph-ansible/group_vars/ directory; certain parameters were renamed
or removed. Therefore, make backup copies of the all.yml and osds.yml files before
creating new copies from the all.yml.sample and osds.yml.sample files after upgrading
to version 3. For more details about the changes, see Appendix H, Changes in Ansible
Variables Between Version 2 and 3.
NOTE
Red Hat Ceph Storage 3.1 and later introduces new Ansible playbooks to optimize storage
for performance when using Object Gateway and high speed NVMe based SSDs (and
SATA SSDs). The playbooks do this by placing journals and bucket indexes together on
SSDs, which can increase performance compared to having all journals on one device.
These playbooks are designed to be used when installing Ceph. Existing OSDs continue to
work and need no extra steps during an upgrade. There is no way to upgrade a Ceph
cluster while simultaneously reconfiguring OSDs to optimize storage in this way. To use
different devices for journals or bucket indexes requires reprovisioning OSDs. For more
information see Using NVMe with LVM optimally in Ceph Object Gateway for Production .
IMPORTANT
The rolling_update.yml playbook includes the serial variable that adjusts the number of
nodes to be updated simultaneously. Red Hat strongly recommends to use the default
value (1), which ensures that Ansible will upgrade cluster nodes one by one.
42
CHAPTER 4. UPGRADING A RED HAT CEPH STORAGE CLUSTER
IMPORTANT
When using the rolling_update.yml playbook to upgrade to any Red Hat Ceph Storage
3.x version, users who use the Ceph File System (CephFS) must manually update the
Metadata Server (MDS) cluster. This is due to a known issue.
Comment out the MDS hosts in /etc/ansible/hosts before upgrading the entire cluster
using ceph-ansible rolling-upgrade.yml, and then upgrade MDS manually. In the
/etc/ansible/hosts file:
#[mdss]
#host-abc
For more details about this known issue, including how to update the MDS cluster, refer
to the Red Hat Ceph Storage 3.0 Release Notes.
IMPORTANT
When upgrading a Red Hat Ceph Storage cluster from a previous version to 3.2, the Ceph
Ansible configuration will default the object store type to BlueStore. If you still want to
use FileStore as the OSD object store, then explicitly set the Ceph Ansible configuration
to FileStore. This ensures newly deployed and replaced OSDs are using FileStore.
IMPORTANT
When using the rolling_update.yml playbook to upgrade to any Red Hat Ceph Storage
3.x version, and if you are using a multisite Ceph Object Gateway configuration, then you
do not have to manually update the all.yml file to specify the multisite configuration.
Prerequisites
Log in as the root user on all nodes in the storage cluster.
If the Ceph nodes are not connected to the Red Hat Content Delivery Network (CDN) and you
used an ISO image to install Red Hat Ceph Storage, update the local repository with the latest
version of Red Hat Ceph Storage. See Section 2.5, “Enabling the Red Hat Ceph Storage
Repositories” for details.
If upgrading from Red Hat Ceph Storage 2.x to 3.x, on the Ansible administration node and the
RBD mirroring node, enable the Red Hat Ceph Storage 3 Tools repository:
On the Ansible administration node, ensure the latest version of the ansible and ceph-ansible
packages are installed.
43
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
health_osd_check_retries: 50
health_osd_check_delay: 30
With these values set, for each OSD node, Ansible will wait up to 25 minutes, and will check the
storage cluster health every 30 seconds, waiting before continuing the upgrade process.
NOTE
If the cluster you want to upgrade contains Ceph Block Device images that use the exclusive-
lock feature, ensure that all Ceph Block Device users have permissions to blacklist clients:
ceph auth caps client.<ID> mon 'allow r, allow command "osd blacklist"' osd '<existing-OSD-
user-capabilities>'
2. Skip this step when upgrading from Red Hat Ceph Storage version 3.x to the latest version. Back
up the group_vars/all.yml and group_vars/osds.yml files.
3. Skip this step when upgrading from Red Hat Ceph Storage version 3.x to the latest version.
When upgrading from Red Hat Ceph Storage 2.x to 3.x, create new copies of the
group_vars/all.yml.sample, group_vars/osds.yml.sample and
group_vars/clients.yml.sample files, and rename them to group_vars/all.yml,
group_vars/osds.yml, and group_vars/clients.yml respectively. Open and edit them
accordingly. For details, see Appendix H, Changes in Ansible Variables Between Version 2 and 3
and Section 3.2, “Installing a Red Hat Ceph Storage Cluster” .
4. Skip this step when upgrading from Red Hat Ceph Storage version 3.x to the latest version.
44
CHAPTER 4. UPGRADING A RED HAT CEPH STORAGE CLUSTER
4. Skip this step when upgrading from Red Hat Ceph Storage version 3.x to the latest version.
When upgrading from Red Hat Ceph Storage 2.x to 3.x, open the group_vars/clients.yml file,
and uncomment the following lines:
keys:
- { name: client.test, caps: { mon: "allow r", osd: "allow class-read object_prefix
rbd_children, allow rwx pool=test" }, mode: "{{ ceph_keyring_permissions }}" }
a. Replace client.test with the real client name, and add the client key to the client definition
line, for example:
key: "ADD-KEYRING-HERE=="
NOTE
To get the client key, run the ceph auth get-or-create command to view the
key for the named client.
upgrade_ceph_packages: True
ceph_rhcs_version: 3
NOTE
Having the ceph_rhcs_version option set to 3 will pull in the latest version of
Red Hat Ceph Storage 3.
ceph_origin: distro
fetch_directory: <full_directory_path>
Replace:
<full_directory_path> with a writable location, such as the Ansible user’s home directory.
Provide the existing path that was used for the initial storage cluster installation.
45
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
fsid: <add_the_fsid>
generate_fsid: false
9. If the cluster you want to upgrade contains any Ceph Object Gateway nodes, add the
radosgw_interface parameter to the group_vars/all.yml file.
radosgw_interface: <interface>
Replace:
<interface> with the interface that the Ceph Object Gateway nodes listen to.
10. Starting with Red Hat Ceph Storage 3.2, the default OSD object store is BlueStore. To keep the
traditional OSD object store, you must explicitly set the osd_objectstore option to filestore in
the group_vars/all.yml file.
osd_objectstore: filestore
NOTE
With the osd_objectstore option set to filestore, replacing an OSD will use
FileStore, instead of BlueStore.
11. In the Ansible inventory file located at /etc/ansible/hosts, add the Ceph Manager ( ceph-mgr)
nodes under the [mgrs] section. Colocate the Ceph Manager daemon with Monitor nodes. Skip
this step when upgrading from version 3.x to the latest version.
[mgrs]
<monitor-host-name>
<monitor-host-name>
<monitor-host-name>
13. Create the /var/log/ansible/ directory and assign the appropriate permissions for the ansible
user:
46
CHAPTER 4. UPGRADING A RED HAT CEPH STORAGE CLUSTER
log_path = /var/log/ansible/ansible.log
To use the playbook only for a particular group of nodes on the Ansible inventory file, use the --
limit option. For details, see Section 3.8, “Understanding the limit option”.
15. While logged in as the root user on the RBD mirroring daemon node, upgrade rbd-mirror
manually:
16. Verify that the cluster health is OK. ..Log into a monitor node as the root user and run the ceph
status command.
1. If working in an OpenStack environment, update all the cephx users to use the RBD profile for
pools. The following commands must be run as the root user:
Glance users
ceph auth caps client.glance mon 'profile rbd' osd 'profile rbd pool=<glance-pool-name>'
Example
[root@monitor ~]# ceph auth caps client.glance mon 'profile rbd' osd 'profile rbd
pool=images'
Cinder users
ceph auth caps client.cinder mon 'profile rbd' osd 'profile rbd pool=<cinder-volume-pool-
name>, profile rbd pool=<nova-pool-name>, profile rbd-read-only pool=<glance-pool-
name>'
Example
[root@monitor ~]# ceph auth caps client.cinder mon 'profile rbd' osd 'profile rbd
pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images'
ceph auth caps client.openstack mon 'profile rbd' osd 'profile rbd-read-only pool=<cinder-
volume-pool-name>, profile rbd pool=<nova-pool-name>, profile rbd-read-only pool=
<glance-pool-name>'
47
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
Example
[root@monitor ~]# ceph auth caps client.openstack mon 'profile rbd' osd 'profile rbd-read-
only pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images'
IMPORTANT
Do these CAPS updates before performing any live client migrations. This
allows clients to use the new libraries running in memory, causing the old
CAPS settings to drop from cache and applying the new RBD profile settings.
Before upgrading, ensure Red Hat Ceph Storage is upgraded from version 3.1 to 3.2. See 4.1. Upgrading
the Storage Cluster for instructions.
WARNING
Procedure
1. As the root user, update the cephmetrics-ansible package from the Ansible administration
node:
48
CHAPTER 5. WHAT TO DO NEXT?
Creating and managing snapshots, see the Snapshots chapter in the Block Device Guide for
Red Hat Ceph Storage 3.
Expanding the Red Hat Ceph Storage cluster, see the Managing Cluster Size chapter in the
Administration Guide for Red Hat Ceph Storage 3.
Mirroring Ceph Block Devices, see the Block Device Mirroring chapter in the Block Device Guide
for Red Hat Ceph Storage 3.
Process management, see the Process Management chapter in the Administration Guide for
Red Hat Ceph Storage 3.
Tunable parameters, see the Configuration Guide for Red Hat Ceph Storage 3.
Using Ceph as the back end storage for OpenStack, see the Back-ends section in the Storage
Guide for Red Hat OpenStack Platform.
49
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
APPENDIX A. TROUBLESHOOTING
- name: fix partitions gpt header or labels of the osd disks (autodiscover disks)
shell: "sgdisk --zap-all --clear --mbrtogpt -- '/dev/{{ item.0.item.key }}' || sgdisk --zap-all --clear --
mbrtogpt -- '/dev/{{ item.0.item.key }}'"
with_together:
- "{{ osd_partition_status_results.results }}"
- "{{ ansible_devices }}"
changed_when: false
when:
- ansible_devices is defined
- item.0.item.value.removable == "0"
- item.0.item.value.partitions|count == 0
- item.0.rc != 0
Example situation:
1. Three OSD nodes (host1, host2, host3) use the /dev/sdb, /dev/sdc, and dev/sdd disks.
3. Upon the next reboot, Ansible fails to detect the removed /dev/sdc disk and expects that only
two disks will be used for host2, /dev/sdb and /dev/sdc (formerly /dev/sdd).
4. Ansible stops the installation process and returns the above error message.
[osds]
host1
host2 devices="[ '/dev/sdb', '/dev/sdc' ]"
host3
50
APPENDIX B. MANUALLY INSTALLING RED HAT CEPH STORAGE
IMPORTANT
Red Hat does not support or test upgrading manually deployed clusters. Therefore, Red
Hat recommends to use Ansible to deploy a new cluster with Red Hat Ceph Storage 3.
See Chapter 3, Deploying Red Hat Ceph Storage for details.
You can use command-line utilities, such as Yum, to install manually deployed clusters.
All Ceph clusters require at least one monitor, and at least as many OSDs as copies of an object stored
on the cluster. Red Hat recommends using three monitors for production environments and a minimum
of three Object Storage Devices (OSD).
Installing a Ceph storage cluster by using the command line interface involves these steps:
B.1. PREREQUISITES
Configuring the Network Time Protocol for Red Hat Ceph Storage
All Ceph Monitor and OSD nodes requires configuring the Network Time Protocol (NTP). Ensure that
Ceph nodes are NTP peers. NTP helps preempt issues that arise from clock drift.
NOTE
When using Ansible to deploy a Red Hat Ceph Storage cluster, Ansible automatically
installs, configures, and enables NTP.
Prerequisites
51
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
$ ntpq -p
Additional Resources
The Configuring NTP Using ntpd chapter in the System Administrator’s Guide for Red Hat
Enterprise Linux 7.
Monitor Bootstrapping
Bootstrapping a Monitor and by extension a Ceph storage cluster, requires the following data:
Unique Identifier
The File System Identifier (fsid) is a unique identifier for the cluster. The fsid was originally used
when the Ceph storage cluster was principally used for the Ceph file system. Ceph now supports
native interfaces, block devices, and object storage gateway interfaces too, so fsid is a bit of a
misnomer.
Cluster Name
Ceph clusters have a cluster name, which is a simple string without spaces. The default cluster name
is ceph, but you can specify a different cluster name. Overriding the default cluster name is
especially useful when you work with multiple clusters.
When you run multiple clusters in a multi-site architecture, the cluster name for example, us-west,
us-east identifies the cluster for the current command-line session.
NOTE
To identify the cluster name on the command-line interface, specify the Ceph
configuration file with the cluster name, for example, ceph.conf, us-west.conf, us-
east.conf, and so on.
Example:
Monitor Name
Each Monitor instance within a cluster has a unique name. In common practice, the Ceph Monitor
name is the node name. Red Hat recommend one Ceph Monitor per node, and no co-locating the
Ceph OSD daemons with the Ceph Monitor daemon. To retrieve the short node name, use the
hostname -s command.
Monitor Map
Bootstrapping the initial Monitor requires you to generate a Monitor map. The Monitor map requires:
Monitor Keyring
Monitors communicate with each other by using a secret key. You must generate a keyring with a
Monitor secret key and provide it when bootstrapping the initial Monitor.
Administrator Keyring
To use the ceph command-line interface utilities, create the client.admin user and generate its
52
APPENDIX B. MANUALLY INSTALLING RED HAT CEPH STORAGE
To use the ceph command-line interface utilities, create the client.admin user and generate its
keyring. Also, you must add the client.admin user to the Monitor keyring.
The foregoing requirements do not imply the creation of a Ceph configuration file. However, as a best
practice, Red Hat recommends creating a Ceph configuration file and populating it with the fsid, the
mon initial members and the mon host settings at a minimum.
You can get and set all of the Monitor settings at runtime as well. However, the Ceph configuration file
might contain only those settings which overrides the default values. When you add settings to a Ceph
configuration file, these settings override the default settings. Maintaining those settings in a Ceph
configuration file makes it easier to maintain the cluster.
3. As root, create a Ceph configuration file in the /etc/ceph/ directory. By default, Ceph uses
ceph.conf, where ceph reflects the cluster name:
Syntax
# touch /etc/ceph/<cluster_name>.conf
Example
# touch /etc/ceph/ceph.conf
4. As root, generate the unique identifier for your cluster and add the unique identifier to the
[global] section of the Ceph configuration file:
Syntax
Example
$ cat /etc/ceph/ceph.conf
[global]
fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993
53
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
Syntax
Example
7. As root, add the IP address of the initial Monitor to the Ceph configuration file:
Syntax
Example
NOTE
To use IPv6 addresses, you set the ms bind ipv6 option to true. For details, see
the Bind section in the Configuration Guide for Red Hat Ceph Storage 3.
8. As root, create the keyring for the cluster and generate the Monitor secret key:
Syntax
Example
Syntax
Example
54
APPENDIX B. MANUALLY INSTALLING RED HAT CEPH STORAGE
Syntax
Example
11. Generate the Monitor map. Specify using the node name, IP address and the fsid, of the initial
Monitor and save it as /tmp/monmap:
Syntax
Example
12. As root on the initial Monitor node, create a default data directory:
Syntax
# mkdir /var/lib/ceph/mon/<cluster_name>-<monitor_host_name>
Example
# mkdir /var/lib/ceph/mon/ceph-node1
13. As root, populate the initial Monitor daemon with the Monitor map and keyring:
Syntax
Example
55
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
# cat /etc/ceph/ceph.conf
[global]
fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993
mon_initial_members = node1
mon_host = 192.168.0.120
For more details on the various Ceph configuration settings, see the Configuration Guide for
Red Hat Ceph Storage 3. The following example of a Ceph configuration file lists some of the
most common configuration settings:
Example
[global]
fsid = <cluster-id>
mon initial members = <monitor_host_name>[, <monitor_host_name>]
mon host = <ip-address>[, <ip-address>]
public network = <network>[, <network>]
cluster network = <network>[, <network>]
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd journal size = <n>
osd pool default size = <n> # Write an object n times.
osd pool default min size = <n> # Allow writing n copy in a degraded state.
osd pool default pg num = <n>
osd pool default pgp num = <n>
osd crush chooseleaf type = <n>
Syntax
# touch /var/lib/ceph/mon/<cluster_name>-<monitor_host_name>/done
Example
# touch /var/lib/ceph/mon/ceph-node1/done
16. As root, update the owner and group permissions on the newly created directory and files:
Syntax
Example
56
APPENDIX B. MANUALLY INSTALLING RED HAT CEPH STORAGE
NOTE
If the Ceph Monitor node is co-located with an OpenStack Controller node, then
the Glance and Cinder keyring files must be owned by glance and cinder
respectively. For example:
# ls -l /etc/ceph/
...
-rw-------. 1 glance glance 64 <date> ceph.client.glance.keyring
-rw-------. 1 cinder cinder 64 <date> ceph.client.cinder.keyring
...
17. For storage clusters with custom names, as root, add the the following line:
Syntax
Example
18. As root, start and enable the ceph-mon process on the initial Monitor node:
Syntax
Example
Syntax
Example
57
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
To add more Red Hat Ceph Storage Monitors to the storage cluster, see the Adding a Monitor section in
the Administration Guide for Red Hat Ceph Storage 3.
Prerequisites
Procedure
Use the following commands on the node where ceph-mgr will be deployed and as the root user or with
the sudo utility.
mkdir /var/lib/ceph/mgr/ceph-hostname
Replace hostname with the host name of the node where the ceph-mgr daemon will be
deployed, for example:
3. In the newly created directory, create an authentication key for the ceph-mgr daemon:
[root@node1 ~]# ceph auth get-or-create mgr.`hostname -s` mon 'allow profile mgr' osd
'allow *' mds 'allow *' -o /var/lib/ceph/mgr/ceph-node1/keyring
58
APPENDIX B. MANUALLY INSTALLING RED HAT CEPH STORAGE
Replace hostname with the host name of the node where the ceph-mgr will be deployed, for
example:
ceph -s
The output will include a line similar to the following one under the services: section:
mgr: node1(active)
8. Install more ceph-mgr daemons to serve as standby daemons that become active if the current
active daemon fails.
Additional resources
OSD Bootstrapping
Once you have your initial monitor running, you can start adding the Object Storage Devices (OSDs).
Your cluster cannot reach an active + clean state until you have enough OSDs to handle the number of
copies of an object.
The default number of copies for an object is three. You will need three OSD nodes at minimum.
However, if you only want two copies of an object, therefore only adding two OSD nodes, then update
the osd pool default size and osd pool default min size settings in the Ceph configuration file.
For more details, see the OSD Configuration Reference section in the Configuration Guide for Red Hat
Ceph Storage 3.
After bootstrapping the initial monitor, the cluster has a default CRUSH map. However, the CRUSH map
does not have any Ceph OSD daemons mapped to a Ceph node.
To add an OSD to the cluster and updating the default CRUSH map, execute the following on each OSD
node:
59
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
3. Copy the Ceph configuration file and administration keyring file from the initial Monitor node to
the OSD node:
Syntax
# scp <user_name>@<monitor_host_name>:<path_on_remote_system>
<path_to_local_file>
Example
$ uuidgen
b367c360-b364-4b1d-8fc6-09408a9cda7a
Syntax
Example
NOTE
This command outputs the OSD number identifier needed for subsequent steps.
Syntax
# mkdir /var/lib/ceph/osd/<cluster_name>-<osd_id>
Example
# mkdir /var/lib/ceph/osd/ceph-0
7. As root, prepare the drive for use as an OSD, and mount it to the directory you just created.
60
APPENDIX B. MANUALLY INSTALLING RED HAT CEPH STORAGE
7. As root, prepare the drive for use as an OSD, and mount it to the directory you just created.
Create a partition for the Ceph data and journal. The journal and the data partitions can be
located on the same disk. This example is using a 15 GB disk:
Syntax
Example
Syntax
Example
NOTE
The directory must be empty before you run ceph-osd with the --mkkey option.
If you have a custom cluster name, the ceph-osd utility requires the --cluster
option.
9. As root, register the OSD authentication key. If your cluster name differs from ceph, insert your
cluster name instead:
Syntax
# ceph auth add osd.<osd_id> osd 'allow *' mon 'allow profile osd' -i
/var/lib/ceph/osd/<cluster_name>-<osd_id>/keyring
Example
61
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
# ceph auth add osd.0 osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-0/keyring
added key for osd.0
Syntax
Example
11. As root, place the OSD node under the default CRUSH tree:
Syntax
Example
Syntax
Example
NOTE
You can also decompile the CRUSH map, and add the OSD to the device list. Add
the OSD node as a bucket, then add the device as an item in the OSD node,
assign the OSD a weight, recompile the CRUSH map and set the CRUSH map.
For more details, see the Editing a CRUSH map section in the Storage Strategies
Guide for Red Hat Ceph Storage 3. for more details.
13. As root, update the owner and group permissions on the newly created directory and files:
Syntax
Example
62
APPENDIX B. MANUALLY INSTALLING RED HAT CEPH STORAGE
14. For storage clusters with custom names, as root, add the following line to the
/etc/sysconfig/ceph file:
Syntax
Example
15. The OSD node is in your Ceph storage cluster configuration. However, the OSD daemon is
down and in. The new OSD must be up before it can begin receiving data. As root, enable and
start the OSD process:
Syntax
Example
Now you have the monitors and some OSDs up and running. You can watch the placement groups peer
by executing the following command:
$ ceph -w
Example
63
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
To expand the storage capacity by adding new OSDs to the storage cluster, see the Adding an OSD
section in the Administration Guide for Red Hat Ceph Storage 3.
64
APPENDIX C. INSTALLING THE CEPH COMMAND LINE INTERFACE
ceph
ceph-authtool
ceph-dencoder
rados
Prerequisites
A running Ceph storage cluster, preferably in the active + clean state.
Procedure
1. On the client node, enable the Red Hat Ceph Storage 3 Tools repository:
3. From the initial monitor node, copy the Ceph configuration file, in this case ceph.conf, and the
administration keyring to the client node:
Syntax
Example
65
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
IMPORTANT
Ceph Block Devices must be deployed on separate nodes from the Ceph Monitor and
OSD nodes. Running kernel clients and kernel server daemons on the same node can
lead to kernel deadlocks.
Prerequisites
Ensure to perform the tasks listed in the Appendix C, Installing the Ceph Command Line
Interface section.
If you use Ceph Block Devices as a back end for virtual machines (VMs) that use QEMU,
increase the default file descriptor. See the Ceph - VM hangs when transferring large amounts
of data to RBD disk Knowledgebase article for details.
Procedure
1. Create a Ceph Block Device user named client.rbd with full permissions to files on OSD nodes
(osd 'allow rwx') and output the result to a keyring file:
ceph auth get-or-create client.rbd mon 'profile rbd' osd 'profile rbd pool=<pool_name>' \
-o /etc/ceph/rbd.keyring
Replace <pool_name> with the name of the pool that you want to allow client.rbd to have
access to, for example rbd:
See the User Management section in the Red Hat Ceph Storage 3 Administration Guide for
more information about creating users.
66
APPENDIX D. MANUALLY INSTALLING CEPH BLOCK DEVICE
WARNING
The default Ceph configuration includes the following Ceph Block Device
features:
layering
exclusive-lock
object-map
deep-flatten
fast-diff
If you use the kernel RBD (krbd) client, you will not be able to map the
block device image because the current kernel version included in Red Hat
Enterprise Linux 7.3 does not support object-map, deep-flatten, and fast-
diff.
To work around this problem, disable the unsupported features. Use one of
the following options to do so:
For example:
Use the --image-feature layering option with the rbd create command
to enable only layering on newly created block device images.
rbd_default_features = 1
This is a known issue, for details see the Known Issues chapter in the
Release Notes for Red Hat Ceph Storage 3.
All these features work for users that use the user-space RBD client to
access the block device images.
For example:
67
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
IMPORTANT
Kernel block devices currently only support the legacy straw bucket algorithm in
the CRUSH map. If you have set the CRUSH tunables to optimal, you must set
them to legacy or an earlier major release, otherwise, you will not be able to map
the image.
Alternatively, replace straw2 with straw in the CRUSH map. For details, see the
Editing a CRUSH Map chapter in the Storage Strategies guide for Red Hat
Ceph Storage 3.
Specify the pool name and the image name, for example:
mkdir <mount_directory>
mount /dev/rbd/<pool_name>/<image_name> <mount_directory>
For example:
# mkdir /mnt/ceph-block-device
# mount /dev/rbd/rbd/image1 /mnt/ceph-block-device
For additional details, see the Block Device Guide for Red Hat Ceph Storage 3.
68
APPENDIX E. MANUALLY INSTALLING CEPH OBJECT GATEWAY
Prerequisites
A running Ceph storage cluster, preferably in the active + clean state.
Perform the tasks listed in Chapter 2, Requirements for Installing Red Hat Ceph Storage .
Procedure
1. Enable the Red Hat Ceph Storage 3 Tools repository:
[client.rgw.<obj_gw_hostname>]
host = <obj_gw_hostname>
rgw frontends = "civetweb port=80"
rgw dns name = <obj_gw_hostname>.example.com
Where <obj_gw_hostname> is a short host name of the gateway node. To view the short
host name, use the hostname -s command.
b. Copy the updated configuration file to the new Object Gateway node and all other nodes in
the Ceph storage cluster:
Syntax
Example
Syntax
# scp /etc/ceph/<cluster_name>.client.admin.keyring
<user_name>@<target_host_name>:/etc/ceph/
69
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
Example
Syntax
Example
5. On the Object Gateway node, add a user and keyring to bootstrap the object gateway:
Syntax
# ceph auth get-or-create client.rgw.`hostname -s` osd 'allow rwx' mon 'allow rw' -o
/var/lib/ceph/radosgw/<cluster_name>-rgw.`hostname -s`/keyring
Example
# ceph auth get-or-create client.rgw.`hostname -s` osd 'allow rwx' mon 'allow rw' -o
/var/lib/ceph/radosgw/ceph-rgw.`hostname -s`/keyring
IMPORTANT
When you provide capabilities to the gateway key you must provide the read
capability. However, providing the Monitor write capability is optional; if you
provide it, the Ceph Object Gateway will be able to create pools automatically.
Syntax
Example
7. On the Object Gateway node, change the owner and group permissions:
70
APPENDIX E. MANUALLY INSTALLING CEPH OBJECT GATEWAY
8. For storage clusters with custom names, as root, add the following line:
Syntax
Example
10. On the Object Gateway node, start and enable the ceph-radosgw process:
Syntax
Example
Once installed, the Ceph Object Gateway automatically creates pools if the write capability is set on the
Monitor. See the Pools chapter in the Storage Strategies Guide for information on creating pools
manually.
Additional Details
The Red Hat Ceph Storage 3 the Object Gateway Guide for Red Hat Enterprise Linux
71
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
Because Ansible manages the Ceph configuration file, edit the /usr/share/ceph-
ansible/group_vars/all.yml file to change the Ceph configuration. Use the ceph_conf_overrides
setting to override the default Ceph configuration.
Ansible supports the same sections as the Ceph configuration file; [global], [mon], [osd], [mds], [rgw],
and so on. You can also override particular instances, such as a particular Ceph Object Gateway instance.
For example:
###################
# CONFIG OVERRIDE #
###################
ceph_conf_overrides:
client.rgw.rgw1:
log_file: /var/log/ceph/ceph-rgw-rgw1.log
NOTE
Ansible does not include braces when referring to a particular section of the Ceph
configuration file. Sections and settings names are terminated with a colon.
IMPORTANT
Do not set the cluster network with the cluster_network parameter in the CONFIG
OVERRIDE section because this can cause two conflicting cluster networks being set in
the Ceph configuration file.
To set the cluster network, use the cluster_network parameter in the CEPH
CONFIGURATION section. For details, see Section 3.2, “Installing a Red Hat Ceph
Storage Cluster”.
72
APPENDIX G. MANUALLY UPGRADING FROM RED HAT CEPH STORAGE 2 TO 3
Red Hat recommends upgrading the Ceph components in the following order:
Monitor nodes
OSD nodes
Red Hat Ceph Storage 3 introduces a new daemon Ceph Manager (ceph-mgr). Install ceph-mgr after
upgrading the Monitor nodes.
After upgrading the storage cluster you can have a health warning regarding the CRUSH map using
legacy tunables. For details, see the CRUSH Tunables section in the Storage Strategies guide for
Red Hat Ceph Storage 3.
Example
$ ceph -s
cluster 848135d7-cdb9-4084-8df2-fb5e41ae60bd
health HEALTH_WARN
crush map has legacy tunables (require bobtail, min is firefly)
monmap e1: 1 mons at {ceph1=192.168.0.121:6789/0}
election epoch 2, quorum 0 ceph1
osdmap e83: 2 osds: 2 up, 2 in
pgmap v1864: 64 pgs, 1 pools, 38192 kB data, 17 objects
10376 MB used, 10083 MB / 20460 MB avail
64 active+clean
IMPORTANT
Red Hat recommends all Ceph clients to be running the same version as the Ceph
storage cluster.
Prerequisites
If the cluster you want to upgrade contains Ceph Block Device images that use the exclusive-
lock feature, ensure that all Ceph Block Device users have permissions to blacklist clients:
ceph auth caps client.<ID> mon 'allow r, allow command "osd blacklist"' osd '<existing-OSD-
user-capabilities>'
73
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
Procedure
Do the following steps on each Monitor node in the storage cluster. Upgrade only one Monitor node at a
time.
1. If you installed Red Hat Ceph Storage 2 by using software repositories, disable the repositories:
Syntax
Example
Syntax
Example
NOTE
74
APPENDIX G. MANUALLY UPGRADING FROM RED HAT CEPH STORAGE 2 TO 3
NOTE
If the Ceph Monitor node is colocated with an OpenStack Controller node, then
the Glance and Cinder keyring files must be owned by glance and cinder
respectively. For example:
# ls -l /etc/ceph/
...
-rw-------. 1 glance glance 64 <date> ceph.client.glance.keyring
-rw-------. 1 cinder cinder 64 <date> ceph.client.cinder.keyring
...
6. If SELinux is in enforcing or permissive mode, relabel the SELinux context on the next reboot.
# touch /.autorelabel
WARNING
# shutdown -r now
9. Once the Monitor node is up, check the health of the Ceph storage cluster before moving to the
next Monitor node:
# ceph -s
Prerequisites
75
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
Procedure
Use the following commands on the node where ceph-mgr will be deployed and as the root user or with
the sudo utility.
mkdir /var/lib/ceph/mgr/ceph-hostname
Replace hostname with the host name of the node where the ceph-mgr daemon will be
deployed, for example:
3. In the newly created directory, create an authentication key for the ceph-mgr daemon:
[root@node1 ~]# ceph auth get-or-create mgr.`hostname -s` mon 'allow profile mgr' osd
'allow *' mds 'allow *' -o /var/lib/ceph/mgr/ceph-node1/keyring
Replace hostname with the host name of the node where the ceph-mgr will be deployed, for
example:
ceph -s
The output will include a line similar to the following one under the services: section:
76
APPENDIX G. MANUALLY UPGRADING FROM RED HAT CEPH STORAGE 2 TO 3
mgr: node1(active)
8. Install more ceph-mgr daemons to serve as standby daemons that become active if the current
active daemon fails.
Additional resources
Prerequisites
When upgrading an OSD node, some placement groups will become degraded because the OSD might
be down or restarting. To prevent Ceph from starting the recovery process, on a Monitor node, set the
noout and norebalance OSD flags:
Procedure
Do the following steps on each OSD node in the storage cluster. Upgrade only one OSD node at a time.
If an ISO-based installation was performed for Red Hat Ceph Storage 2.3, then skip this first step.
Syntax
Example
5. As root, update the owner and group permissions on the newly created directory and files:
Syntax
77
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
Example
NOTE
Using the following find command might quicken the process of changing
ownership by using the chown command in parallel on a Ceph storage cluster
with a large number of disks:
6. If SELinux is set to enforcing or permissive mode, then set a relabelling of the SELinux context
on files for the next reboot:
# touch /.autorelabel
WARNING
NOTE
In environments with large number of objects per placement group (PG), the
directory enumeration speed will decrease, causing a negative impact to
performance. This is caused by the addition of xattr queries which verifies the
SELinux context. Setting the context at mount time removes the xattr queries
for context and helps overall disk performance, especially on slower disks.
Add the following line to the [osd] section in the /etc/ceph/ceph.conf file:
osd_mount_options_xfs=rw,noatime,inode64,context="system_u:object_r:ceph_
var_lib_t:s0"
# udevadm trigger
78
APPENDIX G. MANUALLY UPGRADING FROM RED HAT CEPH STORAGE 2 TO 3
# shutdown -r now
NOTE
If the noout and norebalance flags are set, the storage cluster is in
HEALTH_WARN state
$ ceph health
HEALTH_WARN noout,norebalance flag(s) set
Once you are done upgrading the Ceph Storage Cluster, unset the previously set OSD flags and verify
the storage cluster status.
On a Monitor node, and after all OSD nodes have been upgraded, unset the noout and norebalance
flags:
In addition, execute the ceph osd require-osd-release <release> command. This command ensures
that no more OSDs with Red Hat Ceph Storage 2.3 can be added to the storage cluster. If you do not run
this command, the storage status will be HEALTH_WARN.
Additional Resources
To expand the storage capacity by adding new OSDs to the storage cluster, see the Add an
OSD section in the Administration Guide for Red Hat Ceph Storage 3
Prerequisites
Red Hat recommends putting a Ceph Object Gateway behind a load balancer, such as HAProxy.
If you use a load balancer, remove the Ceph Object Gateway from the load balancer once no
requests are being served.
If you use a custom name for the region pool, specified in the rgw_region_root_pool
parameter, add the rgw_zonegroup_root_pool parameter to the [global] section of the Ceph
configuration file. Set the value of rgw_zonegroup_root_pool to be the same as
rgw_region_root_pool, for example:
79
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
[global]
rgw_zonegroup_root_pool = .us.rgw.root
Procedure
Do the following steps on each Ceph Object Gateway node in the storage cluster. Upgrade only one
node at a time.
1. If you used online repositories to install Red Hat Ceph Storage, disable the 2 repositories.
5. Change the owner and group permissions on the newly created /var/lib/ceph/radosgw/ and
/var/log/ceph/ directories and their content to ceph.
6. If SELinux is set to run in enforcing or permissive mode, instruct it to relabel SELinux context on
the next boot.
# touch /.autorelabel
IMPORTANT
Relabeling takes a long time to complete, because SELinux must traverse every
file system and fix any mislabeled files. To exclude directories from being
relabeled, add them to the /etc/selinux/fixfiles_exclude_dirs file before
rebooting.
Replace <hostname> with the name of the Ceph Object Gateway host, for example gateway-
node.
80
APPENDIX G. MANUALLY UPGRADING FROM RED HAT CEPH STORAGE 2 TO 3
# shutdown -r now
9. If you use a load balancer, add the Ceph Object Gateway node back to the load balancer.
See Also
The Ceph Object Gateway Guide for Red Hat Enterprise Linux
QEMU/KVM hypervisors
Red Hat recommends all Ceph clients to be running the same version as the Ceph storage cluster.
Prerequisites
Stop all I/O requests against a Ceph client node while upgrading the packages to prevent
unexpected errors to occur
Procedure
1. If you installed Red Hat Ceph Storage 2 clients by using software repositories, disable the
repositories:
NOTE
If an ISO-based installation was performed for Red Hat Ceph Storage 2 clients,
skip this first step.
2. On the client node, enable the Red Hat Ceph Storage Tools 3 repository:
Restart any application that depends on the Ceph client-side libraries after upgrading the ceph-
81
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
Restart any application that depends on the Ceph client-side libraries after upgrading the ceph-
common package.
NOTE
If you are upgrading OpenStack Nova compute nodes that have running QEMU/KVM
instances or use a dedicated QEMU/KVM client, stop and start the QEMU/KVM instance
because restarting the instance does not work in this case.
82
APPENDIX H. CHANGES IN ANSIBLE VARIABLES BETWEEN VERSION 2 AND 3
83
Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux
1. After manually upgrading from version 1.3 to version 2, install and configure Ansible on the
administration node.
2. Ensure that the Ansible administration node has passwordless ssh access to all Ceph nodes in
the cluster. See Section 2.11, “Enabling Password-less SSH for Ansible” for more details.
3. As root, create a symbolic link to the Ansible group_vars directory in the /etc/ansible/
directory:
# ln -s /usr/share/ceph-ansible/group_vars /etc/ansible/group_vars
4. As root, create an all.yml file from the all.yml.sample file and open it for editing:
# cd /etc/ansible/group_vars
# cp all.yml.sample all.yml
# vim all.yml
8. Modify the Ansible inventory in /etc/ansible/hosts to include Ceph hosts. Add monitors under a
[mons] section, OSDs under an [osds] section and gateways under an [rgws] section to
identify their roles to Ansible.
9. Make sure ceph_conf_overrides is updated with the original ceph.conf options used for
[global], [osd], [mon], and [client] sections in the all.yml file.
Options like osd journal, public_network and cluster_network should not be added in
ceph_conf_overrides because they are already part of all.yml. Only the options that are not
part of all.yml and are in the original ceph.conf should be added to ceph_conf_overrides.
# cd /usr/share/ceph-ansible/
# cp infrastructure-playbooks/take-over-existing-cluster.yml .
$ ansible-playbook take-over-existing-cluster.yml -u <username>
84
APPENDIX J. PURGING A CEPH CLUSTER BY USING ANSIBLE
IMPORTANT
Purging a Ceph cluster will lose data stored on the cluster’s OSDs.
1. Declare the OSD devices in the osds.yml file. See Section 3.2, “Installing a Red Hat Ceph
Storage Cluster” for more details.
# cd /usr/share/ceph-ansible
# cp infrastructure-playbooks/purge-cluster.yml .
$ ansible-playbook purge-cluster.yml
85