Ceph

**Installation on Ubuntu 22.04 LTS (3-Node Cluster):**

**Prerequisites (All Nodes):**
```bash
# Update system
sudo apt update && sudo apt upgrade -y

# Set hostnames (on each node)
sudo hostnamectl set-hostname ceph-node1 # ceph-node2, ceph-node3
```

**1. Install Cephadm (Bootstrap Node):**
```bash
# Install dependencies
sudo apt install -y python3 python3-pip curl

# Download cephadm
curl --silent --remote-name --location https://github.com/ceph/ceph/raw/quincy/src/cephadm/cephadm
chmod +x cephadm

# Install cephadm
sudo ./cephadm add-repo --release quincy
sudo ./cephadm install
```

**2. Bootstrap Cluster (Node 1):**
```bash
# Bootstrap first node
sudo cephadm bootstrap --mon-ip 10.0.0.10 --cluster-network 10.0.1.0/24

# Save output - contains dashboard URL and admin password
# Dashboard: https://ceph-node1:8443/
# Username: admin
# Password: <generated-password>
```

**3. Add Additional Nodes:**
```bash
# Copy SSH key to other nodes
ssh-copy-id root@ceph-node2
ssh-copy-id root@ceph-node3

# Add hosts to cluster
sudo ceph orch host add ceph-node2 10.0.0.11
sudo ceph orch host add ceph-node3 10.0.0.12

# Label nodes for specific roles
sudo ceph orch host label add ceph-node1 mon
sudo ceph orch host label add ceph-node2 mon
sudo ceph orch host label add ceph-node3 mon
```

**4. Deploy MON and MGR Daemons:**
```bash
# Deploy monitors (3 for quorum)
sudo ceph orch apply mon --placement="3 ceph-node1 ceph-node2 ceph-node3"

# Deploy managers (2 for HA)
sudo ceph orch apply mgr --placement="2 ceph-node1 ceph-node2"
```

**5. Add OSD Storage:**
```bash
# List available devices
sudo ceph orch device ls

# Add all available devices as OSDs
sudo ceph orch apply osd --all-available-devices

# Or add specific devices
sudo ceph orch daemon add osd ceph-node1:/dev/sdb
sudo ceph orch daemon add osd ceph-node2:/dev/sdb
sudo ceph orch daemon add osd ceph-node3:/dev/sdb
```

**6. Deploy MDS for CephFS (Optional):**
```bash
# Create CephFS
sudo ceph fs volume create cephfs --placement="2 ceph-node1 ceph-node2"
```

**7. Enable Dashboard:**
```bash
# Enable dashboard module
sudo ceph mgr module enable dashboard

# Create dashboard admin user
sudo ceph dashboard ac-user-create admin <password> administrator

# Check dashboard status
sudo ceph mgr services
```

**8. Create RBD Pool for Block Storage:**
```bash
# Create replicated pool (3 replicas)
sudo ceph osd pool create rbd 64 64

# Enable RBD application
sudo ceph osd pool application enable rbd rbd

# Initialize pool
sudo rbd pool init rbd
```

**9. Verify Cluster Health:**
```bash
# Check cluster status
sudo ceph -s

# Check OSD status
sudo ceph osd status

# Check MON status
sudo ceph mon stat
```