Intern:Cluster/xora

Cluster/xora war ein erster Test zu einem Verbund (Cluster) von Servern mit Proxmox VE.

Später wurde der Test erneut bei der Server/Aktualisierung/2019 als Intern:ora vorgenommen.

Test

Grundsätzlich soll ein möglichst ausfallsicherer Verbund an Servern mit Proxmox VE getestet werden.

Getestet werden soll:

• Datensicherung
  • Replikation
  • …
• Ausfallüberbrückung
• …

Bezeichnung

Es ist die künstlich geschaffene Bezeichnung für den Verbund von mehreren Servern.

Verbund

Für einen Verbund an Servern mit Proxmox VE bedarf es wohl mindestens 3 Servern.

Zu dem Verbund gehören (theoretisch)

mindestens

• cora (zuvor srs27090)

  und

• dora (zuvor srs27091)

zur Vervollständigung auch

• nora (zuvor srs3008)

  oder/und

• lora (zuvor (angedacht) srs85290)

zur Ergänzung eventuell auch

• zora (zuvor srs5100)

.

Geräte

einzelne Geräte

Gerät	Name	IPv4 OS	DNS (A) OS	WUI OS	Mail OS	Geburtshilfe	Start (Dauer)	IPv4 IPMI	DNS (A) IPMI	WUI IPMI	Mail IPMI	Netzwerkschnittstellen	Massenspeicher	Eigentum
27090	cora	141.56.51.123	cora.stura-dresden.de	https://cora.stura-dresden.de:8006/	cora@stura.htw-dresden.de	James	3 min	141.56.51.113	irmc.cora.stura-dresden.de	https://irmc.cora.stura-dresden.de/	irmc.cora@stura.htw-dresden.de	M 2 1 X	3.5 ″ 3.5 ″ 2 TB 2 TB	bsd.services:user:vater:hw#rx300_s6_2709_0
27091	dora	141.56.51.124	dora.stura-dresden.de	https://dora.stura-dresden.de:8006/	dora@stura.htw-dresden.de	Fullforce	3 min	141.56.51.114	irmc.dora.stura-dresden.de	https://irmc.dora.stura-dresden.de/	irmc.dora@stura.htw-dresden.de	M 2 1 X	3.5 ″ 3.5 ″ 2 TB 2 TB	bsd.services:user:vater:hw#rx300_s6_2709_1
8529	lora	141.56.51.127	lora.stura-dresden.de	https://lora.stura-dresden.de:8006/	lora@stura.htw-dresden.de		min	141.56.51.117	irmc.lora.stura-dresden.de	https://irmc.lora.stura-dresden.de/	irmc.lora@stura.htw-dresden.de	M 2 1	3.5 ″ 3.5 ″ 2 TB 2 TB	bsd.services:user:vater:hw#rx300_s6_8529
8	nora	141.56.51.128	nora.stura-dresden.de	https://nora.stura-dresden.de:8006/	nora@stura.htw-dresden.de		2 min	141.56.51.118	irmc.nora.stura-dresden.de	https://irmc.nora.stura-dresden.de/	irmc.nora@stura.htw-dresden.de	M 2 1 X	3.5 ″ 3.5 ″ 2 TB 2 TB	StuRa (srs3008)
5100	zora	141.56.51.129	zora.stura-dresden.de	https://zora.stura-dresden.de:8006/	zora@stura.htw-dresden.de		min	141.56.51.119	drac.cora.stura-dresden.de	https://drac.cora.stura-dresden.de/	drac.zora@stura.htw-dresden.de	M 1 2	3.5 ″ 3.5 ″ 3.5 ″ 3.5 ″ 2 TB 2 TB 2 TB 2 TB	bsd.services:user:vater:hw#dell_poweredge_r510

BIOS

F2

Boot

alles andere

USB KEY: …

PCI SCSI: #0100 ID000 LN0 HGST H

PCI SCSI: #0100 ID004 LN0 HGST H

Proxmox VE

Installation Betriebssystem

Durchführung Installation Betreibssystem

Install Proxmox VE

Loading Proxmox Installer ...
Loading initial ramdisk ...

Proxmox startup

End User License Agreement (EULA)

I agree

Proxmox Virtualiszation Environment (PVE)

Options

Filesystem

ext4zfs (RAID1)

Disk Setup

Harddisk 0

/dev/sda (1863GB, HUS726020ALS214)

Harddisk 1

/dev/sdb (1863GB, HUS726020ALS214)

Harddisk 2

-- do not use --

Advanced Options

ashift

12

compress

on

checksum

on

copies

1

OK

Target

zfs (RAID1)

Next

Location and Time Zone selection

Country

Germany

Time zone

Europe/Berlin

Keyboard Layout

German

Next

Administration Password and E-Mail Address

Password

8

Confirm

8

E-Mail

siehe #einzelne Geräte

Next

Management Network Configuration

Management Interface

enp8s0f0 - … (igb)

Hostname (FQDN)

siehe #einzelne Geräte

IP Address

siehe #einzelne Geräte

Netmask

255.255.255.0

Gateway

141.56.51.254

DNS Server

141.56.1.1

Next

…

Installation successful!

Reboot

Nachbereitung Installation Betreibssystem

erste Aktualisierung

• Update (WUI)
  • Refresh (WUI)

    Neustart (WUI)

  • Upgrade (WUI)

(optionales) ZFS anschauen

zpool status

  pool: rpool
 state: ONLINE
  scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	rpool       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    sda2    ONLINE       0     0     0
	    sdb2    ONLINE       0     0     0

errors: No known data errors

zfs list

NAME               USED  AVAIL  REFER  MOUNTPOINT
rpool             9.40G  1.75T   104K  /rpool
rpool/ROOT         919M  1.75T    96K  /rpool/ROOT
rpool/ROOT/pve-1   919M  1.75T   919M  /
rpool/data          96K  1.75T    96K  /rpool/data
rpool/swap        8.50G  1.75T    56K  -

(optionales) Anschauen der Partitionierung

fdisk -l /dev/sd{a,b}

Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 0A3CA01C-D0CE-4750-A26A-C07C1541EF1D

Device          Start        End    Sectors  Size Type
/dev/sda1          34       2047       2014 1007K BIOS boot
/dev/sda2        2048 3907012749 3907010702  1.8T Solaris /usr & Apple ZFS
/dev/sda9  3907012750 3907029134      16385    8M Solaris reserved 1


Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C0D3B0CA-C966-4B00-B367-EEDBD04872F7

Device          Start        End    Sectors  Size Type
/dev/sdb1          34       2047       2014 1007K BIOS boot
/dev/sdb2        2048 3907012749 3907010702  1.8T Solaris /usr & Apple ZFS
/dev/sdb9  3907012750 3907029134      16385    8M Solaris reserved 1

Sicherung des initialen Zustandes von PVE (mit der erfolgten Aktualisierung)

Erstellen eines Schnappschusses vom gesamten Pool

zfs snapshot -r rpool@fresh-installed-pve-and-updated

Im Übrigen kann überlegt werden, dass darüber hinaus auch eine Sicherung von /dev/sd{a,b}1 vorgenommen wird.

Anpassung der Quellen

Siehe auch

Server/Proxmox#sources.list

Ergänzung der Quelle pve-no-subscription

---

auf schnell!

echo 'deb http://download.proxmox.com/debian/pve stretch pve-no-subscription' > /etc/apt/sources.list.d/pve-download.list && apt update

---

cat /etc/apt/sources.list.d/pve-enterprise.list

cat: /etc/apt/sources.list.d/pve-enterprise.list: No such file or directory

$EDITOR /etc/apt/sources.list.d/pve-enterprise.list

cat /etc/apt/sources.list.d/pve-enterprise.list

deb http://download.proxmox.com/debian/pve stretch pve-no-subscription

Entfernung der Quelle pve-enterprise

cat /etc/apt/sources.list.d/pve-enterprise.list

deb https://enterprise.proxmox.com/debian/pve stretch pve-enterprise

$EDITOR /etc/apt/sources.list.d/pve-enterprise.list

cat /etc/apt/sources.list.d/pve-enterprise.list

####deb https://enterprise.proxmox.com/debian/pve stretch pve-enterprise

Erstellung vom Cluster

optionales Begutachten für die Erstellung von einem Cluster

	cora	dora	nora
less /etc/network/interfaces
	auto lo iface lo inet loopback iface enp8s0f0 inet manual auto vmbr0 iface vmbr0 inet static address 141.56.51.123 netmask 255.255.255.0 gateway 141.56.51.254 bridge_ports enp8s0f0 bridge_stp off bridge_fd 0 iface ens4f0 inet manual iface ens4f1 inet manual iface ens4f2 inet manual iface ens4f3 inet manual iface enp8s0f1 inet manual	auto lo iface lo inet loopback iface enp8s0f0 inet manual auto vmbr0 iface vmbr0 inet static address 141.56.51.124 netmask 255.255.255.0 gateway 141.56.51.254 bridge_ports enp8s0f0 bridge_stp off bridge_fd 0 iface ens4f0 inet manual iface ens4f1 inet manual iface ens4f2 inet manual iface ens4f3 inet manual iface enp8s0f1 inet manual	auto lo iface lo inet loopback iface eno1 inet manual auto vmbr0 iface vmbr0 inet static address 141.56.51.128 netmask 255.255.255.0 gateway 141.56.51.254 bridge_ports eno1 bridge_stp off bridge_fd 0 iface enp2s0f0 inet manual iface enp2s0f1 inet manual iface enp2s0f2 inet manual iface enp2s0f3 inet manual iface eno2 inet manual
less /etc/hosts
	127.0.0.1 localhost.localdomain localhost 141.56.51.123 cora.stura-dresden.de cora pvelocalhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts	127.0.0.1 localhost.localdomain localhost 141.56.51.124 dora.stura-dresden.de dora pvelocalhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts	127.0.0.1 localhost.localdomain localhost 141.56.51.128 nora.stura-dresden.de nora pvelocalhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts

Erzeugung vom Cluster xora

auf einer der Server, die zum Cluster gehören sollen

vorgenommen auf #8 (nora)

(alternativ) grafische Oberfläche	Kommandozeile
Datacenter -> Cluster -> Create Cluster Create Cluster Cluster Name xora Ring 0 Address Corosync Cluster Engine Authentication key generator. Gathering 1024 bits for key from /dev/urandom. Writing corosync key to /etc/corosync/authkey. Writing corosync config to /etc/pve/corosync.conf Restart corosync and cluster filesystem TASK OK	pvecm create xora Corosync Cluster Engine Authentication key generator. Gathering 1024 bits for key from /dev/urandom. Writing corosync key to /etc/corosync/authkey. Writing corosync config to /etc/pve/corosync.conf Restart corosync and cluster filesystem

less /etc/pve/corosync.conf

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: nora
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 141.56.51.128
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: xora
  config_version: 1
  interface {
    bindnetaddr: 141.56.51.128
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2

pvecm status

Quorum information
------------------
Date:             Fri Mmm dd HH:MM:SS yyyy
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000001
Ring ID:          1/12
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   1
Highest expected: 1
Total votes:      1
Quorum:           1  
Flags:            Quorate 

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 141.56.51.128 (local)

Erstellung von einem internen Netz für das Cluster von Proxmox

frei gewählt verwenden wir 10.10.10.0/24

	cora	dora	nora
grafische Oberfläche (mit notwendigen Neustart)
less /etc/network/interfaces
	# network interface settings; autogenerated # Please do NOT modify this file directly, unless you know what # you're doing. # # If you want to manage part of the network configuration manually, # please utilize the 'source' or 'source-directory' directives to do # so. # PVE will preserve these directives, but will NOT its network # configuration from sourced files, so do not attempt to move any of # the PVE managed interfaces into external files! auto lo iface lo inet loopback iface enp8s0f0 inet manual iface ens4f0 inet manual iface ens4f1 inet manual iface ens4f2 inet manual iface ens4f3 inet manual auto enp8s0f1 iface enp8s0f1 inet static address 10.10.10.123 netmask 255.255.255.0 auto vmbr0 iface vmbr0 inet static address 141.56.51.123 netmask 255.255.255.0 gateway 141.56.51.254 bridge_ports enp8s0f0 bridge_stp off bridge_fd 0	# network interface settings; autogenerated # Please do NOT modify this file directly, unless you know what # you're doing. # # If you want to manage part of the network configuration manually, # please utilize the 'source' or 'source-directory' directives to do # so. # PVE will preserve these directives, but will NOT its network # configuration from sourced files, so do not attempt to move any of # the PVE managed interfaces into external files! auto lo iface lo inet loopback iface enp8s0f0 inet manual iface ens4f0 inet manual iface ens4f1 inet manual iface ens4f2 inet manual iface ens4f3 inet manual auto enp8s0f1 iface enp8s0f1 inet static address 10.10.10.124 netmask 255.255.255.0 auto vmbr0 iface vmbr0 inet static address 141.56.51.124 netmask 255.255.255.0 gateway 141.56.51.254 bridge_ports enp8s0f0 bridge_stp off bridge_fd 0	# network interface settings; autogenerated # Please do NOT modify this file directly, unless you know what # you're doing. # # If you want to manage part of the network configuration manually, # please utilize the 'source' or 'source-directory' directives to do # so. # PVE will preserve these directives, but will NOT its network # configuration from sourced files, so do not attempt to move any of # the PVE managed interfaces into external files! auto lo iface lo inet loopback iface eno1 inet manual iface enp2s0f0 inet manual iface enp2s0f1 inet manual iface enp2s0f2 inet manual iface enp2s0f3 inet manual auto eno2 iface eno2 inet static address 10.10.10.128 netmask 255.255.255.0 auto vmbr0 iface vmbr0 inet static address 141.56.51.128 netmask 255.255.255.0 gateway 141.56.51.254 bridge_ports eno1 bridge_stp off bridge_fd 0

Eintragung der anderen Nodes unabhängig von DNS

	cora	dora	nora
less /etc/hosts
	#### members of the cluster xora 10.10.10.123 cora.xora.stura-dresden.de cora.xora 10.10.10.124 dora.xora.stura-dresden.de dora.xora 10.10.10.128 nora.xora.stura-dresden.de nora.xora # The following lines are desirable for IPv6 capable hosts

Hinzufügen von anderen Servern

	cora	dora	nora
	pvecm add nora.xora Please enter superuser (root) password for 'nora.xora': Password for root@nora.xora: ******** Etablishing API connection with host 'nora.xora' The authenticity of host 'nora.xora' can't be established. X509 SHA256 key fingerprint is F3:A9:2D:9E:D5:59:DA:AE:5E:76:71:1E:02:D9:49:B6:67:5C:40:B0:0C:C0:05:FF:C5:D7:62:37:00:D8:CA:DD. Are you sure you want to continue connecting (yes/no)? yes Login succeeded. Request addition of this node Join request OK, finishing setup locally stopping pve-cluster service backup old database to '/var/lib/pve-cluster/backup/config-1538462961.sql.gz' waiting for quorum...OK (re)generate node files generate new node certificate merge authorized SSH keys and known hosts generated new node certificate, restart pveproxy and pvedaemon services successfully added node 'cora' to cluster.	pvecm add nora.xora Please enter superuser (root) password for 'nora.xora': Password for root@nora.xora: ******** Etablishing API connection with host 'nora.xora' The authenticity of host 'nora.xora' can't be established. X509 SHA256 key fingerprint is F3:A9:2D:9E:D5:59:DA:AE:5E:76:71:1E:02:D9:49:B6:67:5C:40:B0:0C:C0:05:FF:C5:D7:62:37:00:D8:CA:DD. Are you sure you want to continue connecting (yes/no)? yes Login succeeded. Request addition of this node Join request OK, finishing setup locally stopping pve-cluster service backup old database to '/var/lib/pve-cluster/backup/config-1538462994.sql.gz' waiting for quorum...OK (re)generate node files generate new node certificate merge authorized SSH keys and known hosts generated new node certificate, restart pveproxy and pvedaemon services successfully added node 'dora' to cluster.
			/etc/pve/corosync.conf logging { debug: off to_syslog: yes } nodelist { node { name: cora nodeid: 2 quorum_votes: 1 ring0_addr: 141.56.51.123 } node { name: dora nodeid: 3 quorum_votes: 1 ring0_addr: 141.56.51.124 } node { name: nora nodeid: 1 quorum_votes: 1 ring0_addr: 141.56.51.128 } } quorum { provider: corosync_votequorum } totem { cluster_name: xora config_version: 3 interface { bindnetaddr: 141.56.51.128 ringnumber: 0 } ip_version: ipv4 secauth: on version: 2 } (auch in Anlehnung an https://pve.proxmox.com/wiki/Separate_Cluster_Network) /etc/pve/corosync.conf logging { debug: off to_syslog: yes } nodelist { node { name: cora nodeid: 3 quorum_votes: 1 ring0_addr: cora.xora } node { name: dora nodeid: 4 quorum_votes: 1 ring0_addr: dora.xora } node { name: nora nodeid: 8 quorum_votes: 1 ring0_addr: nora.xora } } quorum { provider: corosync_votequorum } totem { cluster_name: xora config_version: 4 interface { bindnetaddr: nora.xora ringnumber: 0 } ip_version: ipv4 secauth: on version: 2 }

Prüfen vom Verbund

	cora	dora	nora
pvecm status			Quorum information ------------------ Date: Fri Mmm dd HH:MM:SS yyyy Quorum provider: corosync_votequorum Nodes: 3 Node ID: 0x00000008 Ring ID: 3/52 Quorate: Yes Votequorum information ---------------------- Expected votes: 3 Highest expected: 3 Total votes: 3 Quorum: 2 Flags: Quorate Membership information ---------------------- Nodeid Votes Name 0x00000003 1 10.10.10.123 0x00000004 1 10.10.10.124 0x00000008 1 10.10.10.128 (local)

Erstellen vom Cluster für Ceph

Erstellung von einem internen Netz für das Cluster von Ceph

frei gewählt verwenden wir 10.10.11.0/24

• Netzwerkschnittstellen

  10.10.11.0/24

  10.10.11.123/24

  10.10.11.124/24

  10.10.11.128/24

Neustart (wegen Netzwerkschnittstellen)

Testweises Verbinden (zu den jeweils anderen beiden Servern im Cluster) per ssh

ssh root@10.10.11.123

ssh root@10.10.11.124

ssh root@10.10.11.128

Installation von Ceph

pveceph install

#Ergänzung der Quelle pve-no-subscription

pveceph install

Initialisieren von Ceph

gemäß doku 10.10.10.0/24, aber das wurde (versehentlich schon für die #Erstellung von einem internen Netz für das Cluster von Proxmox verwendet)

pveceph init --network 10.10.11.0/24

WUI

rados_connect failed - No such file or directory (500)

pveceph createmon

creating /etc/pve/priv/ceph.client.admin.keyring
monmaptool: monmap file /tmp/monmap
monmaptool: generated fsid 4571c8c1-89c2-44e5-8527-247470f74809
epoch 0
fsid 4571c8c1-89c2-44e5-8527-247470f74809
last_changed 2018-10-02 13:30:43.174771
created 2018-10-02 13:30:43.174771
0: 10.10.11.128:6789/0 mon.nora
monmaptool: writing epoch 0 to /tmp/monmap (1 monitors)
Created symlink /etc/systemd/system/ceph-mon.target.wants/ceph-mon@nora.service -> /lib/systemd/system/ceph-mon@.service.
admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
INFO:ceph-create-keys:Key exists already: /etc/ceph/ceph.client.admin.keyring
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
creating manager directory '/var/lib/ceph/mgr/ceph-nora'
creating keys for 'mgr.nora'
setting owner for directory
enabling service 'ceph-mgr@nora.service'
Created symlink /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@nora.service -> /lib/systemd/system/ceph-mgr@.service.
starting service 'ceph-mgr@nora.service'

less /etc/pve/storage.cfg

dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

zfspool: local-zfs
        pool rpool/data
        sparse
        content images,rootdir

less /etc/pve/ceph.conf

[global]
         auth client required = cephx
         auth cluster required = cephx
         auth service required = cephx
         cluster network = 10.10.11.0/24
         fsid = 5df2a0f7-2362-488e-9c5a-4b9ed2a16bfe
         keyring = /etc/pve/priv/$cluster.$name.keyring
         mon allow pool delete = true
         osd journal size = 5120
         osd pool default min size = 2
         osd pool default size = 3
         public network = 10.10.11.0/24

[osd]
         keyring = /var/lib/ceph/osd/ceph-$id/keyring

[mon.nora]
         host = nora
         mon addr = 10.10.11.128:6789

Hinzufügen von anderen Servern zum Pool von Ceph

• nora -> Ceph -> Monitor -> Create

  Create Ceph Monitor/Manager

  Host

  cora

Task viewer: Ceph Monitor mon.cora - Create

WUI

Output

Created symlink /etc/systemd/system/ceph-mon.target.wants/ceph-mon@cora.service -> /lib/systemd/system/ceph-mon@.service.
INFO:ceph-create-keys:ceph-mon is not in quorum: u'synchronizing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:Talking to monitor...
exported keyring for client.admin
updated caps for client.admin
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
creating manager directory '/var/lib/ceph/mgr/ceph-cora'
creating keys for 'mgr.cora'
setting owner for directory
enabling service 'ceph-mgr@cora.service'
Created symlink /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@cora.service -> /lib/systemd/system/ceph-mgr@.service.
starting service 'ceph-mgr@cora.service'
TASK OK

• nora -> Ceph -> Monitor -> Create

  Create Ceph Monitor/Manager

  Host

  dora

  Create

Task viewer: Ceph Monitor mon.cora - Create

WUI

Output

Created symlink /etc/systemd/system/ceph-mon.target.wants/ceph-mon@dora.service -> /lib/systemd/system/ceph-mon@.service.
INFO:ceph-create-keys:ceph-mon is not in quorum: u'synchronizing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:Talking to monitor...
exported keyring for client.admin
updated caps for client.admin
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
creating manager directory '/var/lib/ceph/mgr/ceph-dora'
creating keys for 'mgr.dora'
setting owner for directory
enabling service 'ceph-mgr@dora.service'
Created symlink /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@dora.service -> /lib/systemd/system/ceph-mgr@.service.
starting service 'ceph-mgr@dora.service'
TASK OK

Erstellung vom Pool für das Cluster von Ceph

WUI

• nora -> Ceph -> Pools -> Create

  Create Ceph Monitor/Manager

  Name

  xora

  Size

  3

  Min. Size

  2

  Crush Rule

  replicated_rule

  pg_num

  64

  Add Storage

  X

  Create

Proxmox CT

Proxmox CT Erstellung

Proxmox CT Erstellung alle

Create CT

General

Node

Hostname

(stage) software_zweck_organisation

(stage) ist beispielsweise test oder dev, wenn es nicht produktiv sein soll.

software ist beispielsweise plone oder openldap. (Software, nicht Dienst! also nicht cms oder ldap, oder www oder acc!)

zweck ist beispielsweise website-2010 oder nothilfe-2020.

organisation ist beispielsweise stura-htw-dresden oder kss-sachsen.

Unprivileged container

[X]

Nesting

[?]

Password

nach Lebensdauer

bei Projekten mindestens 8

Template

Storage

cephfs

Type

rdb

Template

Root Disk

Storage

storage

Disk size (GiB)

nach Bedarf

CPU

Cores

2, oder nach Bedarf

Die Verwendung von nur einen Prozessor oder mehr als zwei Prozessor ist zu begründen.

Memory

nach Bedarf

Memory (MiB)

nach Bedarf

Swap (MiB)

nach Bedarf

Die Größe soll der Größe, die bei Memory (MiB) verwendet wird, entsprechen. (Die Verwendung von mehr oder weniger Größe ist zu begründen.)

Die Größe soll höchstens die Hälfte der Größe, die bei (Root Disk ->) Disk size (GiB) verwendet wird, betragen.

Network

Bridge

vmbr1

IPv4

Static

IPv4/CIDR

141.56.51.321/24

Gateway (IPv4)

141.56.51.254

321 ist die "verwendbare" Adresse für IPv4, die unverzüglich bei Intern:Server#Verwendung von IP-Adressen einzutragen ist.

DNS

DNS domain

DNS servers

141.56.1.1

Confirm

Start after created

[ ]

Finish

Datacenter (cluster)

HA

Resources

Add

VM

110

110 ist die ID der Instanz innerhalb von Proxmox.

Group

HA_cluster

Add

Proxmox CT Erstellung TurnKey

Create CT

General

Unprivileged conatainer

[ ]

turnkey-init

Proxmox CT Verwaltung

Auflisten aller Dateien für die Konfiguration der Container auf dem jeweiligen node

ls /etc/pve/lxc/

Ausfallsicherheit

Ausfallsicherheit CT

ping 141.56.51.321

PING 141.56.51.321 (141.56.51.321) 56(84) bytes of data.
64 bytes from 141.56.51.321: icmp_seq=1 ttl=64 time=0.283 ms
64 bytes from 141.56.51.321: icmp_seq=2 ttl=64 time=0.213 ms
64 bytes from 141.56.51.321: icmp_seq=3 ttl=64 time=0.286 ms
From 141.56.51.456 icmp_seq=4 Destination Host Unreachable
From 141.56.51.456 icmp_seq=5 Destination Host Unreachable

From 141.56.51.456 icmp_seq=78 Destination Host Unreachable
From 141.56.51.456 icmp_seq=79 Destination Host Unreachable
64 bytes from 141.56.51.321: icmp_seq=80 ttl=64 time=2609 ms
64 bytes from 141.56.51.321: icmp_seq=81 ttl=64 time=1585 ms
64 bytes from 141.56.51.321: icmp_seq=82 ttl=64 time=561 ms
64 bytes from 141.56.51.321: icmp_seq=83 ttl=64 time=0.260 ms
64 bytes from 141.56.51.321: icmp_seq=84 ttl=64 time=0.295 ms
64 bytes from 141.56.51.157: icmp_seq=85 ttl=64 time=0.200 ms
64 bytes from 141.56.51.157: icmp_seq=86 ttl=64 time=0.274 ms

Ausfallsicherheit Fehler

Fehler bei der Ausfallsicherheit bei der Verwendung von ZFS, statt RDB

2020-04-29 09:14:46 starting migration of CT 110 to node 'n1' (10.1.0.31)
2020-04-29 09:14:46 found local volume 'local-zfs:subvol-110-disk-0' (in current VM config)
cannot open 'rpool/data/subvol-110-disk-0': dataset does not exist
usage:
	snapshot [-r] [-o property=value] ... <filesystem|volume>@<snap> ...
For the property list, run: zfs set|get
2020-04-29 09:14:46 ERROR: zfs error: For the delegated permission list, run: zfs allow|unallow
2020-04-29 09:14:46 aborting phase 1 - cleanup resources
2020-04-29 09:14:46 ERROR: found stale volume copy 'local-zfs:subvol-110-disk-0' on node 'n1'
2020-04-29 09:14:46 start final cleanup
2020-04-29 09:14:46 ERROR: migration aborted (duration 00:00:01): zfs error: For the delegated permission list, run: zfs allow|unallow
TASK ERROR: migration aborted

Anpassungen für Produktivebetrieb

• Backup Limit auf 10 erhöht.
• Vorbereitung für Plone 5 Umzug (101) Plone 5

Siehe auch

• Server/srs2709