Intern:Cluster/xora: Unterschied zwischen den Versionen

← Zum vorherigen Versionsunterschied

Aktuelle Version vom 17. Oktober 2022, 15:06 Uhr

Cluster/xora war ein erster Test zu einem Verbund (Cluster) von Servern mit Proxmox VE.

Später wurde der Test erneut bei der Server/Aktualisierung/2019 als Intern:ora vorgenommen.

Test

Grundsätzlich soll ein möglichst ausfallsicherer Verbund an Servern mit Proxmox VE getestet werden.

Getestet werden soll:

Datensicherung
- Replikation
- …
Ausfallüberbrückung
…

Bezeichnung

Es ist die künstlich geschaffene Bezeichnung für den Verbund von mehreren Servern.

Verbund

Für einen Verbund an Servern mit Proxmox VE bedarf es wohl mindestens 3 Servern.

Zu dem Verbund gehören (theoretisch)

mindestens

cora (zuvor srs27090)
und
dora (zuvor srs27091)

zur Vervollständigung auch

nora (zuvor srs3008)
oder/und
lora (zuvor (angedacht) srs85290)

zur Ergänzung eventuell auch

zora (zuvor srs5100)

.

Geräte

einzelne Geräte

Gerät

Name

IPv4 OS

DNS (A) OS

WUI OS

Mail OS

Geburtshilfe

Start (Dauer)

IPv4 IPMI

DNS (A) IPMI

WUI IPMI

Mail IPMI

Netzwerkschnittstellen

Massenspeicher

Eigentum

27090

cora

141.56.51.123

cora.stura-dresden.de

https://cora.stura-dresden.de:8006/

cora@stura.htw-dresden.de

James

3 min

141.56.51.113

irmc.cora.stura-dresden.de

https://irmc.cora.stura-dresden.de/

~~irmc.cora@stura.htw-dresden.de~~

M	2	1

X

3.5 ″	3.5 ″


2 TB	2 TB

bsd.services:user:vater:hw#rx300_s6_2709_0

27091

dora

141.56.51.124

dora.stura-dresden.de

https://dora.stura-dresden.de:8006/

dora@stura.htw-dresden.de

Fullforce

3 min

141.56.51.114

irmc.dora.stura-dresden.de

https://irmc.dora.stura-dresden.de/

~~irmc.dora@stura.htw-dresden.de~~

M	2	1

X

3.5 ″	3.5 ″


2 TB	2 TB

bsd.services:user:vater:hw#rx300_s6_2709_1

8529

~~lora~~

141.56.51.127

lora.stura-dresden.de

~~https://lora.stura-dresden.de:8006/~~

lora@stura.htw-dresden.de

min

141.56.51.117

irmc.lora.stura-dresden.de

https://irmc.lora.stura-dresden.de/

~~irmc.lora@stura.htw-dresden.de~~

M	2	1

3.5 ″	3.5 ″


2 TB	2 TB

bsd.services:user:vater:hw#rx300_s6_8529

8

nora

141.56.51.128

nora.stura-dresden.de

https://nora.stura-dresden.de:8006/

nora@stura.htw-dresden.de

2 min

141.56.51.118

irmc.nora.stura-dresden.de

https://irmc.nora.stura-dresden.de/

~~irmc.nora@stura.htw-dresden.de~~

M	2	1

X

3.5 ″	3.5 ″


2 TB	2 TB

StuRa (srs3008)

5100

~~zora~~

141.56.51.129

zora.stura-dresden.de

~~https://zora.stura-dresden.de:8006/~~

zora@stura.htw-dresden.de

min

141.56.51.119

drac.cora.stura-dresden.de

https://drac.cora.stura-dresden.de/

~~drac.zora@stura.htw-dresden.de~~

M

1
2

3.5 ″	3.5 ″	3.5 ″	3.5 ″
2 TB	2 TB
2 TB	2 TB

bsd.services:user:vater:hw#dell_poweredge_r510

BIOS

F2

Boot: ~~alles andere~~; USB KEY: …; PCI SCSI: #0100 ID000 LN0 HGST H; PCI SCSI: #0100 ID004 LN0 HGST H

Proxmox VE

Installation Betriebssystem

Vorbereitung Installation Betreibssystem

Durchführung Installation Betreibssystem

Install Proxmox VE

Loading Proxmox Installer ...
Loading initial ramdisk ...

Proxmox startup

End User License Agreement (EULA)

I agree

Proxmox Virtualiszation Environment (PVE)

Options

Filesystem

~~ext4~~zfs (RAID1)

Disk Setup

Harddisk 0: /dev/sda (1863GB, HUS726020ALS214)
Harddisk 1: /dev/sdb (1863GB, HUS726020ALS214)
Harddisk 2: -- do not use --

Advanced Options

ashift: 12
compress: on
checksum: on
copies: 1

OK

Target: zfs (RAID1)

Country: Germany
Time zone: Europe/Berlin
Keyboard Layout: German

Password: 8
Confirm: 8
E-Mail: siehe #einzelne Geräte

Management Interface: enp8s0f0 - … (igb)
Hostname (FQDN): siehe #einzelne Geräte
IP Address: siehe #einzelne Geräte
Netmask: 255.255.255.0
Gateway: 141.56.51.254
DNS Server: 141.56.1.1

Installation successful!

Reboot

Nachbereitung Installation Betreibssystem

erste Aktualisierung

Update (WUI)
- Refresh (WUI)
  Neustart (WUI)
- Upgrade (WUI)

(optionales) ZFS anschauen: zpool status

  pool: rpool
 state: ONLINE
  scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	rpool       ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    sda2    ONLINE       0     0     0
	    sdb2    ONLINE       0     0     0

errors: No known data errors

zfs list

NAME               USED  AVAIL  REFER  MOUNTPOINT
rpool             9.40G  1.75T   104K  /rpool
rpool/ROOT         919M  1.75T    96K  /rpool/ROOT
rpool/ROOT/pve-1   919M  1.75T   919M  /
rpool/data          96K  1.75T    96K  /rpool/data
rpool/swap        8.50G  1.75T    56K  -

(optionales) Anschauen der Partitionierung

fdisk -l /dev/sd{a,b}

Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 0A3CA01C-D0CE-4750-A26A-C07C1541EF1D

Device          Start        End    Sectors  Size Type
/dev/sda1          34       2047       2014 1007K BIOS boot
/dev/sda2        2048 3907012749 3907010702  1.8T Solaris /usr & Apple ZFS
/dev/sda9  3907012750 3907029134      16385    8M Solaris reserved 1


Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C0D3B0CA-C966-4B00-B367-EEDBD04872F7

Device          Start        End    Sectors  Size Type
/dev/sdb1          34       2047       2014 1007K BIOS boot
/dev/sdb2        2048 3907012749 3907010702  1.8T Solaris /usr & Apple ZFS
/dev/sdb9  3907012750 3907029134      16385    8M Solaris reserved 1

Sicherung des initialen Zustandes von PVE (mit der erfolgten Aktualisierung)

Erstellen eines Schnappschusses vom gesamten Pool

zfs snapshot -r rpool@fresh-installed-pve-and-updated

Im Übrigen kann überlegt werden, dass darüber hinaus auch eine Sicherung von /dev/sd{a,b}1 vorgenommen wird.

Anpassung der Quellen

Siehe auch: Server/Proxmox#sources.list

Ergänzung der Quelle pve-no-subscription

auf schnell!

echo 'deb http://download.proxmox.com/debian/pve stretch pve-no-subscription' > /etc/apt/sources.list.d/pve-download.list && apt update

cat /etc/apt/sources.list.d/pve-enterprise.list

cat: /etc/apt/sources.list.d/pve-enterprise.list: No such file or directory

$EDITOR /etc/apt/sources.list.d/pve-enterprise.list

cat /etc/apt/sources.list.d/pve-enterprise.list

deb http://download.proxmox.com/debian/pve stretch pve-no-subscription

Entfernung der Quelle pve-enterprise

cat /etc/apt/sources.list.d/pve-enterprise.list

deb https://enterprise.proxmox.com/debian/pve stretch pve-enterprise

$EDITOR /etc/apt/sources.list.d/pve-enterprise.list

cat /etc/apt/sources.list.d/pve-enterprise.list

####deb https://enterprise.proxmox.com/debian/pve stretch pve-enterprise

Erstellung vom Cluster

optionales Begutachten für die Erstellung von einem Cluster

	cora	dora	nora
`less /etc/network/interfaces`
	auto lo iface lo inet loopback iface enp8s0f0 inet manual auto vmbr0 iface vmbr0 inet static address 141.56.51.123 netmask 255.255.255.0 gateway 141.56.51.254 bridge_ports enp8s0f0 bridge_stp off bridge_fd 0 iface ens4f0 inet manual iface ens4f1 inet manual iface ens4f2 inet manual iface ens4f3 inet manual iface enp8s0f1 inet manual	auto lo iface lo inet loopback iface enp8s0f0 inet manual auto vmbr0 iface vmbr0 inet static address 141.56.51.124 netmask 255.255.255.0 gateway 141.56.51.254 bridge_ports enp8s0f0 bridge_stp off bridge_fd 0 iface ens4f0 inet manual iface ens4f1 inet manual iface ens4f2 inet manual iface ens4f3 inet manual iface enp8s0f1 inet manual	auto lo iface lo inet loopback iface eno1 inet manual auto vmbr0 iface vmbr0 inet static address 141.56.51.128 netmask 255.255.255.0 gateway 141.56.51.254 bridge_ports eno1 bridge_stp off bridge_fd 0 iface enp2s0f0 inet manual iface enp2s0f1 inet manual iface enp2s0f2 inet manual iface enp2s0f3 inet manual iface eno2 inet manual
`less /etc/hosts`
	127.0.0.1 localhost.localdomain localhost 141.56.51.123 cora.stura-dresden.de cora pvelocalhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts	127.0.0.1 localhost.localdomain localhost 141.56.51.124 dora.stura-dresden.de dora pvelocalhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts	127.0.0.1 localhost.localdomain localhost 141.56.51.128 nora.stura-dresden.de nora pvelocalhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts

Erzeugung vom Cluster xora

auf einer der Server, die zum Cluster gehören sollen

vorgenommen auf #8 (nora)

(alternativ) grafische Oberfläche Kommandozeile

Datacenter -> Cluster -> Create Cluster
Create Cluster
Cluster Name

xora

Ring 0 Address

Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/urandom.
Writing corosync key to /etc/corosync/authkey.
Writing corosync config to /etc/pve/corosync.conf
Restart corosync and cluster filesystem
TASK OK

pvecm create xora

Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/urandom.
Writing corosync key to /etc/corosync/authkey.
Writing corosync config to /etc/pve/corosync.conf
Restart corosync and cluster filesystem

less /etc/pve/corosync.conf

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: nora
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 141.56.51.128
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: xora
  config_version: 1
  interface {
    bindnetaddr: 141.56.51.128
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2

pvecm status

Quorum information
------------------
Date:             Fri Mmm dd HH:MM:SS yyyy
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000001
Ring ID:          1/12
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   1
Highest expected: 1
Total votes:      1
Quorum:           1  
Flags:            Quorate 

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 141.56.51.128 (local)

Erstellung von einem internen Netz für das Cluster von Proxmox

frei gewählt verwenden wir 10.10.10.0/24

cora dora nora

grafische Oberfläche (mit notwendigen Neustart)

less /etc/network/interfaces

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp8s0f0 inet manual

iface ens4f0 inet manual

iface ens4f1 inet manual

iface ens4f2 inet manual

iface ens4f3 inet manual

auto enp8s0f1
iface enp8s0f1 inet static
        address  10.10.10.123
        netmask  255.255.255.0

auto vmbr0
iface vmbr0 inet static
        address  141.56.51.123
        netmask  255.255.255.0
        gateway  141.56.51.254
        bridge_ports enp8s0f0
        bridge_stp off
        bridge_fd 0

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp8s0f0 inet manual

iface ens4f0 inet manual

iface ens4f1 inet manual

iface ens4f2 inet manual

iface ens4f3 inet manual

auto enp8s0f1
iface enp8s0f1 inet static
        address  10.10.10.124
        netmask  255.255.255.0

auto vmbr0
iface vmbr0 inet static
        address  141.56.51.124
        netmask  255.255.255.0
        gateway  141.56.51.254
        bridge_ports enp8s0f0
        bridge_stp off
        bridge_fd 0

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual

iface enp2s0f0 inet manual

iface enp2s0f1 inet manual

iface enp2s0f2 inet manual

iface enp2s0f3 inet manual

auto eno2
iface eno2 inet static
        address  10.10.10.128
        netmask  255.255.255.0

auto vmbr0
iface vmbr0 inet static
        address  141.56.51.128
        netmask  255.255.255.0
        gateway  141.56.51.254
        bridge_ports eno1
        bridge_stp off
        bridge_fd 0

Eintragung der anderen Nodes unabhängig von DNS

cora dora nora

less /etc/hosts

####    members of the cluster xora
10.10.10.123 cora.xora.stura-dresden.de cora.xora
10.10.10.124 dora.xora.stura-dresden.de dora.xora
10.10.10.128 nora.xora.stura-dresden.de nora.xora

# The following lines are desirable for IPv6 capable hosts

Hinzufügen von anderen Servern

cora dora nora

pvecm add nora.xora

Please enter superuser (root) password for 'nora.xora':

                                                       Password for root@nora.xora: ********

Etablishing API connection with host 'nora.xora'
The authenticity of host 'nora.xora' can't be established.
X509 SHA256 key fingerprint is F3:A9:2D:9E:D5:59:DA:AE:5E:76:71:1E:02:D9:49:B6:67:5C:40:B0:0C:C0:05:FF:C5:D7:62:37:00:D8:CA:DD.

Are you sure you want to continue connecting (yes/no)? yes

Login succeeded.
Request addition of this node
Join request OK, finishing setup locally
stopping pve-cluster service
backup old database to '/var/lib/pve-cluster/backup/config-1538462961.sql.gz'
waiting for quorum...OK
(re)generate node files
generate new node certificate
merge authorized SSH keys and known hosts
generated new node certificate, restart pveproxy and pvedaemon services
successfully added node 'cora' to cluster.

pvecm add nora.xora

Please enter superuser (root) password for 'nora.xora':

                                                       Password for root@nora.xora: ********

Etablishing API connection with host 'nora.xora'
The authenticity of host 'nora.xora' can't be established.
X509 SHA256 key fingerprint is F3:A9:2D:9E:D5:59:DA:AE:5E:76:71:1E:02:D9:49:B6:67:5C:40:B0:0C:C0:05:FF:C5:D7:62:37:00:D8:CA:DD.

Are you sure you want to continue connecting (yes/no)? yes

Login succeeded.
Request addition of this node
Join request OK, finishing setup locally
stopping pve-cluster service
backup old database to '/var/lib/pve-cluster/backup/config-1538462994.sql.gz'
waiting for quorum...OK
(re)generate node files
generate new node certificate
merge authorized SSH keys and known hosts
generated new node certificate, restart pveproxy and pvedaemon services
successfully added node 'dora' to cluster.

/etc/pve/corosync.conf

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: cora
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 141.56.51.123
  }
  node {
    name: dora
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 141.56.51.124
  }
  node {
    name: nora
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 141.56.51.128
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: xora
  config_version: 3
  interface {
    bindnetaddr: 141.56.51.128
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2
}

(auch in Anlehnung an https://pve.proxmox.com/wiki/Separate_Cluster_Network)

/etc/pve/corosync.conf

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: cora
    nodeid: 3
    quorum_votes: 1
    ring0_addr: cora.xora
  }
  node {
    name: dora
    nodeid: 4
    quorum_votes: 1
    ring0_addr: dora.xora
  }
  node {
    name: nora
    nodeid: 8
    quorum_votes: 1
    ring0_addr: nora.xora
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: xora
  config_version: 4
  interface {
    bindnetaddr: nora.xora
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2
}

Prüfen vom Verbund

cora dora nora

pvecm status

Quorum information
------------------
Date:             Fri Mmm dd HH:MM:SS yyyy
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000008
Ring ID:          3/52
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2  
Flags:            Quorate 

Membership information
----------------------
    Nodeid      Votes Name
0x00000003          1 10.10.10.123
0x00000004          1 10.10.10.124
0x00000008          1 10.10.10.128 (local)

Erstellen vom Cluster für Ceph

Erstellung von einem internen Netz für das Cluster von Ceph

frei gewählt verwenden wir 10.10.11.0/24

Netzwerkschnittstellen
10.10.11.0/24
10.10.11.123/24

10.10.11.124/24

10.10.11.128/24

Neustart (wegen Netzwerkschnittstellen)

Testweises Verbinden (zu den jeweils anderen beiden Servern im Cluster) per ssh

ssh root@10.10.11.123

ssh root@10.10.11.124

ssh root@10.10.11.128

Installation von Ceph

pveceph install

#Ergänzung der Quelle pve-no-subscription

pveceph install

Initialisieren von Ceph

gemäß doku 10.10.10.0/24, aber das wurde (versehentlich schon für die #Erstellung von einem internen Netz für das Cluster von Proxmox verwendet)

pveceph init --network 10.10.11.0/24

WUI

rados_connect failed - No such file or directory (500)

pveceph createmon

creating /etc/pve/priv/ceph.client.admin.keyring
monmaptool: monmap file /tmp/monmap
monmaptool: generated fsid 4571c8c1-89c2-44e5-8527-247470f74809
epoch 0
fsid 4571c8c1-89c2-44e5-8527-247470f74809
last_changed 2018-10-02 13:30:43.174771
created 2018-10-02 13:30:43.174771
0: 10.10.11.128:6789/0 mon.nora
monmaptool: writing epoch 0 to /tmp/monmap (1 monitors)
Created symlink /etc/systemd/system/ceph-mon.target.wants/ceph-mon@nora.service -> /lib/systemd/system/ceph-mon@.service.
admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
INFO:ceph-create-keys:Key exists already: /etc/ceph/ceph.client.admin.keyring
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
creating manager directory '/var/lib/ceph/mgr/ceph-nora'
creating keys for 'mgr.nora'
setting owner for directory
enabling service 'ceph-mgr@nora.service'
Created symlink /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@nora.service -> /lib/systemd/system/ceph-mgr@.service.
starting service 'ceph-mgr@nora.service'

less /etc/pve/storage.cfg

dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

zfspool: local-zfs
        pool rpool/data
        sparse
        content images,rootdir

less /etc/pve/ceph.conf

[global]
         auth client required = cephx
         auth cluster required = cephx
         auth service required = cephx
         cluster network = 10.10.11.0/24
         fsid = 5df2a0f7-2362-488e-9c5a-4b9ed2a16bfe
         keyring = /etc/pve/priv/$cluster.$name.keyring
         mon allow pool delete = true
         osd journal size = 5120
         osd pool default min size = 2
         osd pool default size = 3
         public network = 10.10.11.0/24

[osd]
         keyring = /var/lib/ceph/osd/ceph-$id/keyring

[mon.nora]
         host = nora
         mon addr = 10.10.11.128:6789

Hinzufügen von anderen Servern zum Pool von Ceph

nora -> Ceph -> Monitor -> Create
Create Ceph Monitor/Manager
Host

cora

Task viewer: Ceph Monitor mon.cora - Create

WUI

Output

Created symlink /etc/systemd/system/ceph-mon.target.wants/ceph-mon@cora.service -> /lib/systemd/system/ceph-mon@.service.
INFO:ceph-create-keys:ceph-mon is not in quorum: u'synchronizing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:Talking to monitor...
exported keyring for client.admin
updated caps for client.admin
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
creating manager directory '/var/lib/ceph/mgr/ceph-cora'
creating keys for 'mgr.cora'
setting owner for directory
enabling service 'ceph-mgr@cora.service'
Created symlink /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@cora.service -> /lib/systemd/system/ceph-mgr@.service.
starting service 'ceph-mgr@cora.service'
TASK OK

nora -> Ceph -> Monitor -> Create
Create Ceph Monitor/Manager
Host

dora

Create

Task viewer: Ceph Monitor mon.cora - Create

WUI

Output

Created symlink /etc/systemd/system/ceph-mon.target.wants/ceph-mon@dora.service -> /lib/systemd/system/ceph-mon@.service.
INFO:ceph-create-keys:ceph-mon is not in quorum: u'synchronizing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:Talking to monitor...
exported keyring for client.admin
updated caps for client.admin
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
INFO:ceph-create-keys:Talking to monitor...
creating manager directory '/var/lib/ceph/mgr/ceph-dora'
creating keys for 'mgr.dora'
setting owner for directory
enabling service 'ceph-mgr@dora.service'
Created symlink /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@dora.service -> /lib/systemd/system/ceph-mgr@.service.
starting service 'ceph-mgr@dora.service'
TASK OK

Erstellung vom Pool für das Cluster von Ceph

WUI

nora -> Ceph -> Pools -> Create
Create Ceph Monitor/Manager
Name

xora

Size

3

Min. Size

2

Crush Rule

replicated_rule

pg_num

64

Add Storage

X

Create

Containerisierung

Proxmox CT

Proxmox CT Erstellung

Proxmox CT Erstellung alle

Create CT

General

Node
Hostname: (stage) software_zweck_organisation; (stage) ist beispielsweise test oder dev, wenn es nicht produktiv sein soll.; software ist beispielsweise plone oder openldap. (Software, nicht Dienst! also nicht cms oder ldap, oder www oder acc!); zweck ist beispielsweise website-2010 oder nothilfe-2020.; organisation ist beispielsweise stura-htw-dresden oder kss-sachsen.
Unprivileged container: [X]
Nesting: [?]
Password: nach Lebensdauer; bei Projekten mindestens 8

Template

Storage

cephfs

Type: rdb

Template

Root Disk

Storage: storage
Disk size (GiB): nach Bedarf

CPU

Cores: 2, oder nach Bedarf; Die Verwendung von nur einen Prozessor oder mehr als zwei Prozessor ist zu begründen.

Memory

nach Bedarf
Memory (MiB): nach Bedarf
Swap (MiB): nach Bedarf; Die Größe soll der Größe, die bei Memory (MiB) verwendet wird, entsprechen. (Die Verwendung von mehr oder weniger Größe ist zu begründen.); Die Größe soll höchstens die Hälfte der Größe, die bei (Root Disk ->) Disk size (GiB) verwendet wird, betragen.

Network

Bridge: vmbr1
IPv4: Static
IPv4/CIDR: 141.56.51.321/24
Gateway (IPv4): 141.56.51.254; 321 ist die "verwendbare" Adresse für IPv4, die unverzüglich bei Intern:Server#Verwendung von IP-Adressen einzutragen ist.

DNS

DNS domain
DNS servers: 141.56.1.1

Confirm

Start after created: [ ]
Finish

Datacenter (cluster)

HA

Resources

Add

VM: 110; 110 ist die ID der Instanz innerhalb von Proxmox.
Group: HA_cluster
Add

Proxmox CT Erstellung TurnKey

Create CT

General

Unprivileged conatainer: [ ]

turnkey-init

Proxmox CT Verwaltung

Auflisten aller Dateien für die Konfiguration der Container auf dem jeweiligen node

ls /etc/pve/lxc/

Virtualisierung

Proxmox VM

Ausfallsicherheit

Ausfallsicherheit CT

ping 141.56.51.321

PING 141.56.51.321 (141.56.51.321) 56(84) bytes of data.
64 bytes from 141.56.51.321: icmp_seq=1 ttl=64 time=0.283 ms
64 bytes from 141.56.51.321: icmp_seq=2 ttl=64 time=0.213 ms
64 bytes from 141.56.51.321: icmp_seq=3 ttl=64 time=0.286 ms
From 141.56.51.456 icmp_seq=4 Destination Host Unreachable
From 141.56.51.456 icmp_seq=5 Destination Host Unreachable

From 141.56.51.456 icmp_seq=78 Destination Host Unreachable
From 141.56.51.456 icmp_seq=79 Destination Host Unreachable
64 bytes from 141.56.51.321: icmp_seq=80 ttl=64 time=2609 ms
64 bytes from 141.56.51.321: icmp_seq=81 ttl=64 time=1585 ms
64 bytes from 141.56.51.321: icmp_seq=82 ttl=64 time=561 ms
64 bytes from 141.56.51.321: icmp_seq=83 ttl=64 time=0.260 ms
64 bytes from 141.56.51.321: icmp_seq=84 ttl=64 time=0.295 ms
64 bytes from 141.56.51.157: icmp_seq=85 ttl=64 time=0.200 ms
64 bytes from 141.56.51.157: icmp_seq=86 ttl=64 time=0.274 ms

Ausfallsicherheit Fehler

Fehler bei der Ausfallsicherheit bei der Verwendung von ZFS, statt RDB

2020-04-29 09:14:46 starting migration of CT 110 to node 'n1' (10.1.0.31)
2020-04-29 09:14:46 found local volume 'local-zfs:subvol-110-disk-0' (in current VM config)
cannot open 'rpool/data/subvol-110-disk-0': dataset does not exist
usage:
	snapshot [-r] [-o property=value] ... <filesystem|volume>@<snap> ...
For the property list, run: zfs set|get
2020-04-29 09:14:46 ERROR: zfs error: For the delegated permission list, run: zfs allow|unallow
2020-04-29 09:14:46 aborting phase 1 - cleanup resources
2020-04-29 09:14:46 ERROR: found stale volume copy 'local-zfs:subvol-110-disk-0' on node 'n1'
2020-04-29 09:14:46 start final cleanup
2020-04-29 09:14:46 ERROR: migration aborted (duration 00:00:01): zfs error: For the delegated permission list, run: zfs allow|unallow
TASK ERROR: migration aborted

Anpassungen für Produktivebetrieb

Backup Limit auf 10 erhöht.
Vorbereitung für Plone 5 Umzug (101) Plone 5

Siehe auch

Server/srs2709

@@ Zeile 1: / Zeile 1: @@
-[[{{FULLPAGENAME}}|{{PAGENAME}}]] soll ein Verbund (Cluster) von Servern mit [[Proxmox VE]] sein. Es handelt sich vorerst um einen Test. Wenn der Test zufriedenstellend verläuft, soll damit dann der gesamte Betrieb zu [[Server]]n damit realisiert werden.
+[[{{FULLPAGENAME}}|{{PAGENAME}}]] war ein erster Test zu einem Verbund (Cluster) von Servern mit [[Proxmox VE]].
+Später wurde der Test erneut bei der [[Server/Aktualisierung/2019]] als [[Intern:ora]] vorgenommen.
 == Test ==
@@ Zeile 350: / Zeile 352: @@
 === Installation Betriebssystem ===
+==== Vorbereitung Installation Betreibssystem ====
+==== Durchführung Installation Betreibssystem ====
   '''Install Proxmox VE'''
@@ Zeile 401: / Zeile 407: @@
 : <u>R</u>eboot
-=== nach der Installation ===
+==== Nachbereitung Installation Betreibssystem ====
 ; erste Aktualisierung:
@@ Zeile 473: / Zeile 479: @@
 Im Übrigen kann überlegt werden, dass darüber hinaus auch eine Sicherung von /dev/sd{a,b}1 vorgenommen wird.
-=== Erstellung vom Cluster ===
+==== Anpassung der Quellen ====
+; Siehe auch: [[Server/Proxmox#sources.list]]
+===== Ergänzung der Quelle pve-no-subscription =====
-==== optionales Begutachten für die Erstellung von einem Cluster ====
+----
+auf schnell!
+: <code>echo 'deb http://download.proxmox.com/debian/pve stretch pve-no-subscription' > /etc/apt/sources.list.d/pve-download.list && apt update</code>
+----
-{|
+<code>cat /etc/apt/sources.list.d/pve-enterprise.list</code>
-!
-! cora
-! dora
-! nora
-|-
-|
-: <code>less /etc/network/interfaces</code>
-|-
-|
-|
 <pre>
-auto lo
+cat: /etc/apt/sources.list.d/pve-enterprise.list: No such file or directory
-iface lo inet loopback
+</pre>
-iface enp8s0f0 inet manual
+<code>$EDITOR /etc/apt/sources.list.d/pve-enterprise.list</code>
+<pre>
+</pre>
-auto vmbr0
+<code>cat /etc/apt/sources.list.d/pve-enterprise.list</code>
-iface vmbr0 inet static
+<pre>
-        address 141.56.51.123
+deb http://download.proxmox.com/debian/pve stretch pve-no-subscription
-        netmask 255.255.255.0
+</pre>
-        gateway 141.56.51.254
-        bridge_ports enp8s0f0
-        bridge_stp off
-        bridge_fd 0
-iface ens4f0 inet manual
+===== Entfernung der Quelle pve-enterprise =====
-iface ens4f1 inet manual
+<code>cat /etc/apt/sources.list.d/pve-enterprise.list</code>
+<pre>
+deb https://enterprise.proxmox.com/debian/pve stretch pve-enterprise
+</pre>
-iface ens4f2 inet manual
+<code>$EDITOR /etc/apt/sources.list.d/pve-enterprise.list</code>
+<pre>
+</pre>
-iface ens4f3 inet manual
+<code>cat /etc/apt/sources.list.d/pve-enterprise.list</code>
+<pre>
-iface enp8s0f1 inet manual
+####deb https://enterprise.proxmox.com/debian/pve stretch pve-enterprise
 </pre>
-|
-<pre>
-auto lo
-iface lo inet loopback
-iface enp8s0f0 inet manual
+=== Erstellung vom Cluster ===
-auto vmbr0
+==== optionales Begutachten für die Erstellung von einem Cluster ====
+{|
+!
+! cora
+! dora
+! nora
+|-
+|
+: <code>less /etc/network/interfaces</code>
+|-
+|
+|
+<pre>
+auto lo
+iface lo inet loopback
+iface enp8s0f0 inet manual
+auto vmbr0
 iface vmbr0 inet static
-         address 141.56.51.124
+         address 141.56.51.123
          netmask 255.255.255.0
          gateway 141.56.51.254
@@ Zeile 544: / Zeile 565: @@
 iface lo inet loopback
-iface eno1 inet manual
+iface enp8s0f0 inet manual
 auto vmbr0
 iface vmbr0 inet static
-         address 141.56.51.128
+         address 141.56.51.124
          netmask 255.255.255.0
          gateway 141.56.51.254
-         bridge_ports eno1
+        bridge_ports enp8s0f0
+        bridge_stp off
+        bridge_fd 0
+iface ens4f0 inet manual
+iface ens4f1 inet manual
+iface ens4f2 inet manual
+iface ens4f3 inet manual
+iface enp8s0f1 inet manual
+</pre>
+|
+<pre>
+auto lo
+iface lo inet loopback
+iface eno1 inet manual
+auto vmbr0
+iface vmbr0 inet static
+        address 141.56.51.128
+        netmask 255.255.255.0
+        gateway 141.56.51.254
+         bridge_ports eno1
          bridge_stp off
          bridge_fd 0
@@ Zeile 707: / Zeile 754: @@
 </pre>
-==== Erstellung von einem internen Netz ====
+==== Erstellung von einem internen Netz für das Cluster von Proxmox ====
 frei gewählt verwenden wir 10.10.10.0/24
@@ Zeile 1.085: / Zeile 1.132: @@
 === Erstellen vom Cluster für Ceph ===
+==== Erstellung von einem internen Netz für das Cluster von Ceph ====
+frei gewählt verwenden wir 10.10.11.0/24
 * Netzwerkschnittstellen
@@ Zeile 1.094: / Zeile 1.145: @@
 Neustart (wegen Netzwerkschnittstellen)
-Testweises Verbinden (zu den jeweils anderen beiden Servern im Cluster) per <code>ssh</code>
+<!--
-: <code>ssh root@10.10.11.123</code>
+---- nur vorlage, um die inhalte nachzutragen
-<!--
+{|
+!
+! cora
+! dora
+! nora
+|-
+| colspan="4" |
+grafische Oberfläche (mit notwendigen Neustart)
+|-
+|
+: <code>less /etc/network/interfaces</code>
+|-
+|
+|
 <pre>
-The authenticity of host '10.10.11.123 (10.10.11.123)' can't be established.
+# network interface settings; autogenerated
-</pre>
+# Please do NOT modify this file directly, unless you know what
-<pre></pre>
+# you're doing.
-<pre>
+#
-Are you sure you want to continue connecting (yes/no)? yes
+# If you want to manage part of the network configuration manually,
-</pre>
+# please utilize the 'source' or 'source-directory' directives to do
-<pre>
+# so.
-Warning: Permanently added '10.10.11.123' (ECDSA) to the list of known hosts.
+# PVE will preserve these directives, but will NOT its network
-Linux nora 4.15.17-1-pve #1 SMP PVE 4.15.17-9 (Wed, 9 May 2018 13:31:43 +0200) x86_64
+# configuration from sourced files, so do not attempt to move any of
+# the PVE managed interfaces into external files!
+auto lo
+iface lo inet loopback
+iface enp8s0f0 inet manual
+iface ens4f0 inet manual
+iface ens4f1 inet manual
-The programs included with the Debian GNU/Linux system are free software;
+iface ens4f2 inet manual
-the exact distribution terms for each program are described in the
-individual files in /usr/share/doc/*/copyright.
-Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
+iface ens4f3 inet manual
-permitted by applicable law.
-</pre>
-<pre></pre>
--->
-: <code>ssh root@10.10.11.124</code>
-<!--
-<pre>
-The authenticity of host '10.10.11.124 (10.10.11.124)' can't be established.
-</pre>
-<pre></pre>
-<pre>
-Are you sure you want to continue connecting (yes/no)? yes
-</pre>
-<pre>
-Warning: Permanently added '10.10.11.124' (ECDSA) to the list of known hosts.
-Linux nora 4.15.17-1-pve #1 SMP PVE 4.15.17-9 (Wed, 9 May 2018 13:31:43 +0200) x86_64
-The programs included with the Debian GNU/Linux system are free software;
+auto enp8s0f1
-the exact distribution terms for each program are described in the
+iface enp8s0f1 inet static
-individual files in /usr/share/doc/*/copyright.
+        address  10.10.10.123
+        netmask  255.255.255.0
-Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
+auto vmbr0
-permitted by applicable law.
+iface vmbr0 inet static
+        address  141.56.51.123
+        netmask  255.255.255.0
+        gateway  141.56.51.254
+        bridge_ports enp8s0f0
+        bridge_stp off
+        bridge_fd 0
 </pre>
-<pre></pre>
+|
--->
-: <code>ssh root@10.10.11.128</code>
-<!--
 <pre>
-The authenticity of host '10.10.11.128 (10.10.11.128)' can't be established.
+# network interface settings; autogenerated
-</pre>
+# Please do NOT modify this file directly, unless you know what
-<pre></pre>
+# you're doing.
-<pre>
+#
-Are you sure you want to continue connecting (yes/no)? yes
+# If you want to manage part of the network configuration manually,
-</pre>
+# please utilize the 'source' or 'source-directory' directives to do
-<pre>
+# so.
-Warning: Permanently added '10.10.11.128' (ECDSA) to the list of known hosts.
+# PVE will preserve these directives, but will NOT its network
-Linux nora 4.15.17-1-pve #1 SMP PVE 4.15.17-9 (Wed, 9 May 2018 13:31:43 +0200) x86_64
+# configuration from sourced files, so do not attempt to move any of
+# the PVE managed interfaces into external files!
-The programs included with the Debian GNU/Linux system are free software;
+auto lo
-the exact distribution terms for each program are described in the
+iface lo inet loopback
-individual files in /usr/share/doc/*/copyright.
+iface enp8s0f0 inet manual
+iface ens4f0 inet manual
+iface ens4f1 inet manual
+iface ens4f2 inet manual
-Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
+iface ens4f3 inet manual
-permitted by applicable law.
-</pre>
-<pre></pre>
--->
-: <code>pveceph install</code>
+auto enp8s0f1
-<!--
+iface enp8s0f1 inet static
-<pre>
+        address  10.10.10.124
-update available package list
+        netmask  255.255.255.0
-Reading package lists... Done
-Building dependency tree
+auto vmbr0
-Reading state information... Done
+iface vmbr0 inet static
-gdisk is already the newest version (1.0.1-1).
+        address  141.56.51.124
-The following additional packages will be installed:
+        netmask  255.255.255.0
-  binutils ceph-base ceph-mgr ceph-mon ceph-osd cryptsetup-bin libcephfs2 libcurl3 libgoogle-perftools4
+        gateway  141.56.51.254
-  libjs-jquery libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust-ctl2 liblttng-ust0 libparted2 librados2
+        bridge_ports enp8s0f0
-  libradosstriper1 librbd1 librgw2 libtcmalloc-minimal4 libunwind8 parted python-bs4 python-cephfs
+        bridge_stp off
-  python-cffi-backend python-cherrypy3 python-click python-colorama python-cryptography python-dnspython
+        bridge_fd 0
-  python-enum34 python-flask python-formencode python-idna python-ipaddress python-itsdangerous python-jinja2
+</pre>
-  python-logutils python-mako python-markupsafe python-openssl python-paste python-pastedeploy
+|
-  python-pastedeploy-tpl python-pecan python-prettytable python-pyasn1 python-rados python-rbd python-repoze.lru
+<pre>
-  python-rgw python-routes python-setuptools python-simplegeneric python-singledispatch python-tempita
+# network interface settings; autogenerated
-  python-waitress python-webob python-webtest python-werkzeug
+# Please do NOT modify this file directly, unless you know what
-Suggested packages:
+# you're doing.
-  binutils-doc ceph-mds libparted-dev libparted-i18n parted-doc python-cryptography-doc python-cryptography-vectors
+#
-  python-enum34-doc python-flask-doc python-egenix-mxdatetime python-jinja2-doc python-beaker python-mako-doc
+# If you want to manage part of the network configuration manually,
-  python-openssl-doc python-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi python-pastescript
+# please utilize the 'source' or 'source-directory' directives to do
-  python-pastewebkit doc-base python-setuptools-doc python-waitress-doc python-webob-doc python-webtest-doc ipython
+# so.
-  python-genshi python-lxml python-greenlet python-redis python-pylibmc | python-memcache python-werkzeug-doc
+# PVE will preserve these directives, but will NOT its network
-Recommended packages:
+# configuration from sourced files, so do not attempt to move any of
-  ceph-mds ntp | time-daemon javascript-common python-lxml | python-html5lib python-blinker python-simplejson
+# the PVE managed interfaces into external files!
-  libjs-mochikit python-openid python-scgi python-pastescript python-lxml python-pyquery python-pyinotify
-The following packages will be REMOVED:
+auto lo
-  libpve-guest-common-perl libpve-storage-perl proxmox-ve pve-container pve-ha-manager pve-manager qemu-server
+iface lo inet loopback
-The following NEW packages will be installed:
-  binutils ceph ceph-base ceph-mgr ceph-mon ceph-osd cryptsetup-bin libcephfs2 libcurl3 libgoogle-perftools4
+iface eno1 inet manual
-  libjs-jquery libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust-ctl2 liblttng-ust0 libparted2
-  libtcmalloc-minimal4 libunwind8 parted python-bs4 python-cffi-backend python-cherrypy3 python-click
+iface enp2s0f0 inet manual
-  python-colorama python-cryptography python-dnspython python-enum34 python-flask python-formencode python-idna
-  python-ipaddress python-itsdangerous python-jinja2 python-logutils python-mako python-markupsafe python-openssl
+iface enp2s0f1 inet manual
-  python-paste python-pastedeploy python-pastedeploy-tpl python-pecan python-prettytable python-pyasn1
-  python-repoze.lru python-rgw python-routes python-setuptools python-simplegeneric python-singledispatch
+iface enp2s0f2 inet manual
-  python-tempita python-waitress python-webob python-webtest python-werkzeug
-The following packages will be upgraded:
+iface enp2s0f3 inet manual
-  ceph-common librados2 libradosstriper1 librbd1 librgw2 python-cephfs python-rados python-rbd
-upgraded, 55 newly installed, 7 to remove and 1 not upgraded.
+auto eno2
-Need to get 54.8 MB of archives.
+iface eno2 inet static
-After this operation, 169 MB of additional disk space will be used.
+        address  10.10.10.128
-Do you want to continue? [Y/n]
+         netmask  255.255.255.0
-Get:1 http://ftp.de.debian.org/debian stretch/main amd64 binutils amd64 2.28-5 [3,770 kB]
-Get:2 http://security.debian.org stretch/updates/main amd64 libcurl3 amd64 7.52.1-5+deb9u7 [292 kB]
+auto vmbr0
-Get:3 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 ceph-common amd64 12.2.8-pve1 [12.9 MB]
+iface vmbr0 inet static
-Get:4 http://ftp.de.debian.org/debian stretch/main amd64 liblttng-ust-ctl2 amd64 2.9.0-2+deb9u1 [99.4 kB]
+        address  141.56.51.128
-Get:5 http://ftp.de.debian.org/debian stretch/main amd64 liblttng-ust0 amd64 2.9.0-2+deb9u1 [174 kB]
+        netmask  255.255.255.0
-Get:6 http://ftp.de.debian.org/debian stretch/main amd64 python-prettytable all 0.7.2-3 [22.4 kB]
+         gateway  141.56.51.254
-Get:7 http://ftp.de.debian.org/debian stretch/main amd64 libtcmalloc-minimal4 amd64 2.5-2.2 [121 kB]
+        bridge_ports eno1
-Get:8 http://ftp.de.debian.org/debian stretch/main amd64 libunwind8 amd64 1.1-4.1 [48.7 kB]
+        bridge_stp off
-Get:9 http://ftp.de.debian.org/debian stretch/main amd64 libgoogle-perftools4 amd64 2.5-2.2 [224 kB]
+        bridge_fd 0
-Get:10 http://ftp.de.debian.org/debian stretch/main amd64 libleveldb1v5 amd64 1.18-5 [136 kB]
+</pre>
-Get:11 http://ftp.de.debian.org/debian stretch/main amd64 cryptsetup-bin amd64 2:1.7.3-4 [221 kB]
+|-
-Get:12 http://ftp.de.debian.org/debian stretch/main amd64 python-repoze.lru all 0.6-6 [12.3 kB]
+|}
-Get:13 http://ftp.de.debian.org/debian stretch/main amd64 libjs-jquery all 3.1.1-2 [154 kB]
+-->
-Get:14 http://ftp.de.debian.org/debian stretch/main amd64 libjs-underscore all 1.8.3~dfsg-1 [63.8 kB]
-Get:15 http://ftp.de.debian.org/debian stretch/main amd64 libjs-sphinxdoc all 1.4.9-2 [69.5 kB]
+Testweises Verbinden (zu den jeweils anderen beiden Servern im Cluster) per <code>ssh</code>
-Get:16 http://ftp.de.debian.org/debian stretch/main amd64 python-routes all 2.3.1-2 [100 kB]
+: <code>ssh root@10.10.11.123</code>
-Get:17 http://ftp.de.debian.org/debian stretch/main amd64 python-cherrypy3 all 3.5.0-2 [1,321 kB]
+<!--
-Get:18 http://ftp.de.debian.org/debian stretch/main amd64 python-markupsafe amd64 0.23-3 [14.4 kB]
+<pre>
-Get:19 http://ftp.de.debian.org/debian stretch/main amd64 python-jinja2 all 2.8-1 [111 kB]
+The authenticity of host '10.10.11.123 (10.10.11.123)' can't be established.
-Get:20 http://ftp.de.debian.org/debian stretch/main amd64 python-cffi-backend amd64 1.9.1-2 [69.0 kB]
+</pre>
-Get:21 http://ftp.de.debian.org/debian stretch/main amd64 python-enum34 all 1.1.6-1 [35.0 kB]
+<pre></pre>
-Get:22 http://ftp.de.debian.org/debian stretch/main amd64 python-idna all 2.2-1 [32.6 kB]
+<pre>
-Get:23 http://ftp.de.debian.org/debian stretch/main amd64 python-ipaddress all 1.0.17-1 [18.1 kB]
+Are you sure you want to continue connecting (yes/no)? yes
-Get:24 http://ftp.de.debian.org/debian stretch/main amd64 python-pyasn1 all 0.1.9-2 [51.8 kB]
+</pre>
-Get:25 http://ftp.de.debian.org/debian stretch/main amd64 python-setuptools all 33.1.1-1 [297 kB]
+<pre>
-Get:26 http://ftp.de.debian.org/debian stretch/main amd64 python-cryptography amd64 1.7.1-3 [211 kB]
+Warning: Permanently added '10.10.11.123' (ECDSA) to the list of known hosts.
-Get:27 http://ftp.de.debian.org/debian stretch/main amd64 python-openssl all 16.2.0-1 [43.7 kB]
+Linux nora 4.15.17-1-pve #1 SMP PVE 4.15.17-9 (Wed, 9 May 2018 13:31:43 +0200) x86_64
-Get:28 http://ftp.de.debian.org/debian stretch/main amd64 python-logutils all 0.3.3-5 [17.2 kB]
-Get:29 http://ftp.de.debian.org/debian stretch/main amd64 python-mako all 1.0.6+ds1-2 [62.1 kB]
+The programs included with the Debian GNU/Linux system are free software;
-Get:30 http://ftp.de.debian.org/debian stretch/main amd64 python-simplegeneric all 0.8.1-1 [11.9 kB]
+the exact distribution terms for each program are described in the
-Get:31 http://ftp.de.debian.org/debian stretch/main amd64 python-singledispatch all 3.4.0.3-2 [9,690 B]
+individual files in /usr/share/doc/*/copyright.
-Get:32 http://ftp.de.debian.org/debian stretch/main amd64 python-webob all 1:1.6.2-2 [63.7 kB]
-Get:33 http://ftp.de.debian.org/debian stretch/main amd64 python-bs4 all 4.5.3-1 [86.7 kB]
+Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
-Get:34 http://ftp.de.debian.org/debian stretch/main amd64 python-waitress all 1.0.1-1 [54.3 kB]
+permitted by applicable law.
-Get:35 http://ftp.de.debian.org/debian stretch/main amd64 python-dnspython all 1.15.0-1 [102 kB]
+</pre>
-Get:36 http://ftp.de.debian.org/debian stretch/main amd64 python-formencode all 1.3.0-2 [140 kB]
+<pre></pre>
-Get:37 http://ftp.de.debian.org/debian stretch/main amd64 python-tempita all 0.5.2-2 [13.8 kB]
+-->
-Get:38 http://ftp.de.debian.org/debian stretch/main amd64 python-paste all 2.0.3+dfsg-4 [474 kB]
+: <code>ssh root@10.10.11.124</code>
-Get:39 http://ftp.de.debian.org/debian stretch/main amd64 python-pastedeploy-tpl all 1.5.2-4 [8,024 B]
+<!--
-Get:40 http://ftp.de.debian.org/debian stretch/main amd64 python-pastedeploy all 1.5.2-4 [30.4 kB]
+<pre>
-Get:41 http://ftp.de.debian.org/debian stretch/main amd64 python-webtest all 2.0.24-1 [34.0 kB]
+The authenticity of host '10.10.11.124 (10.10.11.124)' can't be established.
-Get:42 http://ftp.de.debian.org/debian stretch/main amd64 python-pecan all 1.1.2-3 [104 kB]
+</pre>
-Get:43 http://ftp.de.debian.org/debian stretch/main amd64 python-werkzeug all 0.11.15+dfsg1-1 [173 kB]
+<pre></pre>
-Get:44 http://ftp.de.debian.org/debian stretch/main amd64 python-colorama all 0.3.7-1 [25.7 kB]
+<pre>
-Get:45 http://ftp.de.debian.org/debian stretch/main amd64 python-click all 6.6-1 [56.1 kB]
+Are you sure you want to continue connecting (yes/no)? yes
-Get:46 http://ftp.de.debian.org/debian stretch/main amd64 python-itsdangerous all 0.24+dfsg1-2 [13.0 kB]
-Get:47 http://ftp.de.debian.org/debian stretch/main amd64 python-flask all 0.12.1-1 [62.2 kB]
-Get:48 http://ftp.de.debian.org/debian stretch/main amd64 libparted2 amd64 3.2-17 [276 kB]
-Get:49 http://ftp.de.debian.org/debian stretch/main amd64 parted amd64 3.2-17 [194 kB]
-Get:50 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 python-rados amd64 12.2.8-pve1 [289 kB]
-Get:51 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 librgw2 amd64 12.2.8-pve1 [1,806 kB]
-Get:52 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 libradosstriper1 amd64 12.2.8-pve1 [320 kB]
-Get:53 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 python-rbd amd64 12.2.8-pve1 [155 kB]
-Get:54 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 librbd1 amd64 12.2.8-pve1 [993 kB]
-Get:55 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 librados2 amd64 12.2.8-pve1 [2,723 kB]
-Get:56 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 libcephfs2 amd64 12.2.8-pve1 [410 kB]
-Get:57 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 python-cephfs amd64 12.2.8-pve1 [94.2 kB]
-Get:58 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 python-rgw amd64 12.2.8-pve1 [98.2 kB]
-Get:59 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 ceph-base amd64 12.2.8-pve1 [3,330 kB]
-Get:60 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 ceph-mgr amd64 12.2.8-pve1 [3,520 kB]
-Get:61 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 ceph-mon amd64 12.2.8-pve1 [4,474 kB]
-Get:62 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 ceph-osd amd64 12.2.8-pve1 [14.0 MB]
-Get:63 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 ceph amd64 12.2.8-pve1 [7,164 B]
-Fetched 54.8 MB in 2s (19.6 MB/s)
-W: (pve-apt-hook) !! WARNING !!
-W: (pve-apt-hook) You are attempting to remove the meta-package 'proxmox-ve'!
-W: (pve-apt-hook)
-W: (pve-apt-hook) If you really you want to permanently remove 'proxmox-ve' from your system, run the following command
-W: (pve-apt-hook) 	touch '/please-remove-proxmox-ve'
-W: (pve-apt-hook) and repeat your apt-get/apt invocation.
-W: (pve-apt-hook)
-W: (pve-apt-hook) If you are unsure why 'proxmox-ve' would be removed, please verify
-W: (pve-apt-hook) 	- your APT repository settings
-W: (pve-apt-hook) 	- that you are using 'apt-get dist-upgrade' or 'apt full-upgrade' to upgrade your system
-E: Sub-process /usr/share/proxmox-ve/pve-apt-hook returned an error code (1)
-E: Failure running script /usr/share/proxmox-ve/pve-apt-hook
 </pre>
--->
+<pre>
+Warning: Permanently added '10.10.11.124' (ECDSA) to the list of known hosts.
+Linux nora 4.15.17-1-pve #1 SMP PVE 4.15.17-9 (Wed, 9 May 2018 13:31:43 +0200) x86_64
-: <code>echo 'deb http://download.proxmox.com/debian/pve stretch pve-no-subscription' > /etc/apt/sources.list.d/pve-download.list && apt update</code>
+The programs included with the Debian GNU/Linux system are free software;
+the exact distribution terms for each program are described in the
+individual files in /usr/share/doc/*/copyright.
-: <code>pveceph install</code>
+Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
+permitted by applicable law.
+</pre>
+<pre></pre>
+-->
+: <code>ssh root@10.10.11.128</code>
 <!--
 <pre>
-update available package list
+The authenticity of host '10.10.11.128 (10.10.11.128)' can't be established.
-Reading package lists... Done
+</pre>
-Building dependency tree
+<pre></pre>
-Reading state information... Done
+<pre>
-gdisk is already the newest version (1.0.1-1).
+Are you sure you want to continue connecting (yes/no)? yes
-The following additional packages will be installed:
+</pre>
-   binutils ceph-base ceph-fuse ceph-mgr ceph-mon ceph-osd cryptsetup-bin libcephfs2 libcurl3 libgoogle-perftools4
+<pre>
-   libjs-jquery libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust-ctl2 liblttng-ust0 libparted2
+Warning: Permanently added '10.10.11.128' (ECDSA) to the list of known hosts.
-   libpve-common-perl libpve-guest-common-perl libpve-storage-perl librados2 libradosstriper1 librbd1 librgw2
+Linux nora 4.15.17-1-pve #1 SMP PVE 4.15.17-9 (Wed, 9 May 2018 13:31:43 +0200) x86_64
-  libtcmalloc-minimal4 libunwind8 parted pve-manager python-bs4 python-cephfs python-cffi-backend python-cherrypy3
-  python-click python-colorama python-cryptography python-dnspython python-enum34 python-flask python-formencode
+The programs included with the Debian GNU/Linux system are free software;
-  python-idna python-ipaddress python-itsdangerous python-jinja2 python-logutils python-mako python-markupsafe
+the exact distribution terms for each program are described in the
-  python-openssl python-paste python-pastedeploy python-pastedeploy-tpl python-pecan python-prettytable
+individual files in /usr/share/doc/*/copyright.
-  python-pyasn1 python-rados python-rbd python-repoze.lru python-rgw python-routes python-setuptools
-  python-simplegeneric python-singledispatch python-tempita python-waitress python-webob python-webtest
+Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
-  python-werkzeug qemu-server
+permitted by applicable law.
-Suggested packages:
+</pre>
-   binutils-doc ceph-mds libparted-dev libparted-i18n parted-doc python-cryptography-doc python-cryptography-vectors
+<pre></pre>
-   python-enum34-doc python-flask-doc python-egenix-mxdatetime python-jinja2-doc python-beaker python-mako-doc
+-->
-   python-openssl-doc python-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi python-pastescript
+==== Installation von Ceph ====
+: <code>pveceph install</code>
+<!--
+<pre>
+update available package list
+Reading package lists... Done
+Building dependency tree
+Reading state information... Done
+gdisk is already the newest version (1.0.1-1).
+The following additional packages will be installed:
+   binutils ceph-base ceph-mgr ceph-mon ceph-osd cryptsetup-bin libcephfs2 libcurl3 libgoogle-perftools4
+   libjs-jquery libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust-ctl2 liblttng-ust0 libparted2 librados2
+   libradosstriper1 librbd1 librgw2 libtcmalloc-minimal4 libunwind8 parted python-bs4 python-cephfs
+  python-cffi-backend python-cherrypy3 python-click python-colorama python-cryptography python-dnspython
+  python-enum34 python-flask python-formencode python-idna python-ipaddress python-itsdangerous python-jinja2
+  python-logutils python-mako python-markupsafe python-openssl python-paste python-pastedeploy
+  python-pastedeploy-tpl python-pecan python-prettytable python-pyasn1 python-rados python-rbd python-repoze.lru
+  python-rgw python-routes python-setuptools python-simplegeneric python-singledispatch python-tempita
+  python-waitress python-webob python-webtest python-werkzeug
+Suggested packages:
+   binutils-doc ceph-mds libparted-dev libparted-i18n parted-doc python-cryptography-doc python-cryptography-vectors
+   python-enum34-doc python-flask-doc python-egenix-mxdatetime python-jinja2-doc python-beaker python-mako-doc
+   python-openssl-doc python-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi python-pastescript
    python-pastewebkit doc-base python-setuptools-doc python-waitress-doc python-webob-doc python-webtest-doc ipython
    python-genshi python-lxml python-greenlet python-redis python-pylibmc | python-memcache python-werkzeug-doc
@@ Zeile 1.315: / Zeile 1.381: @@
    ceph-mds ntp | time-daemon javascript-common python-lxml | python-html5lib python-blinker python-simplejson
    libjs-mochikit python-openid python-scgi python-pastescript python-lxml python-pyquery python-pyinotify
+The following packages will be REMOVED:
+  libpve-guest-common-perl libpve-storage-perl proxmox-ve pve-container pve-ha-manager pve-manager qemu-server
 The following NEW packages will be installed:
-   binutils ceph ceph-base ceph-fuse ceph-mgr ceph-mon ceph-osd cryptsetup-bin libcephfs2 libcurl3
+   binutils ceph ceph-base ceph-mgr ceph-mon ceph-osd cryptsetup-bin libcephfs2 libcurl3 libgoogle-perftools4
-  libgoogle-perftools4 libjs-jquery libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust-ctl2 liblttng-ust0
+  libjs-jquery libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust-ctl2 liblttng-ust0 libparted2
-   libparted2 libtcmalloc-minimal4 libunwind8 parted python-bs4 python-cffi-backend python-cherrypy3 python-click
+   libtcmalloc-minimal4 libunwind8 parted python-bs4 python-cffi-backend python-cherrypy3 python-click
    python-colorama python-cryptography python-dnspython python-enum34 python-flask python-formencode python-idna
    python-ipaddress python-itsdangerous python-jinja2 python-logutils python-mako python-markupsafe python-openssl
@@ Zeile 1.325: / Zeile 1.393: @@
    python-tempita python-waitress python-webob python-webtest python-werkzeug
 The following packages will be upgraded:
-   ceph-common libpve-common-perl libpve-guest-common-perl libpve-storage-perl librados2 libradosstriper1 librbd1
+   ceph-common librados2 libradosstriper1 librbd1 librgw2 python-cephfs python-rados python-rbd
-  librgw2 pve-manager python-cephfs python-rados python-rbd qemu-server
+upgraded, 55 newly installed, 7 to remove and 1 not upgraded.
-upgraded, 56 newly installed, 0 to remove and 24 not upgraded.
+Need to get 54.8 MB of archives.
-Need to get 4,734 kB/59.5 MB of archives.
+After this operation, 169 MB of additional disk space will be used.
-After this operation, 189 MB of additional disk space will be used.
 Do you want to continue? [Y/n]
-Get:1 http://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 qemu-server amd64 5.0-36 [169 kB]
+Get:1 http://ftp.de.debian.org/debian stretch/main amd64 binutils amd64 2.28-5 [3,770 kB]
-Get:2 http://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 libpve-guest-common-perl all 2.0-18 [16.7 kB]
+Get:2 http://security.debian.org stretch/updates/main amd64 libcurl3 amd64 7.52.1-5+deb9u7 [292 kB]
-Get:3 http://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 pve-manager amd64 5.2-9 [1,922 kB]
+Get:3 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 ceph-common amd64 12.2.8-pve1 [12.9 MB]
-Get:4 http://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 libpve-common-perl all 5.0-40 [90.8 kB]
+Get:4 http://ftp.de.debian.org/debian stretch/main amd64 liblttng-ust-ctl2 amd64 2.9.0-2+deb9u1 [99.4 kB]
-Get:5 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 ceph-fuse amd64 12.2.8-pve1 [2,448 kB]
+Get:5 http://ftp.de.debian.org/debian stretch/main amd64 liblttng-ust0 amd64 2.9.0-2+deb9u1 [174 kB]
-Get:6 http://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 libpve-storage-perl all 5.0-29 [88.3 kB]
+Get:6 http://ftp.de.debian.org/debian stretch/main amd64 python-prettytable all 0.7.2-3 [22.4 kB]
-Fetched 4,734 kB in 0s (13.2 MB/s)
+Get:7 http://ftp.de.debian.org/debian stretch/main amd64 libtcmalloc-minimal4 amd64 2.5-2.2 [121 kB]
-apt-listchanges: Reading changelogs...
+Get:8 http://ftp.de.debian.org/debian stretch/main amd64 libunwind8 amd64 1.1-4.1 [48.7 kB]
-Extracting templates from packages: 100%
+Get:9 http://ftp.de.debian.org/debian stretch/main amd64 libgoogle-perftools4 amd64 2.5-2.2 [224 kB]
-Selecting previously unselected package binutils.
+Get:10 http://ftp.de.debian.org/debian stretch/main amd64 libleveldb1v5 amd64 1.18-5 [136 kB]
-(Reading database ... 40422 files and directories currently installed.)
+Get:11 http://ftp.de.debian.org/debian stretch/main amd64 cryptsetup-bin amd64 2:1.7.3-4 [221 kB]
-Preparing to unpack .../00-binutils_2.28-5_amd64.deb ...
+Get:12 http://ftp.de.debian.org/debian stretch/main amd64 python-repoze.lru all 0.6-6 [12.3 kB]
-Unpacking binutils (2.28-5) ...
+Get:13 http://ftp.de.debian.org/debian stretch/main amd64 libjs-jquery all 3.1.1-2 [154 kB]
-Preparing to unpack .../01-qemu-server_5.0-36_amd64.deb ...
+Get:14 http://ftp.de.debian.org/debian stretch/main amd64 libjs-underscore all 1.8.3~dfsg-1 [63.8 kB]
-Unpacking qemu-server (5.0-36) over (5.0-26) ...
+Get:15 http://ftp.de.debian.org/debian stretch/main amd64 libjs-sphinxdoc all 1.4.9-2 [69.5 kB]
-Preparing to unpack .../02-libpve-guest-common-perl_2.0-18_all.deb ...
+Get:16 http://ftp.de.debian.org/debian stretch/main amd64 python-routes all 2.3.1-2 [100 kB]
-Unpacking libpve-guest-common-perl (2.0-18) over (2.0-16) ...
+Get:17 http://ftp.de.debian.org/debian stretch/main amd64 python-cherrypy3 all 3.5.0-2 [1,321 kB]
-Preparing to unpack .../03-pve-manager_5.2-9_amd64.deb ...
+Get:18 http://ftp.de.debian.org/debian stretch/main amd64 python-markupsafe amd64 0.23-3 [14.4 kB]
-Unpacking pve-manager (5.2-9) over (5.2-1) ...
+Get:19 http://ftp.de.debian.org/debian stretch/main amd64 python-jinja2 all 2.8-1 [111 kB]
-Preparing to unpack .../04-libpve-common-perl_5.0-40_all.deb ...
+Get:20 http://ftp.de.debian.org/debian stretch/main amd64 python-cffi-backend amd64 1.9.1-2 [69.0 kB]
-Unpacking libpve-common-perl (5.0-40) over (5.0-31) ...
+Get:21 http://ftp.de.debian.org/debian stretch/main amd64 python-enum34 all 1.1.6-1 [35.0 kB]
-Selecting previously unselected package libtcmalloc-minimal4.
+Get:22 http://ftp.de.debian.org/debian stretch/main amd64 python-idna all 2.2-1 [32.6 kB]
-Preparing to unpack .../05-libtcmalloc-minimal4_2.5-2.2_amd64.deb ...
+Get:23 http://ftp.de.debian.org/debian stretch/main amd64 python-ipaddress all 1.0.17-1 [18.1 kB]
-Unpacking libtcmalloc-minimal4 (2.5-2.2) ...
+Get:24 http://ftp.de.debian.org/debian stretch/main amd64 python-pyasn1 all 0.1.9-2 [51.8 kB]
-Selecting previously unselected package libunwind8.
+Get:25 http://ftp.de.debian.org/debian stretch/main amd64 python-setuptools all 33.1.1-1 [297 kB]
-Preparing to unpack .../06-libunwind8_1.1-4.1_amd64.deb ...
+Get:26 http://ftp.de.debian.org/debian stretch/main amd64 python-cryptography amd64 1.7.1-3 [211 kB]
-Unpacking libunwind8 (1.1-4.1) ...
+Get:27 http://ftp.de.debian.org/debian stretch/main amd64 python-openssl all 16.2.0-1 [43.7 kB]
-Selecting previously unselected package libgoogle-perftools4.
+Get:28 http://ftp.de.debian.org/debian stretch/main amd64 python-logutils all 0.3.3-5 [17.2 kB]
-Preparing to unpack .../07-libgoogle-perftools4_2.5-2.2_amd64.deb ...
+Get:29 http://ftp.de.debian.org/debian stretch/main amd64 python-mako all 1.0.6+ds1-2 [62.1 kB]
-Unpacking libgoogle-perftools4 (2.5-2.2) ...
+Get:30 http://ftp.de.debian.org/debian stretch/main amd64 python-simplegeneric all 0.8.1-1 [11.9 kB]
-Selecting previously unselected package ceph-fuse.
+Get:31 http://ftp.de.debian.org/debian stretch/main amd64 python-singledispatch all 3.4.0.3-2 [9,690 B]
-Preparing to unpack .../08-ceph-fuse_12.2.8-pve1_amd64.deb ...
+Get:32 http://ftp.de.debian.org/debian stretch/main amd64 python-webob all 1:1.6.2-2 [63.7 kB]
-Unpacking ceph-fuse (12.2.8-pve1) ...
+Get:33 http://ftp.de.debian.org/debian stretch/main amd64 python-bs4 all 4.5.3-1 [86.7 kB]
-Preparing to unpack .../09-libpve-storage-perl_5.0-29_all.deb ...
+Get:34 http://ftp.de.debian.org/debian stretch/main amd64 python-waitress all 1.0.1-1 [54.3 kB]
-Unpacking libpve-storage-perl (5.0-29) over (5.0-23) ...
+Get:35 http://ftp.de.debian.org/debian stretch/main amd64 python-dnspython all 1.15.0-1 [102 kB]
-Preparing to unpack .../10-ceph-common_12.2.8-pve1_amd64.deb ...
+Get:36 http://ftp.de.debian.org/debian stretch/main amd64 python-formencode all 1.3.0-2 [140 kB]
-Unpacking ceph-common (12.2.8-pve1) over (10.2.5-7.2) ...
+Get:37 http://ftp.de.debian.org/debian stretch/main amd64 python-tempita all 0.5.2-2 [13.8 kB]
-Preparing to unpack .../11-python-rados_12.2.8-pve1_amd64.deb ...
+Get:38 http://ftp.de.debian.org/debian stretch/main amd64 python-paste all 2.0.3+dfsg-4 [474 kB]
-Unpacking python-rados (12.2.8-pve1) over (10.2.5-7.2) ...
+Get:39 http://ftp.de.debian.org/debian stretch/main amd64 python-pastedeploy-tpl all 1.5.2-4 [8,024 B]
-Selecting previously unselected package libcurl3:amd64.
+Get:40 http://ftp.de.debian.org/debian stretch/main amd64 python-pastedeploy all 1.5.2-4 [30.4 kB]
-Preparing to unpack .../12-libcurl3_7.52.1-5+deb9u7_amd64.deb ...
+Get:41 http://ftp.de.debian.org/debian stretch/main amd64 python-webtest all 2.0.24-1 [34.0 kB]
-Unpacking libcurl3:amd64 (7.52.1-5+deb9u7) ...
+Get:42 http://ftp.de.debian.org/debian stretch/main amd64 python-pecan all 1.1.2-3 [104 kB]
-Preparing to unpack .../13-librgw2_12.2.8-pve1_amd64.deb ...
+Get:43 http://ftp.de.debian.org/debian stretch/main amd64 python-werkzeug all 0.11.15+dfsg1-1 [173 kB]
-Unpacking librgw2 (12.2.8-pve1) over (10.2.5-7.2) ...
+Get:44 http://ftp.de.debian.org/debian stretch/main amd64 python-colorama all 0.3.7-1 [25.7 kB]
-Preparing to unpack .../14-libradosstriper1_12.2.8-pve1_amd64.deb ...
+Get:45 http://ftp.de.debian.org/debian stretch/main amd64 python-click all 6.6-1 [56.1 kB]
-Unpacking libradosstriper1 (12.2.8-pve1) over (10.2.5-7.2) ...
+Get:46 http://ftp.de.debian.org/debian stretch/main amd64 python-itsdangerous all 0.24+dfsg1-2 [13.0 kB]
-Selecting previously unselected package liblttng-ust-ctl2:amd64.
+Get:47 http://ftp.de.debian.org/debian stretch/main amd64 python-flask all 0.12.1-1 [62.2 kB]
-Preparing to unpack .../15-liblttng-ust-ctl2_2.9.0-2+deb9u1_amd64.deb ...
+Get:48 http://ftp.de.debian.org/debian stretch/main amd64 libparted2 amd64 3.2-17 [276 kB]
-Unpacking liblttng-ust-ctl2:amd64 (2.9.0-2+deb9u1) ...
+Get:49 http://ftp.de.debian.org/debian stretch/main amd64 parted amd64 3.2-17 [194 kB]
-Selecting previously unselected package liblttng-ust0:amd64.
+Get:50 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 python-rados amd64 12.2.8-pve1 [289 kB]
-Preparing to unpack .../16-liblttng-ust0_2.9.0-2+deb9u1_amd64.deb ...
+Get:51 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 librgw2 amd64 12.2.8-pve1 [1,806 kB]
-Unpacking liblttng-ust0:amd64 (2.9.0-2+deb9u1) ...
+Get:52 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 libradosstriper1 amd64 12.2.8-pve1 [320 kB]
-Preparing to unpack .../17-python-rbd_12.2.8-pve1_amd64.deb ...
+Get:53 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 python-rbd amd64 12.2.8-pve1 [155 kB]
-Unpacking python-rbd (12.2.8-pve1) over (10.2.5-7.2) ...
+Get:54 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 librbd1 amd64 12.2.8-pve1 [993 kB]
-Preparing to unpack .../18-librbd1_12.2.8-pve1_amd64.deb ...
+Get:55 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 librados2 amd64 12.2.8-pve1 [2,723 kB]
-Unpacking librbd1 (12.2.8-pve1) over (10.2.5-7.2) ...
+Get:56 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 libcephfs2 amd64 12.2.8-pve1 [410 kB]
-Preparing to unpack .../19-librados2_12.2.8-pve1_amd64.deb ...
+Get:57 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 python-cephfs amd64 12.2.8-pve1 [94.2 kB]
-Unpacking librados2 (12.2.8-pve1) over (10.2.5-7.2) ...
+Get:58 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 python-rgw amd64 12.2.8-pve1 [98.2 kB]
-Selecting previously unselected package libcephfs2.
+Get:59 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 ceph-base amd64 12.2.8-pve1 [3,330 kB]
-Preparing to unpack .../20-libcephfs2_12.2.8-pve1_amd64.deb ...
+Get:60 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 ceph-mgr amd64 12.2.8-pve1 [3,520 kB]
-Unpacking libcephfs2 (12.2.8-pve1) ...
+Get:61 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 ceph-mon amd64 12.2.8-pve1 [4,474 kB]
-Preparing to unpack .../21-python-cephfs_12.2.8-pve1_amd64.deb ...
+Get:62 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 ceph-osd amd64 12.2.8-pve1 [14.0 MB]
-Unpacking python-cephfs (12.2.8-pve1) over (10.2.5-7.2) ...
+Get:63 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 ceph amd64 12.2.8-pve1 [7,164 B]
-Selecting previously unselected package python-prettytable.
+Fetched 54.8 MB in 2s (19.6 MB/s)
-Preparing to unpack .../22-python-prettytable_0.7.2-3_all.deb ...
+W: (pve-apt-hook) !! WARNING !!
-Unpacking python-prettytable (0.7.2-3) ...
+W: (pve-apt-hook) You are attempting to remove the meta-package 'proxmox-ve'!
-Selecting previously unselected package python-rgw.
+W: (pve-apt-hook)
-Preparing to unpack .../23-python-rgw_12.2.8-pve1_amd64.deb ...
+W: (pve-apt-hook) If you really you want to permanently remove 'proxmox-ve' from your system, run the following command
-Unpacking python-rgw (12.2.8-pve1) ...
+W: (pve-apt-hook) 	touch '/please-remove-proxmox-ve'
-Selecting previously unselected package libleveldb1v5:amd64.
+W: (pve-apt-hook) and repeat your apt-get/apt invocation.
-Preparing to unpack .../24-libleveldb1v5_1.18-5_amd64.deb ...
+W: (pve-apt-hook)
-Unpacking libleveldb1v5:amd64 (1.18-5) ...
+W: (pve-apt-hook) If you are unsure why 'proxmox-ve' would be removed, please verify
-Selecting previously unselected package cryptsetup-bin.
+W: (pve-apt-hook) 	- your APT repository settings
-Preparing to unpack .../25-cryptsetup-bin_2%3a1.7.3-4_amd64.deb ...
+W: (pve-apt-hook) 	- that you are using 'apt-get dist-upgrade' or 'apt full-upgrade' to upgrade your system
-Unpacking cryptsetup-bin (2:1.7.3-4) ...
+E: Sub-process /usr/share/proxmox-ve/pve-apt-hook returned an error code (1)
-Selecting previously unselected package ceph-base.
+E: Failure running script /usr/share/proxmox-ve/pve-apt-hook
-Preparing to unpack .../26-ceph-base_12.2.8-pve1_amd64.deb ...
+</pre>
-Unpacking ceph-base (12.2.8-pve1) ...
+-->
-Selecting previously unselected package python-repoze.lru.
-Preparing to unpack .../27-python-repoze.lru_0.6-6_all.deb ...
+[[#Ergänzung der Quelle pve-no-subscription]]
-Unpacking python-repoze.lru (0.6-6) ...
-Selecting previously unselected package libjs-jquery.
+: <code>pveceph install</code>
-Preparing to unpack .../28-libjs-jquery_3.1.1-2_all.deb ...
+<!--
-Unpacking libjs-jquery (3.1.1-2) ...
+<pre>
-Selecting previously unselected package libjs-underscore.
+update available package list
-Preparing to unpack .../29-libjs-underscore_1.8.3~dfsg-1_all.deb ...
+Reading package lists... Done
-Unpacking libjs-underscore (1.8.3~dfsg-1) ...
+Building dependency tree
-Selecting previously unselected package libjs-sphinxdoc.
+Reading state information... Done
-Preparing to unpack .../30-libjs-sphinxdoc_1.4.9-2_all.deb ...
+gdisk is already the newest version (1.0.1-1).
-Unpacking libjs-sphinxdoc (1.4.9-2) ...
+The following additional packages will be installed:
-Selecting previously unselected package python-routes.
+  binutils ceph-base ceph-fuse ceph-mgr ceph-mon ceph-osd cryptsetup-bin libcephfs2 libcurl3 libgoogle-perftools4
-Preparing to unpack .../31-python-routes_2.3.1-2_all.deb ...
+  libjs-jquery libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust-ctl2 liblttng-ust0 libparted2
-Unpacking python-routes (2.3.1-2) ...
+  libpve-common-perl libpve-guest-common-perl libpve-storage-perl librados2 libradosstriper1 librbd1 librgw2
-Selecting previously unselected package python-cherrypy3.
+  libtcmalloc-minimal4 libunwind8 parted pve-manager python-bs4 python-cephfs python-cffi-backend python-cherrypy3
-Preparing to unpack .../32-python-cherrypy3_3.5.0-2_all.deb ...
+  python-click python-colorama python-cryptography python-dnspython python-enum34 python-flask python-formencode
-Unpacking python-cherrypy3 (3.5.0-2) ...
+  python-idna python-ipaddress python-itsdangerous python-jinja2 python-logutils python-mako python-markupsafe
-Selecting previously unselected package python-markupsafe.
+  python-openssl python-paste python-pastedeploy python-pastedeploy-tpl python-pecan python-prettytable
-Preparing to unpack .../33-python-markupsafe_0.23-3_amd64.deb ...
+  python-pyasn1 python-rados python-rbd python-repoze.lru python-rgw python-routes python-setuptools
-Unpacking python-markupsafe (0.23-3) ...
+  python-simplegeneric python-singledispatch python-tempita python-waitress python-webob python-webtest
-Selecting previously unselected package python-jinja2.
+  python-werkzeug qemu-server
-Preparing to unpack .../34-python-jinja2_2.8-1_all.deb ...
+Suggested packages:
-Unpacking python-jinja2 (2.8-1) ...
+  binutils-doc ceph-mds libparted-dev libparted-i18n parted-doc python-cryptography-doc python-cryptography-vectors
-Selecting previously unselected package python-cffi-backend.
+  python-enum34-doc python-flask-doc python-egenix-mxdatetime python-jinja2-doc python-beaker python-mako-doc
-Preparing to unpack .../35-python-cffi-backend_1.9.1-2_amd64.deb ...
+  python-openssl-doc python-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi python-pastescript
-Unpacking python-cffi-backend (1.9.1-2) ...
+  python-pastewebkit doc-base python-setuptools-doc python-waitress-doc python-webob-doc python-webtest-doc ipython
-Selecting previously unselected package python-enum34.
+  python-genshi python-lxml python-greenlet python-redis python-pylibmc | python-memcache python-werkzeug-doc
-Preparing to unpack .../36-python-enum34_1.1.6-1_all.deb ...
+Recommended packages:
-Unpacking python-enum34 (1.1.6-1) ...
+  ceph-mds ntp | time-daemon javascript-common python-lxml | python-html5lib python-blinker python-simplejson
-Selecting previously unselected package python-idna.
+  libjs-mochikit python-openid python-scgi python-pastescript python-lxml python-pyquery python-pyinotify
-Preparing to unpack .../37-python-idna_2.2-1_all.deb ...
+The following NEW packages will be installed:
-Unpacking python-idna (2.2-1) ...
+  binutils ceph ceph-base ceph-fuse ceph-mgr ceph-mon ceph-osd cryptsetup-bin libcephfs2 libcurl3
-Selecting previously unselected package python-ipaddress.
+  libgoogle-perftools4 libjs-jquery libjs-sphinxdoc libjs-underscore libleveldb1v5 liblttng-ust-ctl2 liblttng-ust0
-Preparing to unpack .../38-python-ipaddress_1.0.17-1_all.deb ...
+  libparted2 libtcmalloc-minimal4 libunwind8 parted python-bs4 python-cffi-backend python-cherrypy3 python-click
-Unpacking python-ipaddress (1.0.17-1) ...
+  python-colorama python-cryptography python-dnspython python-enum34 python-flask python-formencode python-idna
-Selecting previously unselected package python-pyasn1.
+  python-ipaddress python-itsdangerous python-jinja2 python-logutils python-mako python-markupsafe python-openssl
-Preparing to unpack .../39-python-pyasn1_0.1.9-2_all.deb ...
+  python-paste python-pastedeploy python-pastedeploy-tpl python-pecan python-prettytable python-pyasn1
-Unpacking python-pyasn1 (0.1.9-2) ...
+  python-repoze.lru python-rgw python-routes python-setuptools python-simplegeneric python-singledispatch
-Selecting previously unselected package python-setuptools.
+  python-tempita python-waitress python-webob python-webtest python-werkzeug
-Preparing to unpack .../40-python-setuptools_33.1.1-1_all.deb ...
+The following packages will be upgraded:
-Unpacking python-setuptools (33.1.1-1) ...
+  ceph-common libpve-common-perl libpve-guest-common-perl libpve-storage-perl librados2 libradosstriper1 librbd1
-Selecting previously unselected package python-cryptography.
+  librgw2 pve-manager python-cephfs python-rados python-rbd qemu-server
-Preparing to unpack .../41-python-cryptography_1.7.1-3_amd64.deb ...
+upgraded, 56 newly installed, 0 to remove and 24 not upgraded.
-Unpacking python-cryptography (1.7.1-3) ...
+Need to get 4,734 kB/59.5 MB of archives.
-Selecting previously unselected package python-openssl.
+After this operation, 189 MB of additional disk space will be used.
-Preparing to unpack .../42-python-openssl_16.2.0-1_all.deb ...
+Do you want to continue? [Y/n]
-Unpacking python-openssl (16.2.0-1) ...
+Get:1 http://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 qemu-server amd64 5.0-36 [169 kB]
-Selecting previously unselected package python-logutils.
+Get:2 http://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 libpve-guest-common-perl all 2.0-18 [16.7 kB]
-Preparing to unpack .../43-python-logutils_0.3.3-5_all.deb ...
+Get:3 http://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 pve-manager amd64 5.2-9 [1,922 kB]
-Unpacking python-logutils (0.3.3-5) ...
+Get:4 http://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 libpve-common-perl all 5.0-40 [90.8 kB]
-Selecting previously unselected package python-mako.
+Get:5 http://download.proxmox.com/debian/ceph-luminous stretch/main amd64 ceph-fuse amd64 12.2.8-pve1 [2,448 kB]
-Preparing to unpack .../44-python-mako_1.0.6+ds1-2_all.deb ...
+Get:6 http://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 libpve-storage-perl all 5.0-29 [88.3 kB]
-Unpacking python-mako (1.0.6+ds1-2) ...
+Fetched 4,734 kB in 0s (13.2 MB/s)
-Selecting previously unselected package python-simplegeneric.
+apt-listchanges: Reading changelogs...
-Preparing to unpack .../45-python-simplegeneric_0.8.1-1_all.deb ...
+Extracting templates from packages: 100%
-Unpacking python-simplegeneric (0.8.1-1) ...
+Selecting previously unselected package binutils.
-Selecting previously unselected package python-singledispatch.
+(Reading database ... 40422 files and directories currently installed.)
-Preparing to unpack .../46-python-singledispatch_3.4.0.3-2_all.deb ...
+Preparing to unpack .../00-binutils_2.28-5_amd64.deb ...
-Unpacking python-singledispatch (3.4.0.3-2) ...
+Unpacking binutils (2.28-5) ...
-Selecting previously unselected package python-webob.
+Preparing to unpack .../01-qemu-server_5.0-36_amd64.deb ...
-Preparing to unpack .../47-python-webob_1%3a1.6.2-2_all.deb ...
+Unpacking qemu-server (5.0-36) over (5.0-26) ...
-Unpacking python-webob (1:1.6.2-2) ...
+Preparing to unpack .../02-libpve-guest-common-perl_2.0-18_all.deb ...
-Selecting previously unselected package python-bs4.
+Unpacking libpve-guest-common-perl (2.0-18) over (2.0-16) ...
-Preparing to unpack .../48-python-bs4_4.5.3-1_all.deb ...
+Preparing to unpack .../03-pve-manager_5.2-9_amd64.deb ...
-Unpacking python-bs4 (4.5.3-1) ...
+Unpacking pve-manager (5.2-9) over (5.2-1) ...
-Selecting previously unselected package python-waitress.
+Preparing to unpack .../04-libpve-common-perl_5.0-40_all.deb ...
-Preparing to unpack .../49-python-waitress_1.0.1-1_all.deb ...
+Unpacking libpve-common-perl (5.0-40) over (5.0-31) ...
-Unpacking python-waitress (1.0.1-1) ...
+Selecting previously unselected package libtcmalloc-minimal4.
-Selecting previously unselected package python-dnspython.
+Preparing to unpack .../05-libtcmalloc-minimal4_2.5-2.2_amd64.deb ...
-Preparing to unpack .../50-python-dnspython_1.15.0-1_all.deb ...
+Unpacking libtcmalloc-minimal4 (2.5-2.2) ...
-Unpacking python-dnspython (1.15.0-1) ...
+Selecting previously unselected package libunwind8.
-Selecting previously unselected package python-formencode.
+Preparing to unpack .../06-libunwind8_1.1-4.1_amd64.deb ...
-Preparing to unpack .../51-python-formencode_1.3.0-2_all.deb ...
+Unpacking libunwind8 (1.1-4.1) ...
-Unpacking python-formencode (1.3.0-2) ...
+Selecting previously unselected package libgoogle-perftools4.
-Selecting previously unselected package python-tempita.
+Preparing to unpack .../07-libgoogle-perftools4_2.5-2.2_amd64.deb ...
-Preparing to unpack .../52-python-tempita_0.5.2-2_all.deb ...
+Unpacking libgoogle-perftools4 (2.5-2.2) ...
-Unpacking python-tempita (0.5.2-2) ...
+Selecting previously unselected package ceph-fuse.
-Selecting previously unselected package python-paste.
+Preparing to unpack .../08-ceph-fuse_12.2.8-pve1_amd64.deb ...
-Preparing to unpack .../53-python-paste_2.0.3+dfsg-4_all.deb ...
+Unpacking ceph-fuse (12.2.8-pve1) ...
-Unpacking python-paste (2.0.3+dfsg-4) ...
+Preparing to unpack .../09-libpve-storage-perl_5.0-29_all.deb ...
-Selecting previously unselected package python-pastedeploy-tpl.
+Unpacking libpve-storage-perl (5.0-29) over (5.0-23) ...
-Preparing to unpack .../54-python-pastedeploy-tpl_1.5.2-4_all.deb ...
+Preparing to unpack .../10-ceph-common_12.2.8-pve1_amd64.deb ...
-Unpacking python-pastedeploy-tpl (1.5.2-4) ...
+Unpacking ceph-common (12.2.8-pve1) over (10.2.5-7.2) ...
-Selecting previously unselected package python-pastedeploy.
+Preparing to unpack .../11-python-rados_12.2.8-pve1_amd64.deb ...
-Preparing to unpack .../55-python-pastedeploy_1.5.2-4_all.deb ...
+Unpacking python-rados (12.2.8-pve1) over (10.2.5-7.2) ...
-Unpacking python-pastedeploy (1.5.2-4) ...
+Selecting previously unselected package libcurl3:amd64.
-Selecting previously unselected package python-webtest.
+Preparing to unpack .../12-libcurl3_7.52.1-5+deb9u7_amd64.deb ...
-Preparing to unpack .../56-python-webtest_2.0.24-1_all.deb ...
+Unpacking libcurl3:amd64 (7.52.1-5+deb9u7) ...
-Unpacking python-webtest (2.0.24-1) ...
+Preparing to unpack .../13-librgw2_12.2.8-pve1_amd64.deb ...
-Selecting previously unselected package python-pecan.
+Unpacking librgw2 (12.2.8-pve1) over (10.2.5-7.2) ...
-Preparing to unpack .../57-python-pecan_1.1.2-3_all.deb ...
+Preparing to unpack .../14-libradosstriper1_12.2.8-pve1_amd64.deb ...
-Unpacking python-pecan (1.1.2-3) ...
+Unpacking libradosstriper1 (12.2.8-pve1) over (10.2.5-7.2) ...
-Selecting previously unselected package python-werkzeug.
+Selecting previously unselected package liblttng-ust-ctl2:amd64.
-Preparing to unpack .../58-python-werkzeug_0.11.15+dfsg1-1_all.deb ...
+Preparing to unpack .../15-liblttng-ust-ctl2_2.9.0-2+deb9u1_amd64.deb ...
-Unpacking python-werkzeug (0.11.15+dfsg1-1) ...
+Unpacking liblttng-ust-ctl2:amd64 (2.9.0-2+deb9u1) ...
-Selecting previously unselected package ceph-mgr.
+Selecting previously unselected package liblttng-ust0:amd64.
-Preparing to unpack .../59-ceph-mgr_12.2.8-pve1_amd64.deb ...
+Preparing to unpack .../16-liblttng-ust0_2.9.0-2+deb9u1_amd64.deb ...
-Unpacking ceph-mgr (12.2.8-pve1) ...
+Unpacking liblttng-ust0:amd64 (2.9.0-2+deb9u1) ...
-Selecting previously unselected package python-colorama.
+Preparing to unpack .../17-python-rbd_12.2.8-pve1_amd64.deb ...
-Preparing to unpack .../60-python-colorama_0.3.7-1_all.deb ...
+Unpacking python-rbd (12.2.8-pve1) over (10.2.5-7.2) ...
-Unpacking python-colorama (0.3.7-1) ...
+Preparing to unpack .../18-librbd1_12.2.8-pve1_amd64.deb ...
-Selecting previously unselected package python-click.
+Unpacking librbd1 (12.2.8-pve1) over (10.2.5-7.2) ...
-Preparing to unpack .../61-python-click_6.6-1_all.deb ...
+Preparing to unpack .../19-librados2_12.2.8-pve1_amd64.deb ...
-Unpacking python-click (6.6-1) ...
+Unpacking librados2 (12.2.8-pve1) over (10.2.5-7.2) ...
-Selecting previously unselected package python-itsdangerous.
+Selecting previously unselected package libcephfs2.
-Preparing to unpack .../62-python-itsdangerous_0.24+dfsg1-2_all.deb ...
+Preparing to unpack .../20-libcephfs2_12.2.8-pve1_amd64.deb ...
-Unpacking python-itsdangerous (0.24+dfsg1-2) ...
+Unpacking libcephfs2 (12.2.8-pve1) ...
-Selecting previously unselected package python-flask.
+Preparing to unpack .../21-python-cephfs_12.2.8-pve1_amd64.deb ...
-Preparing to unpack .../63-python-flask_0.12.1-1_all.deb ...
+Unpacking python-cephfs (12.2.8-pve1) over (10.2.5-7.2) ...
-Unpacking python-flask (0.12.1-1) ...
+Selecting previously unselected package python-prettytable.
-Selecting previously unselected package ceph-mon.
+Preparing to unpack .../22-python-prettytable_0.7.2-3_all.deb ...
-Preparing to unpack .../64-ceph-mon_12.2.8-pve1_amd64.deb ...
+Unpacking python-prettytable (0.7.2-3) ...
-Unpacking ceph-mon (12.2.8-pve1) ...
+Selecting previously unselected package python-rgw.
-Selecting previously unselected package libparted2:amd64.
+Preparing to unpack .../23-python-rgw_12.2.8-pve1_amd64.deb ...
-Preparing to unpack .../65-libparted2_3.2-17_amd64.deb ...
+Unpacking python-rgw (12.2.8-pve1) ...
-Unpacking libparted2:amd64 (3.2-17) ...
+Selecting previously unselected package libleveldb1v5:amd64.
-Selecting previously unselected package parted.
+Preparing to unpack .../24-libleveldb1v5_1.18-5_amd64.deb ...
-Preparing to unpack .../66-parted_3.2-17_amd64.deb ...
+Unpacking libleveldb1v5:amd64 (1.18-5) ...
-Unpacking parted (3.2-17) ...
+Selecting previously unselected package cryptsetup-bin.
-Selecting previously unselected package ceph-osd.
+Preparing to unpack .../25-cryptsetup-bin_2%3a1.7.3-4_amd64.deb ...
-Preparing to unpack .../67-ceph-osd_12.2.8-pve1_amd64.deb ...
+Unpacking cryptsetup-bin (2:1.7.3-4) ...
-Unpacking ceph-osd (12.2.8-pve1) ...
+Selecting previously unselected package ceph-base.
-Selecting previously unselected package ceph.
+Preparing to unpack .../26-ceph-base_12.2.8-pve1_amd64.deb ...
-Preparing to unpack .../68-ceph_12.2.8-pve1_amd64.deb ...
+Unpacking ceph-base (12.2.8-pve1) ...
-Unpacking ceph (12.2.8-pve1) ...
+Selecting previously unselected package python-repoze.lru.
-Setting up python-dnspython (1.15.0-1) ...
+Preparing to unpack .../27-python-repoze.lru_0.6-6_all.deb ...
-Setting up python-idna (2.2-1) ...
+Unpacking python-repoze.lru (0.6-6) ...
-Setting up libjs-jquery (3.1.1-2) ...
+Selecting previously unselected package libjs-jquery.
-Setting up python-repoze.lru (0.6-6) ...
+Preparing to unpack .../28-libjs-jquery_3.1.1-2_all.deb ...
-Setting up python-setuptools (33.1.1-1) ...
+Unpacking libjs-jquery (3.1.1-2) ...
-Setting up python-prettytable (0.7.2-3) ...
+Selecting previously unselected package libjs-underscore.
-Setting up libparted2:amd64 (3.2-17) ...
+Preparing to unpack .../29-libjs-underscore_1.8.3~dfsg-1_all.deb ...
-Setting up python-simplegeneric (0.8.1-1) ...
+Unpacking libjs-underscore (1.8.3~dfsg-1) ...
-Setting up libjs-underscore (1.8.3~dfsg-1) ...
+Selecting previously unselected package libjs-sphinxdoc.
-Setting up liblttng-ust-ctl2:amd64 (2.9.0-2+deb9u1) ...
+Preparing to unpack .../30-libjs-sphinxdoc_1.4.9-2_all.deb ...
-Setting up libpve-common-perl (5.0-40) ...
+Unpacking libjs-sphinxdoc (1.4.9-2) ...
-Setting up libcurl3:amd64 (7.52.1-5+deb9u7) ...
+Selecting previously unselected package python-routes.
-Setting up python-pyasn1 (0.1.9-2) ...
+Preparing to unpack .../31-python-routes_2.3.1-2_all.deb ...
-Setting up libjs-sphinxdoc (1.4.9-2) ...
+Unpacking python-routes (2.3.1-2) ...
-Setting up python-webob (1:1.6.2-2) ...
+Selecting previously unselected package python-cherrypy3.
-Setting up python-colorama (0.3.7-1) ...
+Preparing to unpack .../32-python-cherrypy3_3.5.0-2_all.deb ...
-Setting up python-waitress (1.0.1-1) ...
+Unpacking python-cherrypy3 (3.5.0-2) ...
-update-alternatives: using /usr/bin/waitress-serve-python2 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode
+Selecting previously unselected package python-markupsafe.
-Setting up parted (3.2-17) ...
+Preparing to unpack .../33-python-markupsafe_0.23-3_amd64.deb ...
-Setting up libleveldb1v5:amd64 (1.18-5) ...
+Unpacking python-markupsafe (0.23-3) ...
-Setting up python-markupsafe (0.23-3) ...
+Selecting previously unselected package python-jinja2.
-Setting up python-werkzeug (0.11.15+dfsg1-1) ...
+Preparing to unpack .../34-python-jinja2_2.8-1_all.deb ...
-Setting up python-cffi-backend (1.9.1-2) ...
+Unpacking python-jinja2 (2.8-1) ...
-Setting up libtcmalloc-minimal4 (2.5-2.2) ...
+Selecting previously unselected package python-cffi-backend.
-Setting up libunwind8 (1.1-4.1) ...
+Preparing to unpack .../35-python-cffi-backend_1.9.1-2_amd64.deb ...
-Processing triggers for libc-bin (2.24-11+deb9u3) ...
+Unpacking python-cffi-backend (1.9.1-2) ...
-Setting up python-bs4 (4.5.3-1) ...
+Selecting previously unselected package python-enum34.
-Setting up python-mako (1.0.6+ds1-2) ...
+Preparing to unpack .../36-python-enum34_1.1.6-1_all.deb ...
-Setting up python-enum34 (1.1.6-1) ...
+Unpacking python-enum34 (1.1.6-1) ...
-Processing triggers for systemd (232-25+deb9u4) ...
+Selecting previously unselected package python-idna.
-Setting up cryptsetup-bin (2:1.7.3-4) ...
+Preparing to unpack .../37-python-idna_2.2-1_all.deb ...
-Processing triggers for man-db (2.7.6.1-2) ...
+Unpacking python-idna (2.2-1) ...
-Setting up binutils (2.28-5) ...
+Selecting previously unselected package python-ipaddress.
-Setting up python-singledispatch (3.4.0.3-2) ...
+Preparing to unpack .../38-python-ipaddress_1.0.17-1_all.deb ...
-Setting up python-pastedeploy-tpl (1.5.2-4) ...
+Unpacking python-ipaddress (1.0.17-1) ...
-Setting up python-itsdangerous (0.24+dfsg1-2) ...
+Selecting previously unselected package python-pyasn1.
-Setting up python-tempita (0.5.2-2) ...
+Preparing to unpack .../39-python-pyasn1_0.1.9-2_all.deb ...
-Setting up python-ipaddress (1.0.17-1) ...
+Unpacking python-pyasn1 (0.1.9-2) ...
-Setting up liblttng-ust0:amd64 (2.9.0-2+deb9u1) ...
+Selecting previously unselected package python-setuptools.
-Setting up python-logutils (0.3.3-5) ...
+Preparing to unpack .../40-python-setuptools_33.1.1-1_all.deb ...
-Setting up python-routes (2.3.1-2) ...
+Unpacking python-setuptools (33.1.1-1) ...
-Setting up python-formencode (1.3.0-2) ...
+Selecting previously unselected package python-cryptography.
-Setting up python-jinja2 (2.8-1) ...
+Preparing to unpack .../41-python-cryptography_1.7.1-3_amd64.deb ...
-Setting up python-click (6.6-1) ...
+Unpacking python-cryptography (1.7.1-3) ...
-Setting up python-paste (2.0.3+dfsg-4) ...
+Selecting previously unselected package python-openssl.
-Setting up librados2 (12.2.8-pve1) ...
+Preparing to unpack .../42-python-openssl_16.2.0-1_all.deb ...
-Setting up libcephfs2 (12.2.8-pve1) ...
+Unpacking python-openssl (16.2.0-1) ...
-Setting up libgoogle-perftools4 (2.5-2.2) ...
+Selecting previously unselected package python-logutils.
-Setting up python-cryptography (1.7.1-3) ...
+Preparing to unpack .../43-python-logutils_0.3.3-5_all.deb ...
-Setting up python-flask (0.12.1-1) ...
+Unpacking python-logutils (0.3.3-5) ...
-Setting up python-rados (12.2.8-pve1) ...
+Selecting previously unselected package python-mako.
-Setting up python-cherrypy3 (3.5.0-2) ...
+Preparing to unpack .../44-python-mako_1.0.6+ds1-2_all.deb ...
-Setting up python-cephfs (12.2.8-pve1) ...
+Unpacking python-mako (1.0.6+ds1-2) ...
-Setting up libradosstriper1 (12.2.8-pve1) ...
+Selecting previously unselected package python-simplegeneric.
-Setting up python-openssl (16.2.0-1) ...
+Preparing to unpack .../45-python-simplegeneric_0.8.1-1_all.deb ...
-Setting up librgw2 (12.2.8-pve1) ...
+Unpacking python-simplegeneric (0.8.1-1) ...
-Setting up python-rgw (12.2.8-pve1) ...
+Selecting previously unselected package python-singledispatch.
-Setting up python-pastedeploy (1.5.2-4) ...
+Preparing to unpack .../46-python-singledispatch_3.4.0.3-2_all.deb ...
-Setting up ceph-fuse (12.2.8-pve1) ...
+Unpacking python-singledispatch (3.4.0.3-2) ...
-Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target.
+Selecting previously unselected package python-webob.
-Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target.
+Preparing to unpack .../47-python-webob_1%3a1.6.2-2_all.deb ...
-Setting up librbd1 (12.2.8-pve1) ...
+Unpacking python-webob (1:1.6.2-2) ...
-Setting up python-rbd (12.2.8-pve1) ...
+Selecting previously unselected package python-bs4.
-Setting up ceph-common (12.2.8-pve1) ...
+Preparing to unpack .../48-python-bs4_4.5.3-1_all.deb ...
-Installing new version of config file /etc/default/ceph ...
+Unpacking python-bs4 (4.5.3-1) ...
-Installing new version of config file /etc/logrotate.d/ceph-common ...
+Selecting previously unselected package python-waitress.
-Setting system user ceph properties..usermod: no changes
+Preparing to unpack .../49-python-waitress_1.0.1-1_all.deb ...
-..done
+Unpacking python-waitress (1.0.1-1) ...
-Fixing /var/run/ceph ownership....done
+Selecting previously unselected package python-dnspython.
-Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target.
+Preparing to unpack .../50-python-dnspython_1.15.0-1_all.deb ...
-Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service.
+Unpacking python-dnspython (1.15.0-1) ...
-Setting up libpve-storage-perl (5.0-29) ...
+Selecting previously unselected package python-formencode.
-Setting up python-webtest (2.0.24-1) ...
+Preparing to unpack .../51-python-formencode_1.3.0-2_all.deb ...
-Setting up ceph-base (12.2.8-pve1) ...
+Unpacking python-formencode (1.3.0-2) ...
-Setting up libpve-guest-common-perl (2.0-18) ...
+Selecting previously unselected package python-tempita.
-Setting up python-pecan (1.1.2-3) ...
+Preparing to unpack .../52-python-tempita_0.5.2-2_all.deb ...
-update-alternatives: using /usr/bin/gunicorn_pecan-python2 to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode
+Unpacking python-tempita (0.5.2-2) ...
-update-alternatives: using /usr/bin/pecan-python2 to provide /usr/bin/pecan (pecan) in auto mode
+Selecting previously unselected package python-paste.
-Setting up ceph-osd (12.2.8-pve1) ...
+Preparing to unpack .../53-python-paste_2.0.3+dfsg-4_all.deb ...
-chown: cannot access '/var/lib/ceph/osd/*/block*': No such file or directory
+Unpacking python-paste (2.0.3+dfsg-4) ...
-Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target.
+Selecting previously unselected package python-pastedeploy-tpl.
-Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target.
+Preparing to unpack .../54-python-pastedeploy-tpl_1.5.2-4_all.deb ...
-Setting up qemu-server (5.0-36) ...
+Unpacking python-pastedeploy-tpl (1.5.2-4) ...
-Setting up pve-manager (5.2-9) ...
+Selecting previously unselected package python-pastedeploy.
-Setting up ceph-mon (12.2.8-pve1) ...
+Preparing to unpack .../55-python-pastedeploy_1.5.2-4_all.deb ...
-Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target.
+Unpacking python-pastedeploy (1.5.2-4) ...
-Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target.
+Selecting previously unselected package python-webtest.
-Processing triggers for pve-ha-manager (2.0-5) ...
+Preparing to unpack .../56-python-webtest_2.0.24-1_all.deb ...
-Setting up ceph-mgr (12.2.8-pve1) ...
+Unpacking python-webtest (2.0.24-1) ...
-Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target.
+Selecting previously unselected package python-pecan.
-Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target.
+Preparing to unpack .../57-python-pecan_1.1.2-3_all.deb ...
-Setting up ceph (12.2.8-pve1) ...
+Unpacking python-pecan (1.1.2-3) ...
-Processing triggers for libc-bin (2.24-11+deb9u3) ...
+Selecting previously unselected package python-werkzeug.
-Processing triggers for systemd (232-25+deb9u4) ...
+Preparing to unpack .../58-python-werkzeug_0.11.15+dfsg1-1_all.deb ...
-replacing ceph init script with own ceph.service
+Unpacking python-werkzeug (0.11.15+dfsg1-1) ...
-'/usr/share/doc/pve-manager/examples/ceph.service' -> '/etc/systemd/system/ceph.service'
+Selecting previously unselected package ceph-mgr.
-Synchronizing state of ceph.service with SysV service script with /lib/systemd/systemd-sysv-install.
+Preparing to unpack .../59-ceph-mgr_12.2.8-pve1_amd64.deb ...
-Executing: /lib/systemd/systemd-sysv-install enable ceph
+Unpacking ceph-mgr (12.2.8-pve1) ...
-</pre>
+Selecting previously unselected package python-colorama.
+Preparing to unpack .../60-python-colorama_0.3.7-1_all.deb ...
+Unpacking python-colorama (0.3.7-1) ...
+Selecting previously unselected package python-click.
+Preparing to unpack .../61-python-click_6.6-1_all.deb ...
+Unpacking python-click (6.6-1) ...
+Selecting previously unselected package python-itsdangerous.
+Preparing to unpack .../62-python-itsdangerous_0.24+dfsg1-2_all.deb ...
+Unpacking python-itsdangerous (0.24+dfsg1-2) ...
+Selecting previously unselected package python-flask.
+Preparing to unpack .../63-python-flask_0.12.1-1_all.deb ...
+Unpacking python-flask (0.12.1-1) ...
+Selecting previously unselected package ceph-mon.
+Preparing to unpack .../64-ceph-mon_12.2.8-pve1_amd64.deb ...
+Unpacking ceph-mon (12.2.8-pve1) ...
+Selecting previously unselected package libparted2:amd64.
+Preparing to unpack .../65-libparted2_3.2-17_amd64.deb ...
+Unpacking libparted2:amd64 (3.2-17) ...
+Selecting previously unselected package parted.
+Preparing to unpack .../66-parted_3.2-17_amd64.deb ...
+Unpacking parted (3.2-17) ...
+Selecting previously unselected package ceph-osd.
+Preparing to unpack .../67-ceph-osd_12.2.8-pve1_amd64.deb ...
+Unpacking ceph-osd (12.2.8-pve1) ...
+Selecting previously unselected package ceph.
+Preparing to unpack .../68-ceph_12.2.8-pve1_amd64.deb ...
+Unpacking ceph (12.2.8-pve1) ...
+Setting up python-dnspython (1.15.0-1) ...
+Setting up python-idna (2.2-1) ...
+Setting up libjs-jquery (3.1.1-2) ...
+Setting up python-repoze.lru (0.6-6) ...
+Setting up python-setuptools (33.1.1-1) ...
+Setting up python-prettytable (0.7.2-3) ...
+Setting up libparted2:amd64 (3.2-17) ...
+Setting up python-simplegeneric (0.8.1-1) ...
+Setting up libjs-underscore (1.8.3~dfsg-1) ...
+Setting up liblttng-ust-ctl2:amd64 (2.9.0-2+deb9u1) ...
+Setting up libpve-common-perl (5.0-40) ...
+Setting up libcurl3:amd64 (7.52.1-5+deb9u7) ...
+Setting up python-pyasn1 (0.1.9-2) ...
+Setting up libjs-sphinxdoc (1.4.9-2) ...
+Setting up python-webob (1:1.6.2-2) ...
+Setting up python-colorama (0.3.7-1) ...
+Setting up python-waitress (1.0.1-1) ...
+update-alternatives: using /usr/bin/waitress-serve-python2 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode
+Setting up parted (3.2-17) ...
+Setting up libleveldb1v5:amd64 (1.18-5) ...
+Setting up python-markupsafe (0.23-3) ...
+Setting up python-werkzeug (0.11.15+dfsg1-1) ...
+Setting up python-cffi-backend (1.9.1-2) ...
+Setting up libtcmalloc-minimal4 (2.5-2.2) ...
+Setting up libunwind8 (1.1-4.1) ...
+Processing triggers for libc-bin (2.24-11+deb9u3) ...
+Setting up python-bs4 (4.5.3-1) ...
+Setting up python-mako (1.0.6+ds1-2) ...
+Setting up python-enum34 (1.1.6-1) ...
+Processing triggers for systemd (232-25+deb9u4) ...
+Setting up cryptsetup-bin (2:1.7.3-4) ...
+Processing triggers for man-db (2.7.6.1-2) ...
+Setting up binutils (2.28-5) ...
+Setting up python-singledispatch (3.4.0.3-2) ...
+Setting up python-pastedeploy-tpl (1.5.2-4) ...
+Setting up python-itsdangerous (0.24+dfsg1-2) ...
+Setting up python-tempita (0.5.2-2) ...
+Setting up python-ipaddress (1.0.17-1) ...
+Setting up liblttng-ust0:amd64 (2.9.0-2+deb9u1) ...
+Setting up python-logutils (0.3.3-5) ...
+Setting up python-routes (2.3.1-2) ...
+Setting up python-formencode (1.3.0-2) ...
+Setting up python-jinja2 (2.8-1) ...
+Setting up python-click (6.6-1) ...
+Setting up python-paste (2.0.3+dfsg-4) ...
+Setting up librados2 (12.2.8-pve1) ...
+Setting up libcephfs2 (12.2.8-pve1) ...
+Setting up libgoogle-perftools4 (2.5-2.2) ...
+Setting up python-cryptography (1.7.1-3) ...
+Setting up python-flask (0.12.1-1) ...
+Setting up python-rados (12.2.8-pve1) ...
+Setting up python-cherrypy3 (3.5.0-2) ...
+Setting up python-cephfs (12.2.8-pve1) ...
+Setting up libradosstriper1 (12.2.8-pve1) ...
+Setting up python-openssl (16.2.0-1) ...
+Setting up librgw2 (12.2.8-pve1) ...
+Setting up python-rgw (12.2.8-pve1) ...
+Setting up python-pastedeploy (1.5.2-4) ...
+Setting up ceph-fuse (12.2.8-pve1) ...
+Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target.
+Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target.
+Setting up librbd1 (12.2.8-pve1) ...
+Setting up python-rbd (12.2.8-pve1) ...
+Setting up ceph-common (12.2.8-pve1) ...
+Installing new version of config file /etc/default/ceph ...
+Installing new version of config file /etc/logrotate.d/ceph-common ...
+Setting system user ceph properties..usermod: no changes
+..done
+Fixing /var/run/ceph ownership....done
+Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target.
+Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service.
+Setting up libpve-storage-perl (5.0-29) ...
+Setting up python-webtest (2.0.24-1) ...
+Setting up ceph-base (12.2.8-pve1) ...
+Setting up libpve-guest-common-perl (2.0-18) ...
+Setting up python-pecan (1.1.2-3) ...
+update-alternatives: using /usr/bin/gunicorn_pecan-python2 to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode
+update-alternatives: using /usr/bin/pecan-python2 to provide /usr/bin/pecan (pecan) in auto mode
+Setting up ceph-osd (12.2.8-pve1) ...
+chown: cannot access '/var/lib/ceph/osd/*/block*': No such file or directory
+Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target.
+Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target.
+Setting up qemu-server (5.0-36) ...
+Setting up pve-manager (5.2-9) ...
+Setting up ceph-mon (12.2.8-pve1) ...
+Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target.
+Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target.
+Processing triggers for pve-ha-manager (2.0-5) ...
+Setting up ceph-mgr (12.2.8-pve1) ...
+Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target.
+Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target.
+Setting up ceph (12.2.8-pve1) ...
+Processing triggers for libc-bin (2.24-11+deb9u3) ...
+Processing triggers for systemd (232-25+deb9u4) ...
+replacing ceph init script with own ceph.service
+'/usr/share/doc/pve-manager/examples/ceph.service' -> '/etc/systemd/system/ceph.service'
+Synchronizing state of ceph.service with SysV service script with /lib/systemd/systemd-sysv-install.
+Executing: /lib/systemd/systemd-sysv-install enable ceph
+</pre>
 -->
+==== Initialisieren von Ceph ====
+gemäß doku ''10.10.10.0/24'', aber das wurde (versehentlich schon für die [[#Erstellung von einem internen Netz für das Cluster von Proxmox]] verwendet)
+: <code>pveceph init --network 10.10.11.0/24</code>
+WUI
+<pre>
+rados_connect failed - No such file or directory (500)
+</pre>
+: <code>pveceph createmon</code>
+<pre>
+creating /etc/pve/priv/ceph.client.admin.keyring
+monmaptool: monmap file /tmp/monmap
+monmaptool: generated fsid 4571c8c1-89c2-44e5-8527-247470f74809
+epoch 0
+fsid 4571c8c1-89c2-44e5-8527-247470f74809
+last_changed 2018-10-02 13:30:43.174771
+created 2018-10-02 13:30:43.174771
+: 10.10.11.128:6789/0 mon.nora
+monmaptool: writing epoch 0 to /tmp/monmap (1 monitors)
+Created symlink /etc/systemd/system/ceph-mon.target.wants/ceph-mon@nora.service -> /lib/systemd/system/ceph-mon@.service.
+admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
+INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
+INFO:ceph-create-keys:Key exists already: /etc/ceph/ceph.client.admin.keyring
+INFO:ceph-create-keys:Talking to monitor...
+INFO:ceph-create-keys:Talking to monitor...
+INFO:ceph-create-keys:Talking to monitor...
+INFO:ceph-create-keys:Talking to monitor...
+creating manager directory '/var/lib/ceph/mgr/ceph-nora'
+creating keys for 'mgr.nora'
+setting owner for directory
+enabling service 'ceph-mgr@nora.service'
+Created symlink /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@nora.service -> /lib/systemd/system/ceph-mgr@.service.
+starting service 'ceph-mgr@nora.service'
+</pre>
+: <code>less /etc/pve/storage.cfg</code>
+<pre>
+dir: local
+        path /var/lib/vz
+        content iso,vztmpl,backup
+zfspool: local-zfs
+        pool rpool/data
+        sparse
+        content images,rootdir
+</pre>
+: <code>less /etc/pve/ceph.conf</code>
+<pre>
+[global]
+         auth client required = cephx
+         auth cluster required = cephx
+         auth service required = cephx
+         cluster network = 10.10.11.0/24
+         fsid = 5df2a0f7-2362-488e-9c5a-4b9ed2a16bfe
+         keyring = /etc/pve/priv/$cluster.$name.keyring
+         mon allow pool delete = true
+         osd journal size = 5120
+         osd pool default min size = 2
+         osd pool default size = 3
+         public network = 10.10.11.0/24
+[osd]
+         keyring = /var/lib/ceph/osd/ceph-$id/keyring
+[mon.nora]
+         host = nora
+         mon addr = 10.10.11.128:6789
+</pre>
+==== Hinzufügen von anderen Servern zum Pool von Ceph ====
+* nora -> Ceph -> Monitor -> Create
+*: Create Ceph Monitor/Manager
+*:; Host: cora
+: <tt>Task viewer: Ceph Monitor mon.cora - Create</tt>
+WUI
+: Output
+<pre>
+Created symlink /etc/systemd/system/ceph-mon.target.wants/ceph-mon@cora.service -> /lib/systemd/system/ceph-mon@.service.
+INFO:ceph-create-keys:ceph-mon is not in quorum: u'synchronizing'
+INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
+INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
+INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
+INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
+INFO:ceph-create-keys:Talking to monitor...
+exported keyring for client.admin
+updated caps for client.admin
+INFO:ceph-create-keys:Talking to monitor...
+INFO:ceph-create-keys:Talking to monitor...
+INFO:ceph-create-keys:Talking to monitor...
+INFO:ceph-create-keys:Talking to monitor...
+creating manager directory '/var/lib/ceph/mgr/ceph-cora'
+creating keys for 'mgr.cora'
+setting owner for directory
+enabling service 'ceph-mgr@cora.service'
+Created symlink /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@cora.service -> /lib/systemd/system/ceph-mgr@.service.
+starting service 'ceph-mgr@cora.service'
+TASK OK
+</pre>
+* nora -> Ceph -> Monitor -> Create
+*: Create Ceph Monitor/Manager
+*:; Host: dora
+*:: Create
+: <tt>Task viewer: Ceph Monitor mon.cora - Create</tt>
+WUI
+: Output
+<pre>
+Created symlink /etc/systemd/system/ceph-mon.target.wants/ceph-mon@dora.service -> /lib/systemd/system/ceph-mon@.service.
+INFO:ceph-create-keys:ceph-mon is not in quorum: u'synchronizing'
+INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
+INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
+INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
+INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
+INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
+INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
+INFO:ceph-create-keys:Talking to monitor...
+exported keyring for client.admin
+updated caps for client.admin
+INFO:ceph-create-keys:Talking to monitor...
+INFO:ceph-create-keys:Talking to monitor...
+INFO:ceph-create-keys:Talking to monitor...
+INFO:ceph-create-keys:Talking to monitor...
+creating manager directory '/var/lib/ceph/mgr/ceph-dora'
+creating keys for 'mgr.dora'
+setting owner for directory
+enabling service 'ceph-mgr@dora.service'
+Created symlink /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@dora.service -> /lib/systemd/system/ceph-mgr@.service.
+starting service 'ceph-mgr@dora.service'
+TASK OK
+</pre>
+==== Erstellung vom Pool für das Cluster von Ceph ====
+WUI
+* nora -> Ceph -> Pools -> Create
+*: Create Ceph Monitor/Manager
+*:; Name: xora
+*:; Size: 3
+*:; Min. Size: 2
+*:; Crush Rule: replicated_rule
+*:; pg_num: 64
+*:; Add Storage: X
+*:: Create
+=== Containerisierung ===
+=== Proxmox CT ===
+==== Proxmox CT Erstellung ====
+===== Proxmox CT Erstellung alle =====
+; Create CT:
+:; General:
+::; Node:
+::; Hostname: <tt>(stage)</tt> <tt>software</tt>_<tt>zweck</tt>_<tt>organisation</tt>
+::: <tt>(stage)</tt> ist beispielsweise ''test'' oder ''dev'', wenn es nicht produktiv sein soll.
+::: <tt>software</tt> ist beispielsweise ''plone'' oder ''openldap''. (Software, nicht Dienst! also nicht ''cms'' oder ''ldap'', oder ''www'' oder ''acc''!)
+::: <tt>zweck</tt> ist beispielsweise ''website-2010'' oder ''nothilfe-2020''.
+::: <tt>organisation</tt> ist beispielsweise ''stura-htw-dresden'' oder ''kss-sachsen''.
+::; Unprivileged container:
+::: [X]
+::; Nesting:
+::: [?]
+::; Password: <small>nach Lebensdauer</small>
+::: bei Projekten mindestens ''8''
+:; Template:
+::; Storage: ''cephfs''
+:::; Type: rdb
+::; Template:
+:; Root Disk:
+::; Storage: ''storage''
+::; Disk size (GiB): <small>nach Bedarf</small>
+:; CPU:
+::; Cores: <small>2, oder nach Bedarf</small>
+::: Die Verwendung von nur einen Prozessor oder mehr als zwei Prozessor ist zu begründen.
+:; Memory:
+::: <small>nach Bedarf</small>
+::; Memory (MiB): <small>nach Bedarf</small>
+::; Swap (MiB): <small>nach Bedarf</small>
+::: Die Größe soll der Größe, die bei ''Memory (MiB)'' verwendet wird, entsprechen. (Die Verwendung von mehr oder weniger Größe ist zu begründen.)
+::: Die Größe soll höchstens die Hälfte der Größe, die bei (''Root Disk'' ->) ''Disk size (GiB)'' verwendet wird, betragen.
+:; Network:
+::; Bridge: ''vmbr1''
+::; IPv4: Static
+::; IPv4/CIDR: 141.56.51.<tt>321</tt>/24
+::; Gateway (IPv4): 141.56.51.254
+::: <tt>321</tt> ist die "verwendbare" Adresse für IPv4, die unverzüglich bei [[Intern:Server#Verwendung von IP-Adressen]] einzutragen ist.
+:; DNS:
+::; DNS domain:
+::; DNS servers: 141.56.1.1
+:; Confirm:
+::; Start after created: [ ]
+::; Finish:
+; Datacenter (cluster):
+:; HA:
+::; Resources:
+:::; Add:
+::::; VM: <tt>110</tt>
+::::: <tt>110</tt> ist die ID der Instanz innerhalb von Proxmox.
+::::; Group: ''HA_cluster''
+::::; Add:
+===== Proxmox CT Erstellung TurnKey =====
+; Create CT:
+:; General:
+::; Unprivileged conatainer: [ ]
+: <code>turnkey-init</code>
+==== Proxmox CT Verwaltung ====
+Auflisten aller Dateien für die Konfiguration der Container auf dem jeweiligen node
+: <code>ls /etc/pve/lxc/</code>
+=== Virtualisierung ===
+=== Proxmox VM ===
+=== Ausfallsicherheit ===
+==== Ausfallsicherheit CT ====
+: <code>ping 141.56.51.321</code>
+<pre>
+PING 141.56.51.321 (141.56.51.321) 56(84) bytes of data.
+bytes from 141.56.51.321: icmp_seq=1 ttl=64 time=0.283 ms
+bytes from 141.56.51.321: icmp_seq=2 ttl=64 time=0.213 ms
+bytes from 141.56.51.321: icmp_seq=3 ttl=64 time=0.286 ms
+From 141.56.51.456 icmp_seq=4 Destination Host Unreachable
+From 141.56.51.456 icmp_seq=5 Destination Host Unreachable
+</pre>
+<pre></pre>
+<pre>
+From 141.56.51.456 icmp_seq=78 Destination Host Unreachable
+From 141.56.51.456 icmp_seq=79 Destination Host Unreachable
+bytes from 141.56.51.321: icmp_seq=80 ttl=64 time=2609 ms
+bytes from 141.56.51.321: icmp_seq=81 ttl=64 time=1585 ms
+bytes from 141.56.51.321: icmp_seq=82 ttl=64 time=561 ms
+bytes from 141.56.51.321: icmp_seq=83 ttl=64 time=0.260 ms
+bytes from 141.56.51.321: icmp_seq=84 ttl=64 time=0.295 ms
+bytes from 141.56.51.157: icmp_seq=85 ttl=64 time=0.200 ms
+bytes from 141.56.51.157: icmp_seq=86 ttl=64 time=0.274 ms
+</pre>
+<pre></pre>
+==== Ausfallsicherheit Fehler ====
+===== Fehler bei der Ausfallsicherheit bei der Verwendung von ZFS, statt RDB =====
+<pre>
+-04-29 09:14:46 starting migration of CT 110 to node 'n1' (10.1.0.31)
+-04-29 09:14:46 found local volume 'local-zfs:subvol-110-disk-0' (in current VM config)
+cannot open 'rpool/data/subvol-110-disk-0': dataset does not exist
+usage:
+	snapshot [-r] [-o property=value] ... <filesystem|volume>@<snap> ...
+For the property list, run: zfs set|get
+-04-29 09:14:46 ERROR: zfs error: For the delegated permission list, run: zfs allow|unallow
+-04-29 09:14:46 aborting phase 1 - cleanup resources
+-04-29 09:14:46 ERROR: found stale volume copy 'local-zfs:subvol-110-disk-0' on node 'n1'
+-04-29 09:14:46 start final cleanup
+-04-29 09:14:46 ERROR: migration aborted (duration 00:00:01): zfs error: For the delegated permission list, run: zfs allow|unallow
+TASK ERROR: migration aborted
+</pre>
+== Anpassungen für Produktivebetrieb ==
+* Backup Limit auf 10 erhöht.
+* Vorbereitung für Plone 5 Umzug (101) [[Plone 5]]
 == Siehe auch ==

Intern:Cluster/xora: Unterschied zwischen den Versionen

Aktuelle Version vom 17. Oktober 2022, 15:06 Uhr

Test

Bezeichnung

Verbund

Geräte

einzelne Geräte

27090

27091

8529

8

5100

BIOS

Proxmox VE

Installation Betriebssystem

Vorbereitung Installation Betreibssystem

Durchführung Installation Betreibssystem

Nachbereitung Installation Betreibssystem

Anpassung der Quellen

Ergänzung der Quelle pve-no-subscription

Entfernung der Quelle pve-enterprise

Erstellung vom Cluster

optionales Begutachten für die Erstellung von einem Cluster

Erzeugung vom Cluster xora

Erstellung von einem internen Netz für das Cluster von Proxmox

Eintragung der anderen Nodes unabhängig von DNS

Hinzufügen von anderen Servern

Prüfen vom Verbund

Erstellen vom Cluster für Ceph

Erstellung von einem internen Netz für das Cluster von Ceph

Installation von Ceph

Initialisieren von Ceph

Hinzufügen von anderen Servern zum Pool von Ceph

Erstellung vom Pool für das Cluster von Ceph

Containerisierung

Proxmox CT

Proxmox CT Erstellung

Proxmox CT Erstellung alle

Proxmox CT Erstellung TurnKey

Proxmox CT Verwaltung

Virtualisierung

Proxmox VM

Ausfallsicherheit

Ausfallsicherheit CT

Ausfallsicherheit Fehler

Fehler bei der Ausfallsicherheit bei der Verwendung von ZFS, statt RDB

Anpassungen für Produktivebetrieb

Siehe auch

Navigationsmenü

Suche