"CEPH Performans Optimizasyonu" sayfasının sürümleri arasındaki fark

BulutWiki sitesinden
Şuraya atla: kullan, ara
(OSD lern CPU core map Edilmesi)
13. satır: 13. satır:
  
 
==OSD lern CPU core map Edilmesi==
 
==OSD lern CPU core map Edilmesi==
root@zula201:~# numactl --hardware
+
root@zula201:~# numactl --hardware
available: 2 nodes (0-1)
+
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 20 21 22 23 24 25 26 27 28 29
+
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 20 21 22 23 24 25 26 27 28 29
node 0 size: 64320 MB
+
node 0 size: 64320 MB
node 0 free: 63198 MB
+
node 0 free: 63198 MB
node 1 cpus: 10 11 12 13 14 15 16 17 18 19 30 31 32 33 34 35 36 37 38 39
+
node 1 cpus: 10 11 12 13 14 15 16 17 18 19 30 31 32 33 34 35 36 37 38 39
node 1 size: 64508 MB
+
node 1 size: 64508 MB
node 1 free: 62399 MB
+
node 1 free: 62399 MB
node distances:
+
node distances:
node  0  1  
+
node  0  1  
 
   0:  10  21  
 
   0:  10  21  
 
   1:  21  10  
 
   1:  21  10  
 +
 +
86:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
 +
        Subsystem: Intel Corporation 10GbE 2P X520 Adapter
 +
        Physical Slot: 4
 +
        Flags: bus master, fast devsel, latency 0, IRQ 208
 +
        Memory at fb980000 (64-bit, non-prefetchable) [size=512K]
 +
        I/O ports at e020 [size=32]
 +
        Memory at fba04000 (64-bit, non-prefetchable) [size=16K]
 +
     
 +
86:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
 +
        Subsystem: Intel Corporation 10GbE 2P X520 Adapter
 +
        Physical Slot: 4
 +
        Flags: bus master, fast devsel, latency 0, IRQ 250
 +
        Memory at fb880000 (64-bit, non-prefetchable) [size=512K]
 +
     
 +
 +
83:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
 +
        Subsystem: Mellanox Technologies MT27700 Family [ConnectX-4]
 +
        Physical Slot: 1-1
 +
        Flags: bus master, fast devsel, latency 0, IRQ 40
 +
        Memory at ca000000 (64-bit, prefetchable) [size=32M]
 +
        Expansion ROM at fbc00000 [disabled] [size=1M]
 +
 +
83:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
 +
        Subsystem: Mellanox Technologies MT27700 Family [ConnectX-4]
 +
        Physical Slot: 1-1
 +
        Flags: bus master, fast devsel, latency 0, IRQ 295
 +
        Memory at cc000000 (64-bit, prefetchable) [size=32M]
 +
        Expansion ROM at fbb00000 [disabled] [size=1M]
 +
 +
82:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05)
 +
        Subsystem: LSI Logic / Symbios Logic MegaRAID SAS 9260-8i
 +
        Physical Slot: 1
 +
        Flags: bus master, fast devsel, latency 0, IRQ 38
 +
        I/O ports at f000 [size=256]
 +
        Memory at fbe60000 (64-bit, non-prefetchable) [size=16K]
 +
        Memory at fbe00000 (64-bit, non-prefetchable) [size=256K]
 +
 +
03:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
 +
        Subsystem: Super Micro Computer Inc SAS3008 PCI-Express Fusion-MPT SAS-3
 +
        Physical Slot: 3
 +
        Flags: bus master, fast devsel, latency 0, IRQ 52
 +
        I/O ports at 2000 [size=256]
 +
        Memory at c7240000 (64-bit, non-prefetchable) [size=64K]
 +
        Memory at c7200000 (64-bit, non-prefetchable) [size=256K]
 +
        Expansion ROM at c7100000 [disabled] [size=1M]
 +
 +
 +
root@zula201:~# lspci -s 03:00.0
 +
03:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
 +
 +
 +
SAS->root@zula201:/sys/devices/pci0000:00/0000:00:03.0# cat numa_node
 +
0
  
  

07:47, 9 Ağustos 2018 tarihindeki hâli

Sürücülerin Güncellemesi

TCMalloc Değerinin Değiştirilmesi

ceph.conf

Increasing the TCMalloc thread cache to 128M can improve performance 4-5x. It would be great to have a way of setting the TCMalloc thread cache to 128M, instead of the default 32M. 
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728
ceph-deploy --overwrite-conf admin zula209 zula210 zula211 zula212 
ceph tell osd.* heap stats
ceph tell osd.* heap release

OSD lern CPU core map Edilmesi

root@zula201:~# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 20 21 22 23 24 25 26 27 28 29
node 0 size: 64320 MB
node 0 free: 63198 MB
node 1 cpus: 10 11 12 13 14 15 16 17 18 19 30 31 32 33 34 35 36 37 38 39
node 1 size: 64508 MB
node 1 free: 62399 MB
node distances:
node   0   1 
 0:  10  21 
 1:  21  10 
86:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
       Subsystem: Intel Corporation 10GbE 2P X520 Adapter
       Physical Slot: 4
       Flags: bus master, fast devsel, latency 0, IRQ 208
       Memory at fb980000 (64-bit, non-prefetchable) [size=512K]
       I/O ports at e020 [size=32]
       Memory at fba04000 (64-bit, non-prefetchable) [size=16K]
      
86:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
       Subsystem: Intel Corporation 10GbE 2P X520 Adapter
       Physical Slot: 4
       Flags: bus master, fast devsel, latency 0, IRQ 250
       Memory at fb880000 (64-bit, non-prefetchable) [size=512K]
      
83:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
       Subsystem: Mellanox Technologies MT27700 Family [ConnectX-4]
       Physical Slot: 1-1
       Flags: bus master, fast devsel, latency 0, IRQ 40
       Memory at ca000000 (64-bit, prefetchable) [size=32M]
       Expansion ROM at fbc00000 [disabled] [size=1M]
83:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
       Subsystem: Mellanox Technologies MT27700 Family [ConnectX-4]
       Physical Slot: 1-1
       Flags: bus master, fast devsel, latency 0, IRQ 295
       Memory at cc000000 (64-bit, prefetchable) [size=32M]
       Expansion ROM at fbb00000 [disabled] [size=1M]
82:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05)
       Subsystem: LSI Logic / Symbios Logic MegaRAID SAS 9260-8i
       Physical Slot: 1
       Flags: bus master, fast devsel, latency 0, IRQ 38
       I/O ports at f000 [size=256]
       Memory at fbe60000 (64-bit, non-prefetchable) [size=16K]
       Memory at fbe00000 (64-bit, non-prefetchable) [size=256K]
03:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
       Subsystem: Super Micro Computer Inc SAS3008 PCI-Express Fusion-MPT SAS-3
       Physical Slot: 3
       Flags: bus master, fast devsel, latency 0, IRQ 52
       I/O ports at 2000 [size=256]
       Memory at c7240000 (64-bit, non-prefetchable) [size=64K]
       Memory at c7200000 (64-bit, non-prefetchable) [size=256K]
       Expansion ROM at c7100000 [disabled] [size=1M]


root@zula201:~# lspci -s 03:00.0
03:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)


SAS->root@zula201:/sys/devices/pci0000:00/0000:00:03.0# cat numa_node 0


Ref: Designing for High Performance Ceph at Scale https://www.youtube.com/watch?v=P6SCdZXpM2Q https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/pdf/configuration_guide/Red_Hat_Ceph_Storage-3-Configuration_Guide-en-US.pdf https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/pdf/installation_guide_for_red_hat_enterprise_linux/Red_Hat_Ceph_Storage-3-Installation_Guide_for_Red_Hat_Enterprise_Linux-en-US.pdf