CEPH Performans Optimizasyonu

Miniumum Rezerve Bellek Değerinin Değiştirilmesi

/etc/sysctl.conf altına OSD'lerin anlık bellek ihtiyacının karşılanması için yeterli bellek bırakılması için gerekli yapılandırma. 128G bellek için değer:

vm.min_free_kbytes = 2097152

https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/ceph_object_gateway_for_production/deploying_a_cluster

TCMalloc Değerinin Değiştirilmesi

ceph.conf

Increasing the TCMalloc thread cache to 128M can improve performance 4-5x. It would be great to have a way of setting the TCMalloc thread cache to 128M, instead of the default 32M. 

TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728
ceph-deploy --overwrite-conf admin zula209 zula210 zula211 zula212 
ceph tell osd.* heap stats
ceph tell osd.* heap release

OSD lern CPU core map Edilmesi

root@zula201:~# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 20 21 22 23 24 25 26 27 28 29
node 0 size: 64320 MB
node 0 free: 63198 MB
node 1 cpus: 10 11 12 13 14 15 16 17 18 19 30 31 32 33 34 35 36 37 38 39
node 1 size: 64508 MB
node 1 free: 62399 MB
node distances:
node   0   1 
 0:  10  21 
 1:  21  10 

Nvme SSD

84:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express])
       Subsystem: Intel Corporation DC P3700 SSD
       Physical Slot: 2
       Flags: bus master, fast devsel, latency 0, IRQ 47
       Memory at fbd10000 (64-bit, non-prefetchable) [size=16K]
       Expansion ROM at fbd00000 [disabled] [size=64K]

root@zula201:/sys/devices/pci0000:80/0000:80:02.2/0000:84:00.0# cat numa_node 
1

10G Ethernet

86:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
       Subsystem: Intel Corporation 10GbE 2P X520 Adapter
       Physical Slot: 4
       Flags: bus master, fast devsel, latency 0, IRQ 208
       Memory at fb980000 (64-bit, non-prefetchable) [size=512K]
       I/O ports at e020 [size=32]
       Memory at fba04000 (64-bit, non-prefetchable) [size=16K]
      
root@zula201:/sys/devices/pci0000:80/0000:80:03.2/0000:86:00.0# cat numa_node 
1

86:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
       Subsystem: Intel Corporation 10GbE 2P X520 Adapter
       Physical Slot: 4
       Flags: bus master, fast devsel, latency 0, IRQ 250
       Memory at fb880000 (64-bit, non-prefetchable) [size=512K]

root@zula201:/sys/devices/pci0000:80/0000:80:03.2/0000:86:00.1# cat numa_node 
1

40G Ethernet

83:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
       Subsystem: Mellanox Technologies MT27700 Family [ConnectX-4]
       Physical Slot: 1-1
       Flags: bus master, fast devsel, latency 0, IRQ 40
       Memory at ca000000 (64-bit, prefetchable) [size=32M]
       Expansion ROM at fbc00000 [disabled] [size=1M]

root@zula201:/sys/devices/pci0000:80/0000:80:02.0/0000:83:00.0# cat numa_node 
1

83:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
       Subsystem: Mellanox Technologies MT27700 Family [ConnectX-4]
       Physical Slot: 1-1
       Flags: bus master, fast devsel, latency 0, IRQ 295
       Memory at cc000000 (64-bit, prefetchable) [size=32M]
       Expansion ROM at fbb00000 [disabled] [size=1M]
root@zula201:/sys/devices/pci0000:80/0000:80:02.0/0000:83:00.1# cat numa_node 
1

Root Disk Controller

82:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05)
       Subsystem: LSI Logic / Symbios Logic MegaRAID SAS 9260-8i
       Physical Slot: 1
       Flags: bus master, fast devsel, latency 0, IRQ 38
       I/O ports at f000 [size=256]
       Memory at fbe60000 (64-bit, non-prefetchable) [size=16K]
       Memory at fbe00000 (64-bit, non-prefetchable) [size=256K]
root@zula201:/sys/devices/pci0000:80/0000:80:02.0# cat numa_node 
1

Veri Disk Denetleyicisi

03:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
       Subsystem: Super Micro Computer Inc SAS3008 PCI-Express Fusion-MPT SAS-3
       Physical Slot: 3
       Flags: bus master, fast devsel, latency 0, IRQ 52
       I/O ports at 2000 [size=256]
       Memory at c7240000 (64-bit, non-prefetchable) [size=64K]
       Memory at c7200000 (64-bit, non-prefetchable) [size=256K]
       Expansion ROM at c7100000 [disabled] [size=1M]

root@zula201:/sys/devices/pci0000:00/0000:00:03.0# cat numa_node 
0

root@zula201:~# lspci -s 03:00.0
03:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)

Ref:

• Designing for High Performance Ceph at Scale https://www.youtube.com/watch?v=P6SCdZXpM2Q
• https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/pdf/configuration_guide/Red_Hat_Ceph_Storage-3-Configuration_Guide-en-US.pdf
• https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/pdf/installation_guide_for_red_hat_enterprise_linux/Red_Hat_Ceph_Storage-3-Installation_Guide_for_Red_Hat_Enterprise_Linux-en-US.pdf

• https://events.static.linuxfound.org/sites/events/files/eeus13_shelton.pdf
• http://www.ksingh.co.in/blog/2016/09/08/working-with-numa-cpu-pinning/