"CEPH Performans Optimizasyonu" sayfasının sürümleri arasındaki fark
115. satır: | 115. satır: | ||
Ref: | Ref: | ||
− | Designing for High Performance Ceph at Scale https://www.youtube.com/watch?v=P6SCdZXpM2Q | + | *Designing for High Performance Ceph at Scale https://www.youtube.com/watch?v=P6SCdZXpM2Q |
− | https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/pdf/configuration_guide/Red_Hat_Ceph_Storage-3-Configuration_Guide-en-US.pdf | + | *https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/pdf/configuration_guide/Red_Hat_Ceph_Storage-3-Configuration_Guide-en-US.pdf |
− | https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/pdf/installation_guide_for_red_hat_enterprise_linux/Red_Hat_Ceph_Storage-3-Installation_Guide_for_Red_Hat_Enterprise_Linux-en-US.pdf | + | *https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/pdf/installation_guide_for_red_hat_enterprise_linux/Red_Hat_Ceph_Storage-3-Installation_Guide_for_Red_Hat_Enterprise_Linux-en-US.pdf |
− | https://events.static.linuxfound.org/sites/events/files/eeus13_shelton.pdf | + | *https://events.static.linuxfound.org/sites/events/files/eeus13_shelton.pdf |
+ | *http://www.ksingh.co.in/blog/2016/09/08/working-with-numa-cpu-pinning/ |
09:01, 9 Ağustos 2018 itibarı ile sayfanın şu anki hâli
İçindekiler
Sürücülerin Güncellemesi
Anahtarlama Cihazı Jumbo frame açılması ve flow control kapatılması
Miniumum Rezerve Bellek Değerinin Değiştirilmesi
/etc/sysctl.conf altına OSD'lerin anlık bellek ihtiyacının karşılanması için yeterli bellek bırakılması için gerekli yapılandırma. 128G bellek için değer:
vm.min_free_kbytes = 2097152
TCMalloc Değerinin Değiştirilmesi
ceph.conf
Increasing the TCMalloc thread cache to 128M can improve performance 4-5x. It would be great to have a way of setting the TCMalloc thread cache to 128M, instead of the default 32M.
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728 ceph-deploy --overwrite-conf admin zula209 zula210 zula211 zula212 ceph tell osd.* heap stats ceph tell osd.* heap release
OSD lern CPU core map Edilmesi
root@zula201:~# numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 20 21 22 23 24 25 26 27 28 29 node 0 size: 64320 MB node 0 free: 63198 MB node 1 cpus: 10 11 12 13 14 15 16 17 18 19 30 31 32 33 34 35 36 37 38 39 node 1 size: 64508 MB node 1 free: 62399 MB node distances: node 0 1 0: 10 21 1: 21 10
Nvme SSD
84:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01) (prog-if 02 [NVM Express]) Subsystem: Intel Corporation DC P3700 SSD Physical Slot: 2 Flags: bus master, fast devsel, latency 0, IRQ 47 Memory at fbd10000 (64-bit, non-prefetchable) [size=16K] Expansion ROM at fbd00000 [disabled] [size=64K]
root@zula201:/sys/devices/pci0000:80/0000:80:02.2/0000:84:00.0# cat numa_node 1
10G Ethernet
86:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01) Subsystem: Intel Corporation 10GbE 2P X520 Adapter Physical Slot: 4 Flags: bus master, fast devsel, latency 0, IRQ 208 Memory at fb980000 (64-bit, non-prefetchable) [size=512K] I/O ports at e020 [size=32] Memory at fba04000 (64-bit, non-prefetchable) [size=16K] root@zula201:/sys/devices/pci0000:80/0000:80:03.2/0000:86:00.0# cat numa_node 1
86:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01) Subsystem: Intel Corporation 10GbE 2P X520 Adapter Physical Slot: 4 Flags: bus master, fast devsel, latency 0, IRQ 250 Memory at fb880000 (64-bit, non-prefetchable) [size=512K]
root@zula201:/sys/devices/pci0000:80/0000:80:03.2/0000:86:00.1# cat numa_node 1
40G Ethernet
83:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4] Subsystem: Mellanox Technologies MT27700 Family [ConnectX-4] Physical Slot: 1-1 Flags: bus master, fast devsel, latency 0, IRQ 40 Memory at ca000000 (64-bit, prefetchable) [size=32M] Expansion ROM at fbc00000 [disabled] [size=1M]
root@zula201:/sys/devices/pci0000:80/0000:80:02.0/0000:83:00.0# cat numa_node 1
83:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4] Subsystem: Mellanox Technologies MT27700 Family [ConnectX-4] Physical Slot: 1-1 Flags: bus master, fast devsel, latency 0, IRQ 295 Memory at cc000000 (64-bit, prefetchable) [size=32M] Expansion ROM at fbb00000 [disabled] [size=1M] root@zula201:/sys/devices/pci0000:80/0000:80:02.0/0000:83:00.1# cat numa_node 1
Root Disk Controller
82:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05) Subsystem: LSI Logic / Symbios Logic MegaRAID SAS 9260-8i Physical Slot: 1 Flags: bus master, fast devsel, latency 0, IRQ 38 I/O ports at f000 [size=256] Memory at fbe60000 (64-bit, non-prefetchable) [size=16K] Memory at fbe00000 (64-bit, non-prefetchable) [size=256K] root@zula201:/sys/devices/pci0000:80/0000:80:02.0# cat numa_node 1
Veri Disk Denetleyicisi
03:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02) Subsystem: Super Micro Computer Inc SAS3008 PCI-Express Fusion-MPT SAS-3 Physical Slot: 3 Flags: bus master, fast devsel, latency 0, IRQ 52 I/O ports at 2000 [size=256] Memory at c7240000 (64-bit, non-prefetchable) [size=64K] Memory at c7200000 (64-bit, non-prefetchable) [size=256K] Expansion ROM at c7100000 [disabled] [size=1M]
root@zula201:/sys/devices/pci0000:00/0000:00:03.0# cat numa_node 0
root@zula201:~# lspci -s 03:00.0 03:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
Ref:
- Designing for High Performance Ceph at Scale https://www.youtube.com/watch?v=P6SCdZXpM2Q
- https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/pdf/configuration_guide/Red_Hat_Ceph_Storage-3-Configuration_Guide-en-US.pdf
- https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/pdf/installation_guide_for_red_hat_enterprise_linux/Red_Hat_Ceph_Storage-3-Installation_Guide_for_Red_Hat_Enterprise_Linux-en-US.pdf