Notes

I have an example of per core binding here: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/Tuning a full-system PowerKVM guest (https://www.ibm.com/developerworks/community/wikis/home?lang=en#%21/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/Tuning a full-system PowerKVM guest)
BUT, that is only ok for some CPU intensive workloads (for instance speccpu). If the workload does I/O or lots of hcalls, binding per core can actually hurt on P8 as described in the note in that article (this should be better on P9). For nearly all cases I would recommend binding at the numa node level instead of per core.

is this a PowerKVM 3.1 host? there is an even easier way to bind memory on the newer releases..

5:43:13 PM: Here is an example from a 20core firestone using the newly supported numatune section, just adjust the cpus and mem as needed based on the 'numactl -H' layout in the host:

*** first pin each guest vcpu to a set of host cpus for each numa node:

<memory unit='KiB'>498073600</memory>
<vcpu placement='static'>160</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='0,8,16,24,32,40,48,56,64,72'/>
<vcpupin vcpu='1' cpuset='0,8,16,24,32,40,48,56,64,72'/>
<vcpupin vcpu='2' cpuset='0,8,16,24,32,40,48,56,64,72'/>
{...}
<vcpupin vcpu='79' cpuset='0,8,16,24,32,40,48,56,64,72'/>
<vcpupin vcpu='80' cpuset='80,88,96,104,112,120,128,136,144,152'/>
{...}
<vcpupin vcpu='158' cpuset='80,88,96,104,112,120,128,136,144,152'/>
<vcpupin vcpu='159' cpuset='80,88,96,104,112,120,128,136,144,152'/>
</cputune>

*** and create numa nodes ("cells") in the guest:

<cpu mode='host-passthrough'>
<topology sockets='2' cores='10' threads='8'/>
<numa>
<cell id='0' cpus='0-79' memory='249036800' unit='KiB'/>
<cell id='1' cpus='80-159' memory='249036800' unit='KiB'/>
</numa>
</cpu>

*** In the <numa> section, the 'cpus' are the guest vcpus you want in each numa node in the guest, evenly divide the total guest mem between the numa nodes, then pin those guest cells to each host numa node as needed, ex:

<numatune>
<memnode cellid='0' mode='strict' nodeset='0'/>
<memnode cellid='1' mode='strict' nodeset='8'/>
</numatune>

*** here 'cellid' is the guest cell id you created above, and 'nodeset' is the host numa node you want to pin it to


that is a full system guest example, but obviously you can a small guest to one numa node using the same method

: Cool. The issue here is they want 1:1 correlation with cores and vcpu. So I guess I tie the cpus like this?
<vcpupin vcpu='0' cpuset='0'/>
<vcpupin vcpu='1' cpuset='8'/>


5:48:54 PM: if they want 1:1, that would be correct for an smt off guest. If they use threads=8 to enable smt8 guest, they would need to pin vcpu 0-7 to host cpu 0, vcpu 8-15 to host cpu 8, and so on - basically group the smt threads since the host only has one thread online per core. Just be aware the per cpu pinning can hurt performance!


5:49:26 PM: They first want SMT off and then later they may experiment other SMTs.

5:49:36 PM: I was thinking the same about performance.


5:49:36 PM: ah that makes it easier to try ;)

5:49:39 PM: I had same issue.

5:49:52 PM: yep, especially for I/O workloads it can hurt

5:50:10 PM: They have both. But primarily CPU intensive.



5:50:45 PM: that will get fixed on P9 since we will behave like x86 where all host threads are online and can each run a guest, but right now the P8 per core MMU design makes us have issues w/ grouping the threads and running 1 thread online in the host..

5:51:31 PM: Oh ok.

3:41:08 PM: ok sure

5:34:52 PM: Sure

5:39:20 PM: notes.io/eeCj



5:48:14 PM: 513802240/4
128450560

5:53:16 PM: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Tuning_and_Optimization_Guide/sect-Virtualization_Tuning_Optimization_Guide-Networking-Techniques.html#sect-Virtualization_Tuning_Optimization_Guide-Networking-Multi-queue_virtio-net

5:54:35 PM: <vcpu placement='static' cpuset='0,8,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152'>20</vcpu>


5:54:56 PM: cpuset='0,8,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152'

5:55:39 PM:


5:58:14 PM: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/Basic XML tuning tips for PowerKVM guests (https://www.ibm.com/developerworks/community/wikis/home?lang=en#%21/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/Basic XML tuning tips for PowerKVM guests)

cache=writethrough uses the host page cache, but disables the guest disk write cache. This mode uses O_DSYNC so writes are reported as complete only when the data is actually committed to storage, which can have an impact on write performance. However, this mode may improve read performance because it uses the host page cache.
cache=none enables the guest disk write cache, but does not use the host page cache. This mode uses O_DIRECT so the guest does I/O directly to the storage device. Due to this behavior, write performance is improved while read performance may suffer a bit compared to other modes that use the host page cache.
cache=writeback uses the host page cache and enables the guest disk write cache. Writes are reported as complete once they are in the host page cache, so I/O performance is generally good. There is, however, some risk of losing data if a power failure happens. Therefore, this mode may not be appropriate for production environments.


5:59:06 PM: cache=writeback

5:59:18 PM: read perf = writethrough

6:00:01 PM: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Tuning_and_Optimization_Guide/sect-Virtualization_Tuning_Optimization_Guide-Networking-Techniques.html#sect-Virtualization_Tuning_Optimization_Guide-Networking-Multi-queue_virtio-net

6:00:13 PM: <driver name='vhost' queues='N'/>

6:01:16 PM: <interface type='bridge'>
<mac address='52:54:00:1b:82:ea'/>
<source bridge='brenP3p5s0f0'/>
<target dev='vnet0'/>
<model type='virtio'/>
<driver name='vhost' queues='20'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</interface>

6:01:28 PM: ethtool -L eth0 combined 20

6:03:27 PM: In the XML, add the hugepage backing:

<memory unit='KiB'>14745600</memory>
<memoryBacking>
<hugepages/>
</memoryBacking>

Before starting the guest, reserve enough 16MB hugepages in the host to back the full guest memory. For this example, the number of pages is calculated as: 14745600 / 16384 = 900

# echo 900 > /proc/sys/vm/nr_hugepages

6:04:48 PM: Next step is to reserve the hugepages in the host, it is always best to clear any existing pages before reserving new ones. For the tuned config, we want to make sure the allocation is perfectly balanced, so we reserve the pages per node and check that they were all reserved:

# echo 0 > /proc/sys/vm/nr_hugepages
# echo 13926 >/sys/devices/system/node/node0/hugepages/hugepages-16384kB/nr_hugepages
# echo 13926 >/sys/devices/system/node/node1/hugepages/hugepages-16384kB/nr_hugepages
# echo 13926 >/sys/devices/system/node/node16/hugepages/hugepages-16384kB/nr_hugepages
# echo 13926 >/sys/devices/system/node/node17/hugepages/hugepages-16384kB/nr_hugepages
# cat /sys/devices/system/node/node*/hugepages/hugepages-16384kB/nr_hugepages
13926
13926
13926
13926


6:05:21 PM: <cell id='0' cpus='0-4' memory='128450560' unit='KiB'/>

6:05:44 PM: 7840

6:06:13 PM: 128450560 / 16384

6:07:48 PM: tuned-adm list

6:11:37 PM: /usr/lib/tuned/

6:12:01 PM: # cat /usr/lib/tuned/network-throughput/tuned.conf
#
# tuned configuration
#

[main]
include=throughput-performance

[sysctl]
# Increase kernel buffer size maximums. Currently this seems only necessary at 40Gb speeds.
#
# The buffer tuning values below do not account for any potential hugepage allocation.
# Ensure that you do not oversubscribe system memory.
net.ipv4.tcp_rmem="4096 87380 16777216"
net.ipv4.tcp_wmem="4096 16384 16777216"
net.ipv4.udp_mem="3145728 4194304 16777216"

http://www.ibm.com/support/knowledgecenter/8247-22L/p8ebk/updating_microcode.htm?cp=8247-22L