AWS Nitro instances need special tuning for NVMe devices:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html#timeout-nvme-ebs-volumes
[sysfs]
/sys/module/nvme_core/parameters/io_timeout=4294967295
/sys/module/nvme_core/parameters/max_retries=10
This tuning should probably be moved to Cloud Provider-specific profiles
once the functionality is implemented.
Signed-off-by: Jiri Mencak <jmencak@users.noreply.github.com>
We already disable the clocksource watchdog, it's there to double-check
whether clock results are coherent between reads. But a new reliability
test was introduced in recent kernels[1], tracking tsc drift between
CPUs[2]. It works by programming timers on all CPUs, including isolated
ones, and checks whether the tsc was adjusted in between runs. This
introduces unwarranted latency on real time systems. So let's promote
tsc to 'reliable', which disables both checks.
There shouldn't be any impact on old nor new setups. First, this test
didn't exist in the past, so any eventual drift was left uncorrected.
Second, most telco and HPC use-cases will make heavy use of tsc in
user-space, and its reliability is already assumed in the system's
design.
[1] c7719e793478 x86/tsc: Add a timer to make sure TSC_adjust is always
checked
[2] Generally caused by rogue BIOSes adjusting the tsc on a CPU and not
doing so on the rest
Signed-off-by: Nicolas Saenz Julienne <nsaenzju@redhat.com>
The [scheduler] plug-in can be very CPU intensive, especially on the
OpenShift platform. The bug for this issue is tracked by rhbz#1921738.
Until this is fixed, work around this problem in the openshift
parent profile by adding "runtime=0" [scheduler] plug-in option.
Signed-off-by: Jiri Mencak <jmencak@users.noreply.github.com>
According to the performance team measurements the sched_ tuning
should be dropped in several profiles.
Resolves: rhbz#1957829
Signed-off-by: Jaroslav Škarvada <jskarvad@redhat.com>
New kernels (5.13 and newer) moved some sched_ and numa_ knobs from
the sysctl to the debugfs, thus add and abstract these knobs under the
scheduler plugin. With help of this abstraction it will write
the tuning to the correct place according to the kernel used.
Example:
[scheduler]
sched_migration_cost_ns = 500000
Will work on the old kernel the same way as:
[sysctl]
kernel.sched_migration_cost_ns = 500000
I.e. it will write '500000' to the:
/proc/sys/kernel/sched_migration_cost_ns
And on the new kernel it will write '500000' to the:
/sys/kernel/debug/sched/migration_cost_ns
Also updated TuneD profiles.
Resolves: rhbz#1952687
Signed-off-by: Jaroslav Škarvada <jskarvad@redhat.com>
The 'nx_huge_pages' option, will force any executable page mapping to be
performed at 4KiB granularity and requires any existing overlapping
huge-page mapping to be also split in 4KiB chunks[1]. This is due to a HW
bug that doesn't affect our NFV use-cases.
The way a huge-page mapping is dropped is by simply deleting the EPT
entry and flushing TLB caches on all affected CPUs. Next vCPU access to
that address will trigger an EPT_VIOLATION, which will jump into the
host for it to handle the page-fault. In our specific case, this means
all isolated CPUs running oslat will hit an EPT_VIOLATION almost exactly
at the same time. Which is bad enough already, but, given our systems
might have a huge number of isolated CPUs, will also create a lot of
contention over the KVM MMU lock. This has been observed to trigger
~100us latency spikes while testing with oslat.
So let's disable the 'kvm.nx_huge_pages' module option.
[1] see kernel commit b7e8c8303ff28
Resolves: rhbz#1976825
Signed-off-by: Nicolas Saenz Julienne <nsaenzju@redhat.com>
I/O APIC, one of intel's interrupt controllers, can only change the
affinity on an IRQ line during an EOI (end of interrupt)[1]. In other
words, on boot, although tuned will set a preferred IRQ affinity, it
might not be applied until the next time the interrupt is triggered.
This might be right away, or hours later while oslat or a
latency-sensitive workload is running.
To mitigate this set the 'irqaffinity='kernel command-line[2] option to
match the non-isolated CPU mask when running the realtime-virtual-*
profiles. This way we'll get the right affinity regardless of the
situation stated above.
[1] For more reference see kernel's ioapic_ack_level() function and the
IRQD_SETAFFINITY_PENDING flag.
[2] Actually, this is why irqaffinity was introduced in the first place,
see kernel commit fbf198030e0b0.
Resolves: rhbz#1974820
Signed-off-by: Nicolas Saenz Julienne <nsaenzju@redhat.com>
Enable 'isolate_managed_irq' parameter by default. It is important
for -realtime systems. When it is enabled, kernel directs interrupts
to the housekeeping CPUs and thus reduces latencies for the isolated
ones.
Fixes: RHBZ#1820626
For realtime KVM host and guests it is important to use housekeeping
CPUs for processing interrupts (IRQs) and network packet transmission.
So that isolated CPUs are used for -realtime tasks assigned to them.
The 'isolate_managed_irq' and 'netdev_queue_count' variables help
to configure kernels towards this end. Users can define these
variables via tuned(8) configuration files.
Fixes: RHBZ#1817827
Explicitly setting a value for udp_mem does not make sense.
An explicit value does not take into account how much RAM the user's system
has, so setting values here like "12G 16G 64G" works for few specific use
cases but makes no sense for others.
This actually could cause problems if applied on a system with <16G RAM
and UDP protocol usage is allowed to grow as large as it likes, causing
kernel memory pressure in other areas.
Remove these values for now. If still required, a better way to calculate
a larger value based on system RAM can be implemented later.
Signed-off-by: Jamie Bainbridge <jamie.bainbridge@gmail.com>
tuned(8) realtime-virtual-{host|guest} profiles set Kernel and
KVM module parameters via script.sh plugin. These parameters are
to be verified in a verify() function, invoked by tuned-adm verify
command.
This patch updates verify() functions to validate KVM module
parameters.
It moves kernel parameters to the tuned.conf file under the
[sysfs] & [sysctl] plugin sections. And removes call to
disable_ksm function, no longer required.
Fixes: RHBZ#1947858
This allows the user to adjust the queue count for each
network device.
Also add check_net_queue_count built-in function.
The functions check if the user has specified any
queue count value or not. If no value is passed the function
returns the number of housekeeping CPUs as the queue count.
Resolves: rhbz#1846767
Signed-off-by: Nitesh Narayan Lal <nitesh@redhat.com>
Like the cpu-partitioning profile, exclude all 'pmd' threads
to avoid repinning ovs-vswitchd pmds.
Resolves: rhbz#1861767
Signed-off-by: Christophe Fontaine <cfontain@redhat.com>
The LAPIC advancement improvement is minimal (3 or 4 us) and its
calculation has shown to be problematic under certain
scenarios (for example if qemu-kvm is not installed).
Remove it.
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
E.g. Ubuntu uses /bin/mktemp, while e.g. Fedora uses /usr/bin/mktemp.
There is probably no need to hardcode the path, so rely on the system path.
Signed-off-by: Jaroslav Škarvada <jskarvad@redhat.com>
It turned out that disablement of the numa_balancing could result in
upto 20% performance drop on some loads.
Related: rhbz#1746957
Signed-off-by: Jaroslav Škarvada <jskarvad@redhat.com>
Use the new irqbalance plugin instead of a script to set up
irqbalance.
I'm placing the 'irqbalance' instance before the 'script' instance so
that it's executed at the same time as the script, however I don't
think the 'irqbalance' instance needs to be ordered in any specific
way and could be placed almost anywhere.
Resolves: rhbz#1784645
Signed-off-by: Ondřej Lysoněk <olysonek@redhat.com>
Use the new irqbalance plugin instead of a script to set up
irqbalance.
Note that I'm not setting priority for the 'irqbalance' instance,
whereas the 'script' instance has a priority of 5. However, this was
added just for initrd image generation in commit 843dc8cf5f
and I don't think the irqbalance instance needs to be ordered in any
specific way.
Resolves: rhbz#1784645
Signed-off-by: Ondřej Lysoněk <olysonek@redhat.com>
Profile which tunes down I/O activity to the serial console by reducing the
printk value. This should make the serial console more responsive.
This profile is intended to be used as an overlay on other
profiles (e.g. throughput-performance profile), example:
# tuned-adm profile throughput-performance optimize-serial-console
Also minor fixes to the man page quoting.
Resolves: rhbz#1840689
Signed-off-by: Jaroslav Škarvada <jskarvad@redhat.com>
