The rcu_normal_after_boot=1 kernel parameter (which is enabled by
default on CONFIG_PREEMPT_RT kernels) avoids the use of IPIs
to expedite the runtime of synchronize_rcu_expedited() callers.
Enable this in tuned to avoid such IPIs on !CONFIG_PREEMPT_RT kernels
as well.
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
The ktimer_lockless_check interface was removed from Red Hat kernels.
Drop the tuned configuration as well.
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
The interruptions avoided by the rtentsk logic:
"Add plugin to open a socket with SOF_TIMESTAMPING_OPT_TX_SWHW.
This is for avoiding interruptions due to static key IPIs due
to opening socket with timestamping enabled (by opening a
socket ourselves the static key is kept enabled)."
Are also desired on systems that use cpu-partitioning.
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
Commit 49cc817871
("realtime: Set tsc as 'reliable'") reads:
"We already disable the clocksource watchdog, it's there to double-check
whether clock results are coherent between reads. But a new reliability
test was introduced in recent kernels[1], tracking tsc drift between
CPUs[2]. It works by programming timers on all CPUs, including isolated
ones, and checks whether the tsc was adjusted in between runs. This
introduces unwarranted latency on real time systems. So let's promote
tsc to 'reliable', which disables both checks."
The timer is also undesired interruption to systems that use cpu-partitioning.
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
The following sysctl settings are the same between cpu-partitioning and
realtime profiles:
kernel.hung_task_timeout_secs = 600
kernel.nmi_watchdog = 0
vm.stat_interval = 10
kernel.timer_migration = 0
Unify them in the network-latency profile, parent of both profiles.
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
With openshift/machine-config-operator#3676 we
removed globally setting ip forwarding in MCO. We should not be setting
it in the default profile for OpenShift. We now set it on a per
interface basis as needed. However, by setting proc/sys/net/ipv4/forward
rather than (forwarding) it will reset all the values:
"This variable is special, its change resets all configuration
parameters to their default state (RFC1122 for hosts, RFC1812
for routers)"
We suspect this causes upgrade to fail. NTO sets this to 1, which then
resets all the per interface config that OVNK wrote. Then during upgrade
when there is a tuned profile change, the config change is rolled back
so forward is 0, and now there is no connectivity to kapi and upgrade
fails.
Signed-off-by: Tim Rozet <trozet@redhat.com>
The new profile sets the intel_acpi driver to passive mode to
be able to set the standard acpi governors (ondemand/userspace)
and provides more flexibility on the C-states, as it is exposed
as a new variable.
Signed-off-by: Christophe Fontaine <cfontain@redhat.com>
The PREEMPT_RT upstream tree [1] is reintroducing a per-CPU kthread for
handling timer expiry named ktimers/${cpu}, and RHEL-RT is following this
approach [2].
The recommended priority tuning for that kthread is to simply match that of
ksoftirqd, which is what is implemented here.
Note that ktimersoftd is the predecessor of ktimers, and isn't present on
RT kernels newer than v5.0.19-rt11 / RHEL-RT 7.
[1]: https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=2122220
Resolves: rhbz#2122220
Signed-off-by: Valentin Schneider <vschneid@redhat.com>
In the old kernels, vm.hugepages_treat_as_movable tunable was aimed at
reducing memory fragmentation. Things have changed since then and this
tunable is no longer present in recent kernels:
https://lore.kernel.org/lkml/20171003072619.8654-1-mhocko@kernel.org/t/
Its presence in the hpc-compute profile causes TuneD to log errors.
Remove it.
Signed-off-by: Jiri Mencak <jmencak@users.noreply.github.com>
The kernel tunable fs.aio-max-nr specifies the maximum number of events
specified on the io_setup() call for all currently active aio contexts.
On large systems this causes issues with the io_setup() call failing
with EAGAIN. Libvirt already increases this setting to 1048576:
96e99e4948/src/remote/libvirtd.sysctl
The only potential drawback of a larger limit is that a malicious users
could issue parallel io_setup() requests to cause the kernel to set
aside memory.
Signed-off-by: Jiri Mencak <jmencak@users.noreply.github.com>
According to testing done by the kernel QE and performance teams,
the sched_* tunables should be dropped in several profiles.
The openshift-control-plane profile is based on other profiles
which have already dropped these tunables (network-latency,
throughput-performance).
The calc_isolated_cores function expands to the list of cores to
isolate. It accepts optional argument which specifies how many cores
from each socket reserve for housekeeping. If not specified, one core
from each socket is reserved for housekeeping.
Example:
Machine with 2 sockets, each 4 cores, using the following user variable
configuration file, e.g. /etc/tuned/realtime-variables.conf:
isolated_cores=${f:calc_isolated_cores:2}
It will expand to:
isolated_cores=2, 3, 6, 7
I.e. cores 0, 1 and 4, 5 will be used for housekeeping.
Also updated the TuneD profiles to use this builtin function for
isolated_cores preset.
Resolves: rhbz#2093847
Signed-off-by: Jaroslav Škarvada <jskarvad@redhat.com>
AWS Nitro instances need special tuning for NVMe devices:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html#timeout-nvme-ebs-volumes
[sysfs]
/sys/module/nvme_core/parameters/io_timeout=4294967295
/sys/module/nvme_core/parameters/max_retries=10
This tuning should probably be moved to Cloud Provider-specific profiles
once the functionality is implemented.
Signed-off-by: Jiri Mencak <jmencak@users.noreply.github.com>
We already disable the clocksource watchdog, it's there to double-check
whether clock results are coherent between reads. But a new reliability
test was introduced in recent kernels[1], tracking tsc drift between
CPUs[2]. It works by programming timers on all CPUs, including isolated
ones, and checks whether the tsc was adjusted in between runs. This
introduces unwarranted latency on real time systems. So let's promote
tsc to 'reliable', which disables both checks.
There shouldn't be any impact on old nor new setups. First, this test
didn't exist in the past, so any eventual drift was left uncorrected.
Second, most telco and HPC use-cases will make heavy use of tsc in
user-space, and its reliability is already assumed in the system's
design.
[1] c7719e793478 x86/tsc: Add a timer to make sure TSC_adjust is always
checked
[2] Generally caused by rogue BIOSes adjusting the tsc on a CPU and not
doing so on the rest
Signed-off-by: Nicolas Saenz Julienne <nsaenzju@redhat.com>
* add "^\[" to the start of the regexes, so they only match
the intended kernel threads, and not processes that have
the name of a kthread in their cmdline.
* remove unneccessary ".*" at start/end of regexes
Signed-off-by: Adriaan Schmidt <adriaan.schmidt@siemens.com>
The [scheduler] plug-in can be very CPU intensive, especially on the
OpenShift platform. The bug for this issue is tracked by rhbz#1921738.
Until this is fixed, work around this problem in the openshift
parent profile by adding "runtime=0" [scheduler] plug-in option.
Signed-off-by: Jiri Mencak <jmencak@users.noreply.github.com>
According to the performance team measurements the sched_ tuning
should be dropped in several profiles.
Resolves: rhbz#1957829
Signed-off-by: Jaroslav Škarvada <jskarvad@redhat.com>
New kernels (5.13 and newer) moved some sched_ and numa_ knobs from
the sysctl to the debugfs, thus add and abstract these knobs under the
scheduler plugin. With help of this abstraction it will write
the tuning to the correct place according to the kernel used.
Example:
[scheduler]
sched_migration_cost_ns = 500000
Will work on the old kernel the same way as:
[sysctl]
kernel.sched_migration_cost_ns = 500000
I.e. it will write '500000' to the:
/proc/sys/kernel/sched_migration_cost_ns
And on the new kernel it will write '500000' to the:
/sys/kernel/debug/sched/migration_cost_ns
Also updated TuneD profiles.
Resolves: rhbz#1952687
Signed-off-by: Jaroslav Škarvada <jskarvad@redhat.com>
The 'nx_huge_pages' option, will force any executable page mapping to be
performed at 4KiB granularity and requires any existing overlapping
huge-page mapping to be also split in 4KiB chunks[1]. This is due to a HW
bug that doesn't affect our NFV use-cases.
The way a huge-page mapping is dropped is by simply deleting the EPT
entry and flushing TLB caches on all affected CPUs. Next vCPU access to
that address will trigger an EPT_VIOLATION, which will jump into the
host for it to handle the page-fault. In our specific case, this means
all isolated CPUs running oslat will hit an EPT_VIOLATION almost exactly
at the same time. Which is bad enough already, but, given our systems
might have a huge number of isolated CPUs, will also create a lot of
contention over the KVM MMU lock. This has been observed to trigger
~100us latency spikes while testing with oslat.
So let's disable the 'kvm.nx_huge_pages' module option.
[1] see kernel commit b7e8c8303ff28
Resolves: rhbz#1976825
Signed-off-by: Nicolas Saenz Julienne <nsaenzju@redhat.com>
I/O APIC, one of intel's interrupt controllers, can only change the
affinity on an IRQ line during an EOI (end of interrupt)[1]. In other
words, on boot, although tuned will set a preferred IRQ affinity, it
might not be applied until the next time the interrupt is triggered.
This might be right away, or hours later while oslat or a
latency-sensitive workload is running.
To mitigate this set the 'irqaffinity='kernel command-line[2] option to
match the non-isolated CPU mask when running the realtime-virtual-*
profiles. This way we'll get the right affinity regardless of the
situation stated above.
[1] For more reference see kernel's ioapic_ack_level() function and the
IRQD_SETAFFINITY_PENDING flag.
[2] Actually, this is why irqaffinity was introduced in the first place,
see kernel commit fbf198030e0b0.
Resolves: rhbz#1974820
Signed-off-by: Nicolas Saenz Julienne <nsaenzju@redhat.com>
Enable 'isolate_managed_irq' parameter by default. It is important
for -realtime systems. When it is enabled, kernel directs interrupts
to the housekeeping CPUs and thus reduces latencies for the isolated
ones.
Fixes: RHBZ#1820626
For realtime KVM host and guests it is important to use housekeeping
CPUs for processing interrupts (IRQs) and network packet transmission.
So that isolated CPUs are used for -realtime tasks assigned to them.
The 'isolate_managed_irq' and 'netdev_queue_count' variables help
to configure kernels towards this end. Users can define these
variables via tuned(8) configuration files.
Fixes: RHBZ#1817827
Explicitly setting a value for udp_mem does not make sense.
An explicit value does not take into account how much RAM the user's system
has, so setting values here like "12G 16G 64G" works for few specific use
cases but makes no sense for others.
This actually could cause problems if applied on a system with <16G RAM
and UDP protocol usage is allowed to grow as large as it likes, causing
kernel memory pressure in other areas.
Remove these values for now. If still required, a better way to calculate
a larger value based on system RAM can be implemented later.
Signed-off-by: Jamie Bainbridge <jamie.bainbridge@gmail.com>
