CPU performance - disable bug mitigations

[Goran]

Hi all,

I have some servers with AMD Opteron 6276 CPUs and doing some number
crunching on these.

I installed Ubuntu 16.04 and was pretty much satisfied (never did an
update - running local only). Some days ago I installed 18.04 and got
a performance penalty of 33%.

My guess is, that it has something to do with Spectre. So I did a

# grep . /sys/devices/system/cpu/vulnerabilities/*
/sys/devices/system/cpu/vulnerabilities/meltdown:Not affected
/sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Mitigation:
Speculative Store Bypass disabled via prctl and seccomp
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: __user
pointer sanitization
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Full
AMD retpoline, IBPB

How can I disable these mitigations without restart (without boot
command line)? I just found some examples for Red Hat which are not
working for Ubuntu.

Further I read these pages:
https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown
https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/Variant4
https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/LazyFP
https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/BCBS

I would like to disable all mitigations related to any of these hardware-bugs.

After all I got the harshest penalty when I switched back to 16.04.5
(fresh new install) and could not get my performance back. My old
16.04 is lost so no way back.

How to get back my performance?

Regards
Goran

[Liam Proven]

Post by Goran
After all I got the harshest penalty when I switched back to 16.04.5
(fresh new install) and could not get my performance back. My old
16.04 is lost so no way back.

I think the answer is in your own post.

You said you were running 16.04, never updated.

When you reinstalled, you reinstall 16.05-05, i.e., updated, i.e. with
Spectre etc. mitigations.

Revert to unupdated 16.04 -dot-nothing and the Spectre mitigations
should not be there.
--
Liam Proven - Profile: https://about.me/liamproven
Email: ***@cix.co.uk - Google Mail/Hangouts/Plus: ***@gmail.com
Twitter/Facebook/Flickr: lproven - Skype/LinkedIn: liamproven
UK: [phone] - ČR (+ WhatsApp/Telegram/Signal): [phone]

[Ralf Mardorf via ubuntu-users]

Post by Goran
How can I disable these mitigations without restart (without boot
command line)?

You can't! You have to downgrade or remove the firmware and to reboot
to perhaps get rid of some of it. To get rid of most of it, you
likely need to downgrade the kernel.

I once tested this for an Intel machine with the microcode for Meltdown
and Spectre, for an Arch Linux install with the syslinux bootloader.

[***@archlinux ~]$ grep -i security /boot/syslinux/syslinux.cfg
LABEL Securityink
MENU LABEL Arch Linux Rt ^Securityink
LINUX ../vmlinuz-linux-rt-securityink
INITRD ../intel-ucode.img,../initramfs-linux-rt-securityink.img
LABEL Securityink_no_micro
MENU LABEL Arch Linux Rt Securityink no micro
LINUX ../vmlinuz-linux-rt-securityink
INITRD ../initramfs-linux-rt-securityink.img
LABEL Securityink_nopti
MENU LABEL Arch Linux Rt Securityink nopt^i
LINUX ../vmlinuz-linux-rt-securityink
INITRD ../intel-ucode.img,../initramfs-linux-rt-securityink.img

I don't have got the results at hand, but even without loading the
microcode I didn't get rid of everything.

You are aware that for the later kernels you anyway might have changes
that could make a difference regarding performance? I've got in mind
that for older kernels disabling CONFIG_AUDIT could make a difference,
for newer kernels it gains nothing, see
https://lists.archlinux.org/pipermail/arch-general/2018-September/045580.html .
I guess its irrelevant for Ubuntu kernels that anyway always were
configured regarding security and not regarding performance, but it
could make a difference for local build Ubuntu kernels.

[Ralf Mardorf via ubuntu-users]

Post by Ralf Mardorf via ubuntu-users
I once tested this for an Intel machine with the microcode for Meltdown
and Spectre, for an Arch Linux install with the syslinux bootloader.
LABEL Securityink
MENU LABEL Arch Linux Rt ^Securityink
LINUX ../vmlinuz-linux-rt-securityink
INITRD ../intel-ucode.img,../initramfs-linux-rt-securityink.img
LABEL Securityink_no_micro
MENU LABEL Arch Linux Rt Securityink no micro
LINUX ../vmlinuz-linux-rt-securityink
INITRD ../initramfs-linux-rt-securityink.img
LABEL Securityink_nopti
MENU LABEL Arch Linux Rt Securityink nopt^i
LINUX ../vmlinuz-linux-rt-securityink
INITRD ../intel-ucode.img,../initramfs-linux-rt-securityink.img

Oops! It doesn't really matter, however, just in case all relevant
entries:

[***@archlinux ~]$ grep -i security /boot/syslinux/syslinux.cfg -A1
LABEL Securityink
MENU LABEL Arch Linux Rt ^Securityink
LINUX ../vmlinuz-linux-rt-securityink
APPEND root=LABEL=s3.archlinux ro
INITRD ../intel-ucode.img,../initramfs-linux-rt-securityink.img

LABEL Securityink_no_micro
MENU LABEL Arch Linux Rt Securityink no micro
LINUX ../vmlinuz-linux-rt-securityink
APPEND root=LABEL=s3.archlinux ro
INITRD ../initramfs-linux-rt-securityink.img

--
LABEL Securityink_nopti
MENU LABEL Arch Linux Rt Securityink nopt^i
LINUX ../vmlinuz-linux-rt-securityink
APPEND root=LABEL=s3.archlinux ro nopti
INITRD ../intel-ucode.img,../initramfs-linux-rt-securityink.img

[Colin Watson]

This post might be inappropriate. Click to display it.

[Ralf Mardorf via ubuntu-users]

On Wed, Nov 21, 2018 at 08:20:56AM +0100, Ralf Mardorf via

Post by Ralf Mardorf via ubuntu-users

Post by Goran
How can I disable these mitigations without restart (without boot
command line)?

You can't! You have to downgrade or remove the firmware and to reboot
to perhaps get rid of some of it. To get rid of most of it, you
likely need to downgrade the kernel.

Or you could just use the kernel parameters that control it.
pti=off spectre_v2=off spec_store_bypass_disable=off
(This does require restarting, but is less invasive than downgrading
the kernel.)

Hi,

excepted of "pti=off"/"nopti" I wasn't aware about those kernel
parameters. However, "pti=off"/"nopti" isn't required for the OP's CPU,
since Meltdown is not affected see on Tue, 20 Nov 2018 17:05:08 +0100,

# grep . /sys/devices/system/cpu/vulnerabilities/*
/sys/devices/system/cpu/vulnerabilities/meltdown:Not affected
Speculative Store Bypass disabled via prctl and seccomp
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: __user
pointer sanitization
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Full
AMD retpoline, IBPB

I'm uncertain if

"spectre_v2=off spec_store_bypass_disable=off"

does disable all mitigations, let alone that

$ grep CONFIG_AUDIT= config-4.4.0-139-lowlatency
CONFIG_AUDIT=y

for the 16.04 kernels is enabled and I don't know if they still provide
a "faster path" with mitigations disabled, while CONFIG_AUDIT is
enabled.

Btw. I'm not booted into Ubuntu 16.04's 4.4.0-139-lowlatency now, but
into a newer Arch kernel and each upgrade to a new kernel could provide
more mitigations, currently I get

$ uname -r; cat /sys/devices/system/cpu/vulnerabilities/*; pacman -Q linux-firmware intel-ucode; hwinfo --cpu | grep Model | sort -u
4.19.1-rt3-0-rt
Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT disabled
Mitigation: PTI
Mitigation: Speculative Store Bypass disabled via prctl and seccomp
Mitigation: __user pointer sanitization
Mitigation: Full generic retpoline, IBPB, IBRS_FW
linux-firmware 20181026.1cb4e51-1
intel-ucode 20180807.a-1
Model: 6.60.3 "Intel(R) Celeron(R) CPU G1840 @ 2.80GHz"

IOW this likely could become a kernel parameter building lot with each
upgrade, for an Ubuntu machine with an AMD CPU, too.

IIRC

$ ls re*
retpoline-4.4.0-139-lowlatency

wasn't provided by the default 16.04 install, AFAIR it was installed by
an upgrade.

Regards,
Ralf

[Ralf Mardorf via ubuntu-users]

However, "pti=off"/"nopti" isn't required for the OP's CPU, since
Meltdown is not affected

I might be mistaken, that just the CPU isn't affect, not necessarily
might mean that the kernel's page table isolation does have no impact
on the machine using this CPU. I don't know how this actually is
handled. My apologies for the "guessing".

[Goran]

Hi Ralf, as far as I understand your mails, it won't be enough just to
put disable-mitigations commands on the kernel command line. Is this a
correct understanding?

Further as far as I understand one kernel option (CONFIG_AUDIT=n) must
be deactivated in the running kernel. Is this correct?

You are aright, nopti is not needed for my CPU. I putted all
disable-mitigations commands on it, just to be sure. Maybe the server
will change. It's booting over the network and could easily replaced.
Am Mi., 21. Nov. 2018 um 10:27 Uhr schrieb Ralf Mardorf via

Post by Ralf Mardorf via ubuntu-users

However, "pti=off"/"nopti" isn't required for the OP's CPU, since
Meltdown is not affected

I might be mistaken, that just the CPU isn't affect, not necessarily
might mean that the kernel's page table isolation does have no impact
on the machine using this CPU. I don't know how this actually is
handled. My apologies for the "guessing".
--
ubuntu-users mailing list
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users

[Ralf Mardorf via ubuntu-users]

Post by Goran
Hi Ralf, as far as I understand your mails, it won't be enough just to
put disable-mitigations commands on the kernel command line. Is this a
correct understanding?

Hi,

yesno. Perhaps it works, I just didn't know all those kernel parameters
before Colin mentioned them. However, assuming the kernel parameters
should do the job, you might need to add new kernel parameters to the
boot options after upgrading the kernel, linux-firmware or
amd64-microcode (intel-microcode for other CPUs).

Post by Goran
Further as far as I understand one kernel option (CONFIG_AUDIT=n) must
be deactivated in the running kernel. Is this correct?

I don't know. Perhaps it's irrelevant, perhaps it makes a difference.

Post by Goran
You are aright, nopti is not needed for my CPU.

The CPU might not need page table isolation to increase security, but
perhaps you need to disable it. That the CPU doesn't need it, does not
necessarily mean that enabled page table isolation has got no impact
when using your AMD CPU. I don't know if it makes a difference if you
stay with page table isolation or if you disable it.

I also wonder how much the kernel code changes with each upgrade,
perhaps downgrading a kernel still makes a difference, even with all
mitigations disabled.

I'm very lazy with testing all those options. I tend to use my computer
for real-time audio usage with all mitigations enabled, with audit
enabled, latest kernel, firmware and microcode updates and assuming I
should run into a serious issues, I would consider to buy a faster CPU
for my machine, since I've got not enough knowledge about this domain
and trial and error tests are too time consuming, too annoying for my
taste.

Regards,
Ralf