[xen iommu] After upgrading to Linux 3.19, desktop no longer works in Xen 4.5.0 dom0

added Community GPU hang platform: ILK priority::medium severity::major + 1 deleted label

Ting-Wei Lan @lantw uploaded an attachment:

Attachment 115079, "Screenshot when the system is running in single user mode":

Ting-Wei Lan @lantw uploaded an attachment:

~~Attachment 115080~~, "dmesg":
i915-dmesg

Ting-Wei Lan @lantw uploaded an attachment:

~~Attachment 115081~~, "/sys/class/drm/card0/error":
i915-error

Ting-Wei Lan @lantw said:

git bisect shows the bad commit is https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=47591df

Jani Nikula @jani said:

(In reply to Ting-Wei Lan from comment 4)

git bisect shows the bad commit is
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/
?id=47591df

commit 47591df505129c9774af6cca2debf283a6e56ed7
Author: Juergen Gross
Date: Mon Nov 3 14:02:04 2014 +0100

xen: Support Xen pv-domains using PAT

Please report this to xen folks. I'll leave this open for tracking purposes for now, although I was tempted to resolve NOTOURBUG.

Ander Conselvan de Oliveira said:

Was this reported to Xen folks? I don't think i915 developers will attempt to fix this, and it has been over a month, so closing as NOTOURBUG.

Ting-Wei Lan @lantw said:

It seems this problem is related to Intel VT-d. If I disable VT-d by adding iommu=off to Xen boot options, this error will not happen.

Ting-Wei Lan @lantw said:

I think I should reopen this bug because the problem also happens without using Xen.

http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg02394.html
http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg02387.html

This problem also happens on Linux >= 3.7 without using Xen when 'intel_iommu=on' is used. It can be worked around by adding 'intel_iommu=igfx_off'. Is it an expected behavior or a bug? Here are some 'dmesg | grep -i iommu' outputs.

Linux 3.6.11 with intel_iommu=on works fine.
[ +0.000000] Intel-IOMMU: enabled
[ +0.005366] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap
c9008020e30272 ecap 1000
[ +0.005360] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap
c0000020230272 ecap 1000
[ +0.005359] dmar: IOMMU 2: reg_base_addr fed93000 ver 1:0 cap
c9008020630272 ecap 1000
[ +0.003267] IOMMU 0 0xfed90000: using Register based invalidation
[ +0.006143] IOMMU 2 0xfed93000: using Register based invalidation
[ +0.006141] IOMMU: Setting RMRR:
[ +0.003298] IOMMU: Setting identity map for device 0000:00:1d.0
[0xd7aec000 - 0xd7afffff]
[ +0.008310] IOMMU: Setting identity map for device 0000:00:1a.0
[0xd7aec000 - 0xd7afffff]
[ +0.008269] IOMMU: Setting identity map for device 0000:00:1d.0
[0xe4000 - 0xe7fff]
[ +0.007753] IOMMU: Setting identity map for device 0000:00:1a.0
[0xe4000 - 0xe7fff]
[ +0.007753] IOMMU: Prepare 0-16MiB unity mapping for LPC
[ +0.005376] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 -
0xffffff]

Linux >= 3.7 without any intel_iommu argument works fine.
[ +0.005391] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap
c9008020e30272 ecap 1000
[ +0.005385] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap
c0000020230272 ecap 1000
[ +0.005384] dmar: IOMMU 2: reg_base_addr fed93000 ver 1:0 cap
c9008020630272 ecap 1000

Linux >= 3.7 with intel_iommu=on causes grahpics problems.
[ +0.000000] Intel-IOMMU: enabled
[ +0.005391] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap
c9008020e30272 ecap 1000
[ +0.005382] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap
c0000020230272 ecap 1000
[ +0.005383] dmar: IOMMU 2: reg_base_addr fed93000 ver 1:0 cap
c9008020630272 ecap 1000
[ +0.003430] IOMMU: dmar1 using Register based invalidation
[ +0.005553] IOMMU: dmar0 using Register based invalidation
[ +0.005559] IOMMU: dmar2 using Register based invalidation
[ +0.005560] IOMMU: Setting RMRR:
[ +0.003314] IOMMU: Setting identity map for device 0000:00:1a.0
[0xd7aec000 - 0xd7afffff]
[ +0.008341] IOMMU: Setting identity map for device 0000:00:1d.0
[0xd7aec000 - 0xd7afffff]
[ +0.008334] IOMMU: Setting identity map for device 0000:00:02.0
[0xd7c00000 - 0xdfffffff]
[ +0.009797] IOMMU: Setting identity map for device 0000:00:1a.0
[0xe4000 - 0xe7fff]
[ +0.007795] IOMMU: Setting identity map for device 0000:00:1d.0
[0xe4000 - 0xe7fff]
[ +0.007798] IOMMU: Prepare 0-16MiB unity mapping for LPC
[ +0.005398] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 -
0xffffff]

Linux >= 3.7 with intel_iommu=igfx_off works fine.
[ +0.000000] Intel-IOMMU: disable GFX device mapping
[ +0.005388] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap
c9008020e30272 ecap 1000
[ +0.005385] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap
c0000020230272 ecap 1000
[ +0.005383] dmar: IOMMU 2: reg_base_addr fed93000 ver 1:0 cap
c9008020630272 ecap 1000

Linux >= 3.7 with both intel_iommu=on and intel_iommu=igfx_off also
works fine.
[ 0.000000] Intel-IOMMU: disable GFX device mapping
[ 0.000000] Intel-IOMMU: enabled
[ 0.205011] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap
c9008020e30272 ecap 1000
[ 0.218432] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap
c0000020230272 ecap 1000
[ 0.231848] dmar: IOMMU 2: reg_base_addr fed93000 ver 1:0 cap
c9008020630272 ecap 1000
[ 1.873199] IOMMU: dmar0 using Register based invalidation
[ 1.878757] IOMMU: dmar2 using Register based invalidation
[ 1.884315] IOMMU: Setting RMRR:
[ 1.887631] IOMMU: Setting identity map for device 0000:00:1a.0
[0xd7aec000 - 0xd7afffff]
[ 1.895972] IOMMU: Setting identity map for device 0000:00:1d.0
[0xd7aec000 - 0xd7afffff]
[ 1.904285] IOMMU: Setting identity map for device 0000:00:1a.0
[0xe4000 - 0xe7fff]
[ 1.912079] IOMMU: Setting identity map for device 0000:00:1d.0
[0xe4000 - 0xe7fff]
[ 1.919871] IOMMU: Prepare 0-16MiB unity mapping for LPC
[ 1.925268] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0

0xffffff]

It seems the difference between working and broken arguments is 'device 0000:00:02.0', which is the Intel integrated graphics controller.

David Woodhouse @dwmw2 said:

It's odd that it was triggered (in the Xen case) by a PAT patch.

What was the actual effect of that patch on the caching mode used by the machine in question?

[ +0.005382] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap
c0000020230272 ecap 1000

cap & (1<<4) is set, which is the RWBF bit:

1: Indicates software must explicitly flush
the write buffers to ensure updates made to
memory-resident remapping structures are
visible to hardware.

ecap & (1<<0) is clear, which is the Coherency bit:

This field indicates if hardware access to the
root, context, extended-context and
interrupt-remap tables, and second-level
paging structures for requests-without-
PASID, are coherent (snooped) or not.
• 0:Indicates hardware accesses to
remapping structures are non-coherent.

So basically this hardware is in a mode where the IOMMU page tables are non-cache coherent. Not only do you have to clflush every cache line in the page tables to main memory when you write it, but you *also* have to jump through hoops to ensure that the writes are pushed through chipset-specific write buffers (see §6.8 of the VT-d specification).

That may help to explain why a seemingly innocent PAT change might have triggered something odd. But it would be good to know precisely what went wrong.

Also, does it help to add 'iommu=pt' to the kernel command line? That would make the IOMMU use a 1:1 mapping of all memory, rather than dynamically setting up mappings.

You say it can be reproduced without Xen, with Linux >= 3.7 — can you show the details of that please? And if it doesn't occur in 3.6, can you also bisect the non-Xen case to find when it started happening, please?

Thanks,

Ting-Wei Lan @lantw said:

(In reply to David Woodhouse from comment 9)

It's odd that it was triggered (in the Xen case) by a PAT patch.

What was the actual effect of that patch on the caching mode used by the
machine in question?

[ +0.005382] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap
c0000020230272 ecap 1000

cap & (1<<4) is set, which is the RWBF bit:

1: Indicates software must explicitly flush
the write buffers to ensure updates made to
memory-resident remapping structures are
visible to hardware.

ecap & (1<<0) is clear, which is the Coherency bit:

This field indicates if hardware access to the
root, context, extended-context and
interrupt-remap tables, and second-level
paging structures for requests-without-
PASID, are coherent (snooped) or not.
• 0:Indicates hardware accesses to
remapping structures are non-coherent.

So basically this hardware is in a mode where the IOMMU page tables are
non-cache coherent. Not only do you have to clflush every cache line in the
page tables to main memory when you write it, but you *also* have to jump
through hoops to ensure that the writes are pushed through chipset-specific
write buffers (see §6.8 of the VT-d specification).

That may help to explain why a seemingly innocent PAT change might have
triggered something odd. But it would be good to know precisely what went
wrong.

Can you tell me how can I test it or provide me a link that describes steps to get needed information? I am not familiar with VT-d spec.

There were discussion on Xen-devel when I tried to make a workaround.
http://lists.xenproject.org/archives/html/xen-devel/2015-07/msg03642.html
http://lists.xenproject.org/archives/html/xen-devel/2015-07/msg03723.html

>
> Also, does it help to add 'iommu=pt' to the kernel command line? That would
> make the IOMMU use a 1:1 mapping of all memory, rather than dynamically
> setting up mappings.

No, screen output is still broken.

>
> You say it can be reproduced without Xen, with Linux >= 3.7 — can you show
> the details of that please? And if it doesn't occur in 3.6, can you also
> bisect the non-Xen case to find when it started happening, please?

Non-Xen case is already reported here:
https://bugs.freedesktop.org/show_bug.cgi?id=91127

Bisect result:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=edef7e6

Non-Xen case is partially fixed now. Screen output works fine, but the system crashes after using for several hours.

>
> Thanks,

Ting-Wei Lan @lantw closed a related bug:

*** Bug 91400 has been marked as a duplicate of this bug. ***

Elizabeth said:

Good afternoon,
Sorry for the long delay. Last kernel reported on this case has been 4.0 that is quite old and lots of changes have been made since that, so I'm closing this bug as invalid. If problem persist on newest kernel versions https://www.kernel.org/ please open a new bug with HW and SW information, logs and steps to reproduce. Thank you.

Ting-Wei Lan @lantw said:

I can reproduce the problem with the same hardware running Xen 4.8.2 and Linux 4.13.2 unless iommu=no-igfx is passed to Xen hypervisor command line.

Elizabeth said:

Hello again,
Could you please attach a new dmesg log and error state with newer kernel version with parameters drm.debug=0x1e log_bug_len=2M (or bigger) on grub?
Thank you.

Elizabeth said:

I'm probably wrong, but this issue may be related to bug 89360.

Ting-Wei Lan @lantw uploaded an attachment:

It took me more than 1 hour to get this file ... It crashed too quickly.

Xen dmesg messages were obtained from serial console and 'xl dmesg' command. Linux dmesg messages earlier than timestamp 520.360867 were obtained from 'dmesg' command. All messages after it were obtained from serial console because the system crashed and the ssh connection was broken.

I disabled wayland in /etc/gdm/custom.conf in order to get the result. The system also crashed in wayland mode, but there was no crash dump file or drm message.

Steps of operations:

In GRUB menu, remove 'iommu=no-igfx' from Xen command line and add 'drm.debug=0x1e log_buf_len=64M s' to Linux command line.
Boot the system and wait 5 minutes to get single user shell.
Delete /var/run/nologin.
Mount /proc/xen.
Start NetworkManager and sshd.
Connect to the host from ssh and run 'xl dmesg' and 'dmesg -w' commands.
Leave single user shell to continue normal boot.
Once the screen output becomes more broken, type 'sudo cat /sys/class/drm/card0/error > gpu_crash_dump; sudo sync' command as soon as possible because the system will stop responding within a few seconds.
Reboot the system with Xen console command 'R'.
Boot the system normally to download 'gpu_crash_dump' file.

Attachment 136084, "dmesg (Xen 4.8.2 + Linux 4.14.4)":
linux-4.14.4-xen-crash-dmesg

Ting-Wei Lan @lantw uploaded an attachment:

Attachment 136085, "/sys/class/drm/card0/error":
i915-error-414

Elizabeth said:

(In reply to Elizabeth from comment 15)

I'm probably wrong, but this issue may be related to bug 89360.
Yep, wrong. By previous comments situation seems to be the same pointing to a NOTOURBUG, though there is the VT-d. Let me ping some people to verify.

Jani Saarinen @jani.saarinen said:

First of all. Sorry about spam.
This is mass update for our bugs.

Sorry if you feel this annoying but with this trying to understand if bug still valid or not.
If bug investigation still in progress, please ignore this and I apologize!

If you think this is not anymore valid, please comment to the bug that can be closed.
If you haven't tested with our latest pre-upstream tree(drm-tip), can you do that also to see if issue is valid there still and if you cannot see issue there, please comment to the bug.

[xen iommu] After upgrading to Linux 3.19, desktop no longer works in Xen 4.5.0 dom0

Submitted by Ting-Wei Lan `@lantw`

Description

Blocking

Child items 0

Activity

Admin message

[xen iommu] After upgrading to Linux 3.19, desktop no longer works in Xen 4.5.0 dom0

Submitted by Ting-Wei Lan @lantw

Description

Blocking

Activity

Submitted by Ting-Wei Lan `@lantw`