GCC memory starvation caused by flatten attribute with LTO
Submitted by Martin Liska
Assigned to Chris Wilson @ickle
Description
Created attachment 97520
LTO patch
Hello,
I've been testing GCC 4.9 for a virtual gentoo machine and I noticed that you us flatten attribute in source code. In case of src/sna/sna_glyphs.c flatten functions, inliner inlines about 3.3M functions and crashes because of no free memory (I have 8GB memory).
Please notice that LTO has ability to optimize whole program. As a result, it sees almost all function bodies and that leads to enormous inlining.
Suggested patch removes these flatten attributes for selected functions.
Thank you,
MArtin
Attachment 97520, "LTO patch":
xf86-intel-drive.patch
Activity
I have wanted for a long time to be able to use LTO on xf86-video-intel, so I was very pleased when I found this patch submission. I tried it on the latest development version of the Intel driver from git. It applied successfully, and I was able to successfully compile the Intel driver using LTO instead of experiencing the seeming infinite compile time otherwise resulting from LTO.
However, despite compilation being successful, my tests which involved glxgears, monitoring CPU usage, and watching videos on Youtube showed significantly poorer video performance with the LTO-compiled driver than without LTO. Though glxgears showed no discernible difference, Youtube performance was incredibly slow such that the audio portion of the video continued at normal speed while the video lagged progressively further behind in slow motion.
I have no way of knowing if LTO is directly responsible for the poor performance or if the patch somehow led to poor optimization by LTO, but this should be investigated further. I used GCC 4.9.3 for the test. My Linux distro currently doesn't offer the 5.x branch of GCC, so I was unable to test with GCC 5.3.(In reply to Patrick McMunn from comment 2)
I have wanted for a long time to be able to use LTO on xf86-video-intel, so
I was very pleased when I found this patch submission. I tried it on the
latest development version of the Intel driver from git. It applied
successfully, and I was able to successfully compile the Intel driver using
LTO instead of experiencing the seeming infinite compile time otherwise
resulting from LTO.
However, despite compilation being successful, my tests which involved
glxgears, monitoring CPU usage, and watching videos on Youtube showed
significantly poorer video performance with the LTO-compiled driver than
without LTO. Though glxgears showed no discernible difference, Youtube
performance was incredibly slow such that the audio portion of the video
continued at normal speed while the video lagged progressively further
behind in slow motion.
I have no way of knowing if LTO is directly responsible for the poor
performance or if the patch somehow led to poor optimization by LTO, but
this should be investigated further. I used GCC 4.9.3 for the test. My Linux
distro currently doesn't offer the 5.x branch of GCC, so I was unable to
test with GCC 5.3.
Hi Patrick.
Well, it looks that xf86-video-intel driver needs flattened functions to produce optimal code. It would be interesting, if you rebuild the driver with the suggested patch applied (or is it part of mainline?) and try to generate perf report that can provide comparison between LTO and non-LTO build.
MartinI know it's been a while since I followed up. First of all, I don't know how to generate a perf report, so that's partly why I didn't follow up sooner. Perhaps you can point me to a tutorial?
Also, in retrospect, I suppose there's a possibility that I may have compiled incorrectly. They -flto compiler flag, as well as the same compiler optimization flags used during compilation, must also be used during the linking phase. I use Funtoo, a variant of Gentoo, and I had -flto in my CFLAGS/CXXFLAGS but not in my LDFLAGS. This is handled by different programs makefiles. Some automatically add CXXFLAGS to LDFLAGS during the linking phase; some don't. I'm not sure about xf86-video-intel's linking phase. So it's possible that, in my earlier attempt, it wasn't linked and optimized properly. So I should probably revisit that.
In my earlier attempt, I was using GCC 4.9.3. Since that time, I've been using GCC 5.3.0 and, more recently, 6.1.0. GCC 5 and 6 fail the compilation phase before it even gets to the linking phase. Though that can be resolved by removing a few force_inline directives in the source code. In fact, I tried removing every instance of force_inline, and it not only allowed compilation to complete, it slightly reduced the memory used during linking. Alas linking still failed, apparently due to using too much memory.
Anyway, I'll give your patch another go just in case what I described above is what caused poorer performance with the LTO build. I really would like to get LTO working with this driver. LTO has really improved a lot in GCC with the 6.1.0 release.
I'm attaching a patch that takes a somewhat simpler approach to disabling "flatten" and "force_inline" which cause problems with LTO.
I used glxgears to measure FPS. The system was a Pentium 4 with an integrated 845G graphics chip, so it's pretty underpowered hardware, hence the low FPS. Without LTO, I got ~19.5 FPS running glxgears at fullscreen at 1680x1050 resolution. With LTO, I got about 22.5 FPS. That may not seem like much, but that's a 10% performance improvement! Except for the LTO-related flags, I used the same compiler and linker flags during compilation.
flags used during compilation:
CFLAGS="-O2 -march=pentium4 -pipe -fdevirtualize-speculatively ${LTO}"
CXXFLAGS="${CFLAGS}"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--enable-new-dtags -Wl,--sort-common -Wl,--sort-section=name -Wl,-Bsymbolic-functions -Wl,-z,combreloc ${CXXFLAGS}"Patch 125089, "Patch to allow LTO optmization of xf86-video-intel":
xf86-video-intel-lto.patch@haarpsaid:(In reply to Patrick McMunn from comment 5)
Created attachment 125089 [details] [review]
Patch to allow LTO optmization of xf86-video-intel
Nice work! This allows building with LTO on gcc-4.9. gcc-5.3.0 however fails to build it, throwing a bunch of these around:
/usr/include/bits/string3.h:50:1: error: inlining failed in call to always_inline ‘memcpy’: target specific option mismatch
Also, unlike your P4, I cannot detect any measurable performance improvements (Intel Sandy Bridge). Unfortunate.
All tests done with xf86-video-intel from today's git.@marxin.liskasaid:I can confirm that the attached patch helps to build the project with LTO enabled. I tried both 5.3.1 and latest trunk (7.0.0) and both work fine.
Can you please attach pre-processed source code for the issue:
/usr/include/bits/string3.h:50:1: error: inlining failed in call to always_inline ‘memcpy’: target specific option mismatch
and also command line arguments for GCC.
Thanks(In reply to main.haarp from comment 6)
(In reply to Patrick McMunn from comment 5)
Created attachment 125089 [details] [review] [review]
Patch to allow LTO optmization of xf86-video-intel
Nice work! This allows building with LTO on gcc-4.9. gcc-5.3.0 however fails
to build it, throwing a bunch of these around:
/usr/include/bits/string3.h:50:1: error: inlining failed in call to
always_inline ‘memcpy’: target specific option mismatch
Also, unlike your P4, I cannot detect any measurable performance
improvements (Intel Sandy Bridge). Unfortunate.
All tests done with xf86-video-intel from today's git.
Hmm... You shouldn't be getting that error. That's the error I was getting before I removed the "always_inline" definitions. If the patch worked properly for you, the compiler shouldn't be running into a "always_inline" directive.
I don't think I actually tested it with 5.3.0. I only verified that it builds with 4.9.3, and all my tests with performance involved 6.1.0.
I did do numerous tests involving various compiler flags such as -O2, -O3, graphite compiler flags, etc, and I found that I actually got reduced performance with -O3 compared to -O2, and I got reduced performance with using additional optimizations like graphite. The biggest improvement I got was with simple -O2 and a proper -march setting.Also, make sure that you're passing the same flags to the linker that you're passing to the compiler. If your compiler flags are something like
CXXFLAGS="-O2 -march=native -flto -ffat-lto-objects -fuse-linker-plugin"
then your linker flags should look something like
LDFLAGS="-Wl,-O1 -Wl,--as-needed -O2 -march=native -flto -ffat-lto-objects -fuse-linker-plugin"
Otherwise it won't be properly optimized during the linking phase.
@haarpuploaded an attachment:(In reply to Martin Liška from comment 7)
I can confirm that the attached patch helps to build the project with LTO
enabled. I tried both 5.3.1 and latest trunk (7.0.0) and both work fine.
Can you please attach pre-processed source code for the issue:
/usr/include/bits/string3.h:50:1: error: inlining failed in call to
always_inline ‘memcpy’: target specific option mismatch
and also command line arguments for GCC.
Thanks
Here are the command line arguments and the full error log.
-------------------------
gcc-5.4.0 -DHAVE_CONFIG_H -I. -I/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src/sna -I../.. -Wall -Wpointer-arith -Wmissing-declarations -Wformat=2 -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -Wbad-function-cast -Wold-style-definition -Wdeclaration-after-statement -Wunused -Wuninitialized -Wshadow -Wmissing-noreturn -Wmissing-format-attribute -Wredundant-decls -Wlogical-op -Werror=implicit -Werror=nonnull -Werror=init-self -Werror=main -Werror=missing-braces -Werror=sequence-point -Werror=return-type -Werror=trigraphs -Werror=array-bounds -Werror=write-strings -Werror=address -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -fno-strict-aliasing -I/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src -I/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src/render_program -fvisibility=hidden -I/usr/include/xorg -I/usr/include/X11/dri -I/usr/include/libdrm -I/usr/include/pixman-1 -I/usr/include/libdrm -Wno-cast-qual -Wno-redundant-decls -Wno-maybe-uninitialized -march=native -O2 -pipe -fno-stack-protector -flto=4 -c /var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src/sna/blt.c
In file included from /usr/include/features.h:365:0,
from /usr/include/stdint.h:25,
from /usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/include/stdint.h:9,
from /var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src/sna/sna.h:40,
from /var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src/sna/blt.c:32:
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src/sna/blt.c: In function ‘to_memcpy’:
/usr/include/bits/string3.h:50:1: error: inlining failed in call to always_inline ‘memcpy’: target specific option mismatch
__NTH (memcpy (void *__restrict __dest, const void *__restrict __src,
^
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src/sna/blt.c:490:4: error: called from here
memcpy(dst, src, len);
^
In file included from /usr/include/features.h:365:0,
from /usr/include/stdint.h:25,
from /usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/include/stdint.h:9,
from /var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src/sna/sna.h:40,
from /var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src/sna/blt.c:32:
/usr/include/bits/string3.h:50:1: error: inlining failed in call to always_inline ‘memcpy’: target specific option mismatch
__NTH (memcpy (void *__restrict __dest, const void *__restrict __src,
^
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src/sna/blt.c:555:2: error: called from here
memcpy(dst, src, len & 3);
^
In file included from /usr/include/features.h:365:0,
from /usr/include/stdint.h:25,
from /usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/include/stdint.h:9,
from /var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src/sna/sna.h:40,
from /var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src/sna/blt.c:32:
/usr/include/bits/string3.h:50:1: error: inlining failed in call to always_inline ‘memcpy’: target specific option mismatch
__NTH (memcpy (void *__restrict __dest, const void *__restrict __src,
^
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src/sna/blt.c:490:4: error: called from here
memcpy(dst, src, len);
^
In file included from /usr/include/features.h:365:0,
from /usr/include/stdint.h:25,
from /usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/include/stdint.h:9,
from /var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src/sna/sna.h:40,
from /var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src/sna/blt.c:32:
/usr/include/bits/string3.h:50:1: error: inlining failed in call to always_inline ‘memcpy’: target specific option mismatch
__NTH (memcpy (void *__restrict __dest, const void *__restrict __src,
^
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-9999/src/sna/blt.c:555:2: error: called from here
memcpy(dst, src, len & 3);
^
Makefile:655: recipe for target 'blt.lo' failed
-----------------------------------
Attached is what gcc generates with -E, I think that's what you needed, correct?
Thanks!Attachment 125281, "pre-processed sources":
blt.E.xz@marxin.liskasaid:(In reply to main.haarp from comment 10)
Created attachment 125281 [details]
pre-processed sources
(In reply to Martin Liška from comment 7)I can confirm that the attached patch helps to build the project with LTO
enabled. I tried both 5.3.1 and latest trunk (7.0.0) and both work fine.
Can you please attach pre-processed source code for the issue:
/usr/include/bits/string3.h:50:1: error: inlining failed in call to
always_inline ‘memcpy’: target specific option mismatch
and also command line arguments for GCC.
Thanks
Here are the command line arguments and the full error log.
-------------------------
gcc-5.4.0 -DHAVE_CONFIG_H -I.
-I/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src/sna -I../.. -Wall -Wpointer-arith -Wmissing-declarations -Wformat=2
-Wstrict-prototypes -Wmissing-prototypes -Wnested-externs
-Wbad-function-cast -Wold-style-definition -Wdeclaration-after-statement
-Wunused -Wuninitialized -Wshadow -Wmissing-noreturn
-Wmissing-format-attribute -Wredundant-decls -Wlogical-op -Werror=implicit
-Werror=nonnull -Werror=init-self -Werror=main -Werror=missing-braces
-Werror=sequence-point -Werror=return-type -Werror=trigraphs
-Werror=array-bounds -Werror=write-strings -Werror=address
-Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -fno-strict-aliasing
-I/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src
-I/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src/render_program -fvisibility=hidden -I/usr/include/xorg
-I/usr/include/X11/dri -I/usr/include/libdrm -I/usr/include/pixman-1
-I/usr/include/libdrm -Wno-cast-qual -Wno-redundant-decls
-Wno-maybe-uninitialized -march=native -O2 -pipe -fno-stack-protector
-flto=4 -c
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src/sna/blt.c
In file included from /usr/include/features.h:365:0,
from /usr/include/stdint.h:25,
from
/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/include/stdint.h:9,
from
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src/sna/sna.h:40,
from
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src/sna/blt.c:32:
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src/sna/blt.c: In function ‘to_memcpy’:
/usr/include/bits/string3.h:50:1: error: inlining failed in call to
always_inline ‘memcpy’: target specific option mismatch
__NTH (memcpy (void *__restrict __dest, const void *__restrict __src,
^
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src/sna/blt.c:490:4: error: called from here
memcpy(dst, src, len);
^
In file included from /usr/include/features.h:365:0,
from /usr/include/stdint.h:25,
from
/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/include/stdint.h:9,
from
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src/sna/sna.h:40,
from
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src/sna/blt.c:32:
/usr/include/bits/string3.h:50:1: error: inlining failed in call to
always_inline ‘memcpy’: target specific option mismatch
__NTH (memcpy (void *__restrict __dest, const void *__restrict __src,
^
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src/sna/blt.c:555:2: error: called from here
memcpy(dst, src, len & 3);
^
In file included from /usr/include/features.h:365:0,
from /usr/include/stdint.h:25,
from
/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/include/stdint.h:9,
from
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src/sna/sna.h:40,
from
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src/sna/blt.c:32:
/usr/include/bits/string3.h:50:1: error: inlining failed in call to
always_inline ‘memcpy’: target specific option mismatch
__NTH (memcpy (void *__restrict __dest, const void *__restrict __src,
^
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src/sna/blt.c:490:4: error: called from here
memcpy(dst, src, len);
^
In file included from /usr/include/features.h:365:0,
from /usr/include/stdint.h:25,
from
/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/include/stdint.h:9,
from
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src/sna/sna.h:40,
from
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src/sna/blt.c:32:
/usr/include/bits/string3.h:50:1: error: inlining failed in call to
always_inline ‘memcpy’: target specific option mismatch
__NTH (memcpy (void *__restrict __dest, const void *__restrict __src,
^
/var/tmp/portage/x11-drivers/xf86-video-intel-9999/work/xf86-video-intel-
9999/src/sna/blt.c:555:2: error: called from here
memcpy(dst, src, len & 3);
^
Makefile:655: recipe for target 'blt.lo' failed
-----------------------------------
Attached is what gcc generates with -E, I think that's what you needed,
correct?
Thanks!
I can confirm that, please take a look at the created PR:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71991@haarpsaid:(In reply to Martin Liška from comment 11)
I can confirm that, please take a look at the created PR:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71991
Interesting, thanks for filing this!assigned to @ickle
