GNU Assembler error building CsGui from 11.28 pull

Post Reply
asahukar
Posts: 5
Joined: Sat Dec 19 2020 12:25 am

GNU Assembler error building CsGui from 11.28 pull

Post by asahukar »

I have run into what I believe is a GNU assembler error building a number of the source files in CsGui.

The following MinGW64 thread seems to have a good handle in what I'm seeing:

https://sourceforge.net/p/mingw-w64/mailman/mingw-w64-public/thread/5097CA60.9090000%40doxos.eu/

Ruben and Vaclav detail the problem this way:

<Ruben>
It seems you are running into a binutils "as" implementation limit. I think it might be related to the PE+ object format, which probably
cannot handle more than 2**15 sections (all the errors have numbers like 38k sections and such)? I've never ever seen this problem in Linux (ELF).

<Vaclav>
Ok, got it:
http://waleedassar.blogspot.cz/2012/04/ollydbg-numberofsections-crash.html (that mentions 65239 (0xFEFF) however);
there is link to MSDN (http://msdn.microsoft.com/en-us/windows/hardware/gg463119.aspx) as well.

<Vaclav>
It seems therefore that using -finline-functions for heavy-template code is the way to go - I imagine a new section is created for each template instantiation; since most of those functions are small and get only called one or twice (serialization code), they can be effectively inlined and need no additional sections afterwards.

<Vaclav>
It is just a wild guess that new section gets created for each template (I can't imagine why there would be 38k sections otherwise). The final link is not a problem, it is the intermediary .o file written by as.exe which is problematic.

I reran the single file compilation and watched /tmp seeing:

asahukar@Satchmo MSYS /tmp
$ ls -al
total 116M
drwxr-xr-x 1 asahukar None 0 Dec 18 20:42 ./
drwxr-xr-x 1 asahukar None 0 Dec 15 18:54 ../
-rw-r--r-- 1 asahukar None 116M Dec 18 20:40 ccCUrEe0.s

This, along with the error output below (qcolordialog - reformatted a bit) seems confirmatory to me.

What is a version of Cs I should pull to avoid this error?

Assuming the diagnosis offered makes sense to you, would you attack this problem by splitting the source as the above thread suggests or recommend I take some other action?

Thanks in advance for the help,

AJ

Error Details:

------------------------------>
From build.ninja >
------------------------------>

build src/gui/CMakeFiles/CsGui.dir/dialogs/qcolordialog.cpp.obj:
CXX_COMPILER__CsGui_Debug ../src/gui/dialogs/qcolordialog.cpp || cmake_object_order_depends_target_CsGui
DEFINES = -DCsGui_EXPORTS -DFT2_BUILD_LIBRARY -DHAVE_ATEXIT -DHB_EXTERN="" -DHB_NDEBUG
-DHB_NO_UNICODE_FUNCS -DMNG_BUILD_SO -DMNG_NO_INCLUDE_JNG -DPNG_NO_CONFIG_H
-DQT_BUILD_GUI_LIB -DQT_BUILTIN_GIF_READER -DQT_NO_CUPS -DQT_NO_DIRECTDRAW
-DQT_NO_GLIB -DQT_NO_LPR -DQT_NO_STYLE_GTK -DQT_NO_STYLE_MAC -DQT_SHARED
-DQT_USE_FREETYPE -DUNICODE
DEP_FILE = src\gui\CMakeFiles\CsGui.dir\dialogs\qcolordialog.cpp.obj.d
FLAGS = -Wa,-mbig-obj -g -std=gnu++1z
INCLUDES = -Isrc/gui -I../src/gui -I../src/3rdparty/libjpeg -I../src/3rdparty/libpng -I../src/3rdparty/libmng
-I../src/3rdparty/libtiff/libtiff -I../src/3rdparty/zlib -I../src/3rdparty/harfbuzz/src
-I../src/3rdparty/freetype/include/freetype -I../src/3rdparty/freetype/include -I../src/gui/widgets
-I../src/gui/util -I../src/gui/text -I../src/gui/styles -I../src/gui/statemachine -I../src/gui/printing
-I../src/gui/platform -I../src/gui/painting -I../src/gui/math3d -I../src/gui/kernel
-I../src/gui/itemviews -I../src/gui/image -I../src/gui/graphicsview -I../src/gui/effects
-I../src/gui/dialogs -I../src/gui/animation -I../src/gui/accessible -Iprivateinclude/QtGui/private
-Iinclude/QtGui -Iprivateinclude/QtCore/private -Iinclude/QtCore -Iprivateinclude -Iinclude
OBJECT_DIR = src\gui\CMakeFiles\CsGui.dir
OBJECT_FILE_DIR = src\gui\CMakeFiles\CsGui.dir\dialogs
RSP_FILE = src\gui\CMakeFiles\CsGui.dir\dialogs\qcolordialog.cpp.obj.rsp

------------------------------>
From Ninja output >
------------------------------>

[584/3850] Building CXX object src/gui/CMakeFiles/CsGui.dir/dialogs/qcolordialog.cpp.obj
FAILED: src/gui/CMakeFiles/CsGui.dir/dialogs/qcolordialog.cpp.obj

D:\DevApps\MinGW64\7.3\bin\c++.exe @src\gui\CMakeFiles\CsGui.dir\dialogs\qcolordialog.cpp.obj.rsp
-MD -MT src/gui/CMakeFiles/CsGui.dir/dialogs/qcolordialog.cpp.obj
-MF src\gui\CMakeFiles\CsGui.dir\dialogs\qcolordialog.cpp.obj.d
-o src/gui/CMakeFiles/CsGui.dir/dialogs/qcolordialog.cpp.obj
-c ../src/gui/dialogs/qcolordialog.cpp

D:/DevApps/MinGW64/7.3/bin/../lib/gcc/x86_64-w64-mingw32/7.3.0/../../../../x86_64-w64-mingw32/bin/as.exe:
src/gui/CMakeFiles/CsGui.dir/dialogs/qcolordialog.cpp.obj:
section .pdata$_ZNSt8__detail9__variant15_Variadic_unionIJPvSt10shared_ptrIN8QVariant10CustomTypeEEEEC1ILy1EJS3_I12CustomType_TIN7QAction8MenuRoleEEEEEESt16in_place_index_tIXT_EEDpOT0_: string table overflow at offset 10000004
D:\DevApps\MSys2\tmp\ccRRVbg4.s: Assembler messages:
D:\DevApps\MSys2\tmp\ccRRVbg4.s: Fatal error: can't close src/gui/CMakeFiles/CsGui.dir/dialogs/qcolordialog.cpp.obj: File too big

------------------------------------>
From qcolordialog.cpp.obj.rsp >
------------------------------------>

-DCsGui_EXPORTS -DFT2_BUILD_LIBRARY -DHAVE_ATEXIT -DHB_EXTERN="" -DHB_NDEBUG
-DHB_NO_UNICODE_FUNCS -DMNG_BUILD_SO -DMNG_NO_INCLUDE_JNG -DPNG_NO_CONFIG_H
-DQT_BUILD_GUI_LIB -DQT_BUILTIN_GIF_READER -DQT_NO_CUPS -DQT_NO_DIRECTDRAW
-DQT_NO_GLIB -DQT_NO_LPR -DQT_NO_STYLE_GTK -DQT_NO_STYLE_MAC -DQT_SHARED
-DQT_USE_FREETYPE -DUNICODE

-Isrc/gui -I../src/gui -I../src/3rdparty/libjpeg -I../src/3rdparty/libpng -I../src/3rdparty/libmng
-I../src/3rdparty/libtiff/libtiff -I../src/3rdparty/zlib -I../src/3rdparty/harfbuzz/src
-I../src/3rdparty/freetype/include/freetype -I../src/3rdparty/freetype/include -I../src/gui/widgets
-I../src/gui/util -I../src/gui/text -I../src/gui/styles -I../src/gui/statemachine -I../src/gui/printing
-I../src/gui/platform -I../src/gui/painting -I../src/gui/math3d -I../src/gui/kernel
-I../src/gui/itemviews -I../src/gui/image -I../src/gui/graphicsview -I../src/gui/effects
-I../src/gui/dialogs -I../src/gui/animation -I../src/gui/accessible -Iprivateinclude/QtGui/private
-Iinclude/QtGui -Iprivateinclude/QtCore/private -Iinclude/QtCore -Iprivateinclude -Iinclude

-Wa,-mbig-obj -g -std=gnu++1z

ansel
Posts: 116
Joined: Fri Apr 10 2015 8:23 am

Re: GNU Assembler error building CsGui from 11.28 pull

Post by ansel »

asahukar wrote:
Sat Dec 19 2020 3:10 pm
I have run into what I believe is a GNU assembler error building a number of the source files in CsGui.
This issue occurs when "-Wa,-mbigobj" is not enabled. Our CMake files detect a windows non-MSVC build and turn this option on automatically.

What platform are you building on, and what platform are you building for? Is this a 32-bit or 64-bit build? Which compiler and which version are you using? If you are on Windows, are you using MSys or MSys2?
Ansel Sermersheim
CopperSpice Cofounder

asahukar
Posts: 5
Joined: Sat Dec 19 2020 12:25 am

Re: GNU Assembler error building CsGui from 11.28 pull

Post by asahukar »

This is being built on Windows, using MinGW64 in the Debug configuration, target is 64-bit Windows. Release builds fine.
- The MinGW64 version is 7.3, pulled from the repository mentioned in your documentation.
- I am building on top of MSys2 as that is my current dev platform
- To conform completely with your instructions, I have a side build area based on MSys I'm setting up to have equivalent configuration for my dev environment

As you almost certainly know, CMake generates the -Wa,-mbig-obj option set in the toplevel CMakeLists.txt, lines 149:153 in the 11.28 pull.

I added a CMake message to ensure it is being set as well as:
- Checked the FLAGS variable in the build.ninja file (first listing) -- found it is there in both places.
- Checked The qcolordialog.cpp.obj.rsp file (last listing) -- found -Wa,-mbig-obj is being used on the compile line to pass through to the assembler.

I notice in your response that you indicate -Wa,-mbigobj (no '-') should be specified, so I tried editing that in the CMakeLists.txt file and found that -mbig-obj is the proper option (assembler doesn't recognize -mbigobj).

D:\DevApps\MinGW64\7.3\bin\c++.exe @src\core\CMakeFiles\CsCore.dir\kernel\qabstractitemmodel.cpp.obj.rsp -MD -MT src/core/CMakeFiles/CsCore.dir/kernel/qabstractitemmodel.cpp.obj -MF src\core\CMakeFiles\CsCore.dir\kernel\qabstractitemmodel.cpp.obj.d -o src/core/CMakeFiles/CsCore.dir/kernel/qabstractitemmodel.cpp.obj -c ../src/core/kernel/qabstractitemmodel.cpp
D:/DevApps/MinGW64/7.3/bin/../lib/gcc/x86_64-w64-mingw32/7.3.0/../../../../x86_64-w64-mingw32/bin/as.exe: unknown option -- mbigobj

I have spent some time yesterday narrowing the problem area, in the hopes using MSys2 is even sane. If QColSpinBox is split into its own header and QColorLuminancePicker,QColorPicker, QColorShower (and QColorShowLabel), QColorWell (and QWellArray) are split out into separate headers and translation units, they compile and assemble to objects.

The leaves QColorDlalog and QColorDialogPrivate which do not.

I split those two into separate header and translation units and generated a source + assembly input listing with the following command for the QDialogPrivate portion:

/d/DevApps/MinGW64/7.3/bin/g++.exe @src/gui/CMakeFiles/CsGui.dir/dialogs/qcolordialog_p.cpp.obj.rsp -MD -MT src/gui/CMakeFiles/CsGui.dir/dialogs/qcolordialog_p.cpp.obj -c ../src/gui/dialogs/qcolordialog_p.cpp > qcolordialog_p.lst
D:/DevApps/MinGW64/7.3/bin/../lib/gcc/x86_64-w64-mingw32/7.3.0/../../../../x86_64-w64-mingw32/bin/as.exe: qcolordialog_p.o: section .debug_frame$_ZNSt10unique_ptrIN8CsSignal8Internal5BentoIM7QWindowFvdEEESt14default_deleteIS6_EEC1IS8_vEEPS6_: string table overflow at offset 10000034
D:\DevApps\MSys2\tmp\ccClEO8O.s: Assembler messages:
D:\DevApps\MSys2\tmp\ccClEO8O.s: Fatal error: can't close qcolordialog_p.o: File too big

The .rsp file adds the flags -Wa,-a,-ad.

The resultant qcolordialog_p.lst is 237 MBytes in size and has some interesting content, but I have no practical experience with which to interpret it.

It can be uploaded to you along with my updated source if you wish, although I can imagine and accept one reasonable proper conclusion is that MSys2 is not a viable platform.
Last edited by asahukar on Sun Dec 20 2020 2:58 pm, edited 1 time in total.

ansel
Posts: 116
Joined: Fri Apr 10 2015 8:23 am

Re: GNU Assembler error building CsGui from 11.28 pull

Post by ansel »

Thanks for sending the additional information.

Our "CopperSpice Overview" documentation for building on Windows MinGW uses MSys and not MSys2. I can confirm when building CsGui there are a few large files and you must enable "big object files" as we have done in our CMake files.

Looking at your error message I see it says "string table overflow". You also mention you are doing a Debug build. Please try adding -O1 or -O2 to your compile options since it may reduce the size of the string table.
Ansel Sermersheim
CopperSpice Cofounder

asahukar
Posts: 5
Joined: Sat Dec 19 2020 12:25 am

Re: GNU Assembler error building CsGui from 11.28 pull

Post by asahukar »

I appreciate your comment and you are quite correct, the instructions do specify MSys.

In an effort to follow your instructions to the letter, I have:
- Pulled the MSys image and the MinGW64 image from the recommended sources on your instruction page.
- I am using your CMake (v3.19.1) with the command line:

cmake -G "Ninja" -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=/e/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/build/msys_mingw64-v73_debug /e/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice

- I have verified the -Wa,-mbig-obj directive is specified on the compiler command line (checked .rsp file - see below)

While the Release configuration builds successfully, the Debug fails in a similar manner as reported earlier.

For example:

[624/3850] Building CXX object src/gui/CMakeFiles/CsGui.dir/kernel/qgesturemanager.cpp.obj
FAILED: src/gui/CMakeFiles/CsGui.dir/kernel/qgesturemanager.cpp.obj
D:\DevApps\MinGW64\7.3\bin\c++.exe @src\gui\CMakeFiles\CsGui.dir\kernel\qgesturemanager.cpp.obj.rsp -MD -MT src/gui/CMakeFiles/CsGui.dir/kernel/qgesturemanager.cpp.obj -MF src\gui\CMakeFiles\CsGui.dir\kernel\qgesturemanager.cpp.obj.d -o src/gui/CMakeFiles/CsGui.dir/kernel/qgesturemanager.cpp.obj -c E:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/kernel/qgesturemanager.cpp
D:/DevApps/MinGW64/7.3/bin/../lib/gcc/x86_64-w64-mingw32/7.3.0/../../../../x86_64-w64-mingw32/bin/as.exe: src/gui/CMakeFiles/CsGui.dir/kernel/qgesturemanager.cpp.obj: section .debug_frame$_ZNSt10_Head_baseILy0EPN8CsSignal8Internal5BentoIM7QWindowFvN2Qt14WindowModalityEEEELb0EE7_M_headERSA_: string table overflow at offset 10000093
C:\Users\asahukar\AppData\Local\Temp\ccCdxR6X.s: Assembler messages:
C:\Users\asahukar\AppData\Local\Temp\ccCdxR6X.s: Fatal error: can't close src/gui/CMakeFiles/CsGui.dir/kernel/qgesturemanager.cpp.obj: File too big

qgesturemanager.cpp.obj.rsp is:

-DCsGui_EXPORTS -DFT2_BUILD_LIBRARY -DHAVE_ATEXIT -DHB_EXTERN="" -DHB_NDEBUG -DHB_NO_UNICODE_FUNCS -DMNG_BUILD_SO -DMNG_NO_INCLUDE_JNG -DPNG_NO_CONFIG_H -DQT_BUILD_GUI_LIB -DQT_BUILTIN_GIF_READER -DQT_NO_CUPS -DQT_NO_DIRECTDRAW -DQT_NO_GLIB -DQT_NO_LPR -DQT_NO_STYLE_GTK -DQT_NO_STYLE_MAC -DQT_SHARED -DQT_USE_FREETYPE -DUNICODE
-Isrc/gui -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/3rdparty/libjpeg -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/3rdparty/libpng -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/3rdparty/libmng -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/3rdparty/libtiff/libtiff -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/3rdparty/zlib -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/3rdparty/harfbuzz/src -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/3rdparty/freetype/include/freetype -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/3rdparty/freetype/include -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/widgets -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/util -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/text -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/styles -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/statemachine -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/printing -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/platform -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/painting -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/math3d -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/kernel -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/itemviews -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/image -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/graphicsview -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/effects -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/dialogs -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/animation -IE:/DevRoot/WS_ROOT/GitHub/CopperSpice/copperspice/src/gui/accessible -Iprivateinclude/QtGui/private -Iinclude/QtGui -Iprivateinclude/QtCore/private -Iinclude/QtCore -Iprivateinclude -Iinclude -Wa,-mbig-obj -g -std=gnu++1z

I did notice that the portion of your TestLargeFiles.cmake that checks for large file support fails on its off_t check, so I took a look into that, modifying your test program a little bit.

$ cat large_file_test.cxx
#include <sys/types.h>
#include <cstdio>

int main(int argc, char **argv)
{
printf("sizeof off_t : %i bytes\n", sizeof(off_t));

/* Cause a compile-time error if off_t is smaller than 64 bits */
#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
int off_t_is_large[ (LARGE_OFF_T % 2147483629 == 721 && LARGE_OFF_T % 2147483647 == 1) ? 1 : -1 ];
return 0;
}

This fails to compile as I expected:

$ g++ -Wa,-mbig-obj -std=gnu++1z -c large_file_test.cxx
large_file_test.cxx: In function 'int main(int, char**)':
large_file_test.cxx:9:40: warning: left shift count >= width of type [-Wshift-count-overflow]
#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
^
large_file_test.cxx:10:26: note: in expansion of macro 'LARGE_OFF_T'
int off_t_is_large[ (LARGE_OFF_T % 2147483629 == 721 && LARGE_OFF_T % 2147483647 == 1) ? 1 : -1 ];
^~~~~~~~~~~
large_file_test.cxx:9:64: warning: left shift count >= width of type [-Wshift-count-overflow]
#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
^
large_file_test.cxx:10:26: note: in expansion of macro 'LARGE_OFF_T'
int off_t_is_large[ (LARGE_OFF_T % 2147483629 == 721 && LARGE_OFF_T % 2147483647 == 1) ? 1 : -1 ];
^~~~~~~~~~~
large_file_test.cxx:10:101: error: size of array 'off_t_is_large' is negative
int off_t_is_large[ (LARGE_OFF_T % 2147483629 == 721 && LARGE_OFF_T % 2147483647 == 1) ? 1 : -1 ];

If I provide _FILE_OFFSET_BITS=64, I get:

$ g++ -D_FILE_OFFSET_BITS=64 -Wa,-mbig-obj -std=gnu++1z -c large_file_test.cxx
$ g++ large_file_test.o -o aj_FILE_OFFSET_BITS.exe
$ ./aj_FILE_OFFSET_BITS.exe
sizeof off_t : 8 bytes

Interestingly, I don't see the _FILE_OFFSET_BITS=64 in my .rsp file -- it would seem it is necessary.

Is the CMake configuration working as you expect?

Are the path lengths perhaps bloating the size of the string table such that shorter paths might help?

ansel
Posts: 116
Joined: Fri Apr 10 2015 8:23 am

Re: GNU Assembler error building CsGui from 11.28 pull

Post by ansel »

You have listed a couple of possible reasons the string table may be too big. The space available in the string table is limited to 10 MB in the windows object file format.

Using a shorter path may help, as well as turning on some level of optimization to reduce the number of symbols and template instantiations overall. There are also a number of command line options in GCC to reduce the debugging symbols which are produced, you might want to try adding something like "-g1".
Ansel Sermersheim
CopperSpice Cofounder

asahukar
Posts: 5
Joined: Sat Dec 19 2020 12:25 am

Re: GNU Assembler error building CsGui from 11.28 pull

Post by asahukar »

I have found two things of consequence.

1. The large file test script in CMake (TestLargeFiles.cmake) does not function as expected.
2. Using the -g0 optimization is effective

With these two changes in combination, I was able to generate debug libraries. I did not test them separately ... builds for debug take 5 hours just for the CsGui library.

With respect to [1], I do have a change I can suggest:
1. Unsetting the test variable FILE64_OK in the cache prior to calling check_cxx_source_compiles as check_cxx_source_compiles skips its check when the variable exists.
2. Adding the lines to the TestFileOffsetBits tests for viable compilation of fseeko/ftello:

off_t x = ftello(nullptr);
int fseeko_res = fseeko(nullptr, x, SEEK_CUR);

With [1], I also found I had to use the set(CMAKE_REQUIRED_DEFINITIONS "-D_FILE_OFFSET_BITS=64") command prior to the call to check_cxx_source_compiles because once FILE64_OK did not exist, CMake throws an error with the COMPILE_DEFINITIONS argument.

I would be happy to offer these changes via a pull request. However, I don't know your policy and in addition, as I am very new to both you and CMake, I can imagine the intent of my changes may be fine, but the implementation may be naive.

I also have small changes for CMakeLists.txt specifying '-0g' for the Debug build type.

Post Reply