CMake: how create a single shared library from all static libraries of subprojects?
CmakeShared LibrariesCmake Problem Overview
I have the following layout:
top_project
+ subproject1
+ subproject2
Each of subproject1 and subproject2 creates a static library. I would like to link these static libraries in a single shared library at the top_project level.
The information I gathered so far is:
- Either compile using
-fPic(necessary on everything but Windows) in order to create position-independent code which will allow linking the static libraries into a single shared library or decompress all static libraries (e.g. usingar) and re-link them into a shared library (which I think is an inelegant & non-portable solution) - All source files must be given explicitly to the
add_librarycommand: for some reason which I cannot comprehend, simply writingadd_library(${PROJECT_NAME} SHARED subproject1 subproject2)does not work as expected (it essentially creates an empty library & does not register the dependencies properly) - There is an OBJECT library feature in CMake but I don't think it's purpose is really to do what I want.
Any thoughts?
Cmake Solutions
Solution 1 - Cmake
OK, I figured it out: this is much more painful than it should be. Until very recently, people at Kitware didn't understand why anyone would ever want to create a DLL from static libs. Their argument is that there should always be source files in the main (e.g. top_project in my case) directory because it is effectively a project of its own. I see things differently & I need to break top_project into smaller subprojects which should not exist independently (i.e. there is no point in creating a full-blown project for them & add them using ExternalProject_Add). Besides, when I ship my shared library (for use, e.g. with a Java Native Interface), I don't want to ship dozens of shared libraries because that would amount to exposing the internal layout of my project. Anyway, having - I think - made a case for creating a shared library from static libraries, I'll proceed to the technical details.
In the CMakeLists.txt of subproject1 and subproject2, you should create your target using the OBJECT library feature (introduced in CMake 2.8.8):
add_library(${PROJECT_NAME} OBJECT ${SRC})
where SRC designates the list of source files (note that these should be set explicitly in the CMakeLists.txt file as it allows make to re-launch CMake when a modification of CMakeLists.txt is detected, e.g. when adding or removing a file)
In the top_project, add the subprojects using:
add_subdirectory(subproject1)
add_subdirectory(subproject2)
In order to see the symbols from the static library, use:
set(CMAKE_SHARED_LINKER_FLAGS "-Wl,--export-all-symbols")
You can then create the shared library using:
add_library(${PROJECT_NAME} SHARED $<TARGET_OBJECTS:subproject1>
$<TARGET_OBJECTS:subproject2>)
I've found that any "normal" library (i.e. not object) needs to be added in a separate add_library command, otherwise it is simply ignored.
For executables, you can use:
add_executable(name_of_executable $<TARGET_OBJECTS:subproject1>
$<TARGET_OBJECTS:subproject2>)
set(LINK_FLAGS ${LINK_FLAGS} "-Wl,-whole-archive")
target_link_libraries(name_of_executable ${PROJECT_NAME}
I repeat that this only works as of version 2.8.8 of CMake. Just as well CMake manages the dependencies extremely well & is cross-platform because it's not much less painful than plain old Makefiles & certainly less flexible.
Solution 2 - Cmake
My solution is simply to add /WHOLEARCHIVE, -all_load, or --whole-archive to the linker flags, so that when your main library is linked, all of the sub libraries are included, including all their symbols (the default behaviour is to only include symbols of the sub libraries that are used by the main library. For example:
Source Files
$ echo "void Func1() { }" > source1.cpp
$ echo "void Func2() { }" > source2.cpp
$ echo "void Func3() { }" > source3.cpp
$ echo "void Func4() { }" > source4.cpp
Naive CMakeLists.txt
cmake_minimum_required(VERSION 3.7)
# The 'sub' libraries, e.g. from an `add_subdirectory()` call.
add_library(sublib_a STATIC source1.cpp source2.cpp)
add_library(sublib_b STATIC source3.cpp source4.cpp)
# The main library that contains all of the sub libraries.
add_library(mainlib SHARED)
target_link_libraries(mainlib sublib_a sublib_b)
Running it (on OSX):
$ make VERBOSE=1
...
[100%] Linking CXX shared library libmainlib.dylib
/usr/local/Cellar/cmake/3.7.1/bin/cmake -E cmake_link_script CMakeFiles/mainlib.dir/link.txt --verbose=1
/Library/Developer/CommandLineTools/usr/bin/c++ -dynamiclib -Wl,-headerpad_max_install_names -o libmainlib.dylib -install_name @rpath/libmainlib.dylib libsublib_a.a libsublib_b.a
[100%] Built target mainlib
$ nm libmainlib.dylib | grep Func
$
Correct CMakeLists.txt
Append this:
# By default, symbols provided by the sublibs that are not used by mainlib (which is all of them in this case)
# are not used. This changes that.
if (WIN32)
set_target_properties(mainlib PROPERTIES
LINK_FLAGS "/WHOLEARCHIVE"
)
elseif (APPLE)
set_target_properties(mainlib PROPERTIES
LINK_FLAGS "-Wl,-all_load"
)
else ()
set_target_properties(mainlib PROPERTIES
LINK_FLAGS "-Wl,--whole-archive"
)
endif ()
Running it (note the extra -all_load):
$ make VERBOSE=1
[100%] Linking CXX shared library libmainlib.dylib
/usr/local/Cellar/cmake/3.7.1/bin/cmake -E cmake_link_script CMakeFiles/mainlib.dir/link.txt --verbose=1
/Library/Developer/CommandLineTools/usr/bin/c++ -dynamiclib -Wl,-headerpad_max_install_names -Wl,-all_load -o libmainlib.dylib -install_name @rpath/libmainlib.dylib libsublib_a.a libsublib_b.a
[100%] Built target mainlib
$ nm libmainlib.dylib | grep Func
0000000000001da0 T __Z5Func1v
0000000000001db0 T __Z5Func2v
0000000000001dc0 T __Z5Func3v
0000000000001dd0 T __Z5Func4v
Note that I've only actually tested -all_load so far, and /WHOLEARCHIVE is an MSVC 2015 option.
Solution 3 - Cmake
Another way of doing it.
This way seems simpler, but I'm not sure how perfect it is:
Solution 4 - Cmake
Another way of doing it is to provide the path of the source files and the header files of all your projects, and build them together to produce the .so . This is usually the recommended way, instead of creating the static libraries and then a shared library out of those.
Basically you should do the following:
FILE(GLOB subproject1_sources
<sub_project1_lib_sources_dir>/file1.c
<sub_project1_lib_sources_dir>/file2.c //... etc
)
FILE(GLOB subproject2_sources
<sub_project2_lib_sources_dir>/file1.c
<sub_project2_lib_sources_dir>/file2.c //... etc
)
FILE(GLOB topProject_sources
<top_project_lib_sources_dir>/file1.c
<top_project_lib_sources_dir>/file2.c //... etc
)
include_directories("<sub_project1_lib_sources_dir>")
include_directories("<sub_project2_lib_sources_dir>")
include_directories("<top_project_lib_sources_dir>") //should be "." if you're building from here
add_library(topProject SHARED ${topProject_sources} ${subproject1_sources} ${subproject2_sources})
Solution 5 - Cmake
I am not sure if this is what suits your need, but cmake also offers INTERFACE libraries, which serve (among others) precisely this need.
add_library(bundle INTERFACE)
target_link_libraries(bundle lib1 lib2)
will bundle lib1 and lib2 into a single library, and inherit the PUBLIC and INTERFACE section of lib1 and lib2.
More info here.
Solution 6 - Cmake
Add following macro to your cmake scripts.
MACRO (TARGET_LINK_LIBRARIES_WHOLE_ARCHIVE target)
IF (WIN32)
FOREACH (arg IN LISTS ARGN)
SET_TARGET_PROPERTIES(
${target} PROPERTIES LINK_FLAGS "/WHOLEARCHIVE:${lib}"
)
ENDFOREACH ()
ELSE ()
IF (APPLE)
SET(LINK_FLAGS "-Wl,-all_load")
SET(UNDO_FLAGS "-Wl,-noall_load")
ELSE ()
SET(LINK_FLAGS "-Wl,--whole-archive")
SET(UNDO_FLAGS "-Wl,--no-whole-archive")
ENDIF ()
TARGET_LINK_LIBRARIES(${target} ${LINK_FLAGS} ${ARGN} ${UNDO_FLAGS})
ENDIF ()
ENDMACRO ()
Then, in CMakeLists.txt/*.cmake, you can use like this TARGET_LINK_LIBRARIES_WHOLE_ARCHIVE(target libs...)