Linux Dynamic and Static Libraries

Basic principles of dynamic static libraries

A static library is essentially a “half-baked” executable.

We all know that a bunch of source files and headers eventually become an executable program needs to go through the following four steps.

• Preprocessing: complete header file expansion, de-commenting, macro replacement, conditional compilation, etc. to form the final xxx.i file.
• Compile: Complete lexical analysis, syntax analysis, semantic analysis, symbol summarization, etc. After checking for errors, translate the code into assembly instructions, and finally form the xxx.s file.
• Assembly: Convert the assembly instructions into binary instructions to form the xxx.o file.
• Linking: Link the generated xxx.o files to form an executable program.

For example, if test1.c, test2.c, test3.c, test4.c and main1.c are used to form an executable file, we need to get the target files test1.o, test2.o, test3.o, test4.o and main1.o for each file first, and then link these target files together to form an executable program.

If we also need to use test1.c, test2.c, test3.c, test4.c and the project’s main2.c or main3.c in another project to form the executable respectively, then the steps for executable generation are the same.

In fact, for the source files that may be used frequently, such as test1.c, test2.c, test3.c, test4.c, we can package their target files test1.o, test2.o, test3.o, test4.o, and then link the four target files among the package when we need to use them. The package can actually be called a library.

In fact, all libraries are essentially a collection of target files (xxx.o), the library files do not contain the main function but only a large number of methods for invocation, so that the static library is essentially a “semi-finished” executable.

Getting to know the dynamic static library

Create the file to write the following code under Linux and generate the executable.

#include <stdio.h>

int main()
{
printf(“hello world\n”); //library function
return 0;
}
This is the simplest code, and the result is hello world, as you know.

Let’s go through this simple code to get to know the dynamic static library

In this code we can output hello world by calling printf. The main reason is that the gcc compiler links the C standard library when it generates the executable.

Under Linux, we can view the library files that an executable depends on by using the ldd filename.

In this case, libc.so.6 is the library file that the executable depends on, and we can find out that libc.so.6 is actually just a soft link by using the ls command.

In fact, the source file libc-2.17.so is in the same directory as libc.so.6. To learn more, we can use the file filename command to see the file type of libc-2.17.so.

At this point we can see that libc-2.17.so is actually a shared target file library, or a dynamic library to be exact.

In Linux, the .so suffix is a dynamic library, and the .a suffix is a static library.
In Windows, the .dll suffix is a dynamic library and the .lib suffix is a static library.
The libc.so.6 that the executable relies on is actually a C dynamic library. When we remove the prefix lib from a dynamic static library, and then remove the suffix .so or .a and the version number after it, what remains is the name of the library.

And gcc/g++ compilers are all dynamically linked by default, if you want to do static linking, you can carry a -static option.

[cl@VM-0-15-centos testlib]$ gcc -o mytest-s mytest.c -static

You can notice that the file size of the statically linked executable is much larger than that of the dynamically linked executable.

The statically linked executable does not depend on other library files, so when we use the ldd filename command to view the library files on which the executable depends, we see the following information.

In addition, when we look at the file types of the dynamic and static linked executables separately, we can also see that they are dynamically linked and statically linked respectively.

Features of static and dynamic libraries

Static libraries

Static library is a library code copied into the executable file when the program is compiled and linked, the generated executable will not need the static library when running, so the size of the executable program generated by using static library is generally larger.

Advantages.

Once an executable is generated using a static library, the executable can run on its own and no longer needs the library.
Disadvantages.

Using static libraries to generate executable programs can take up a lot of space, especially when there are multiple static programs loaded at the same time and they all use the same library, there will be a lot of duplicate code in memory.
Dynamic Libraries

Dynamic libraries are linked to the corresponding dynamic library code only when the program is running, and multiple programs share the code that uses the library. An executable file linked to a dynamic library contains only a table of the entry addresses of the functions it uses, not the entire machine code of the target file where the external functions are located.

Before the executable starts running, the machine code of the external function is copied from the dynamic library on disk to memory by the operating system, a process called dynamic linking. Dynamic libraries are shared among multiple programs, saving disk space. The operating system uses a virtual memory mechanism to allow a copy of a dynamic library in physical memory to be shared by all processes that want to use it, saving memory and disk space.

Advantages.

Saves disk space, and when multiple programs using the same dynamic library are running at the same time, the library files are shared through the process address space and there is no duplicate code in memory.
Disadvantages.

Must depend on dynamic libraries, otherwise it will not run.
Static library packaging and use
To make it easier to understand, the following four files are used as examples to demonstrate the packaging and use of dynamic static libraries: two source files add.c and sub.c, and two header files add.h and sub.h.

The contents of add.h are as follows.

#pragma once

extern int my_add(int x, int y);

The contents of add.c are as follows.

#include “add.h”

int my_add(int x, int y)
{
return x + y;
}
The contents of sub.h are as follows.

#pragma once

extern int my_sub(int x, int y);

The contents of sub.c are as follows.

#include “sub.h”

int my_sub(int x, int y)
{
return x – y;
}
The code content is all very simple, so I won’t go into too much detail.

Packaging

We will use these four files to package a static library.

Step 1: Let all the source files generate the corresponding target files

Step 2: Use the ar command to package all the target files into a static library

The ar command is gnu’s archiving tool, which is often used to package target files as static libraries, we will use the -r option and -c option of the ar command to package them.

-r(replace): If the target file in the static library file has been updated, replace the old target file with the new one.
-c(create): Create the static library file.
[cl@VM-0-15-centos static]$ ar -rc libcal.a add.o sub.o

In addition, we can use the -t and -v options of the ar command to view the files in the static library.

-t: lists the files in the static library.
-v(verbose): shows detailed information.
[cl@VM-0-15-centos static]$ ar -tv libcal.a

Step 3: Organize the headers and the generated static libraries

When we give our libraries to others, we actually need to give them two folders, one folder under which we put a collection of header files, and another folder under which we put all the library files.

So here we can put the two header files add.h and sub.h into a directory named include, and the generated static library file libcal.a into a directory named lib, and then put both of these directories under mathlib, at which point we can give mathlib to others to use.

Using Makefile

Of course, we can write all the above commands into the Makefile, so that we can generate static libraries and organize the header and library files in one step, instead of knocking so many commands every time we regenerate, which also shows the power of Makefile.

After writing the Makefile, a single make can generate all the target files corresponding to the source files and then generate the static libraries.

A single make output organizes the headers and static libraries.

Using
Create the source file main.c and write the following simple program to try to use our packaged static library.

#include <stdio.h>
#include <add.h>

int main()
{
int x = 20;
int y = 10;
int z = my_add(x, y);
printf(“%d + %d = %d\n”, x, y, z);
return 0;
}
Now the only thing in that directory is main.c and the static library we just packaged.

Method 1: Using options

At this point, when you compile main.c with gcc to generate an executable, you need to carry three options.

-I: Specify the header file search path.
-L: Specify the path to search for library files.
-L: Specify which library under the library file path needs to be linked.
[cl@VM-0-15-centos project]$ gcc main.c -I./mathlib/include -L./mathlib/lib -lcal

At this point it will successfully use our own packaged library files and generate the executable.

A note.

Because the compiler does not know where the header file add.h is that you include, you need to specify the search path for the header file.
Since the header file add.h contains only the declaration of the my_add function and not the definition of the function, you also need to specify the search path of the library file you want to link.
In practice, there may be a large number of library files in the lib directory of the library file, so we need to specify which library under the path of the library file we need to link. The library file name is stripped of the prefix lib, then the suffix .so or .a and the version number that follows it, and the rest is the name of the library.
The -I, -L and -l options can be followed by a space or no space.
Method 2: Copy the header file and library file to the system path

Since the compiler can’t find our header and library files, we can just copy the header and library files to the system path.

[cl@VM-0-15-centos project]$ sudo cp mathlib/include/* /usr/include/
[cl@VM-0-15-centos project]$ sudo cp mathlib/lib/libcal.a /lib64/

Note that although the header and library files have been copied to the system path, when we compile main.c with gcc to generate the executable, we still need to specify which library under the library file path we need to link.

This is the only way to successfully use our own packaged library files and generate the executable.

Why didn’t we specify the library name when we compiled with gcc before?

Because we use gcc to compile C language, and gcc is used to compile C program, so gcc compile by default is looking for C library, but at this time we want to link which library compiler does not know, so we still need to use -l option to specify which library under the path of the library file need to link.

Extension.
In fact the process of copying our header and library files to the system path is the process of installing the libraries. However, it is not recommended to copy your own written header and library files to the system path, as doing so will pollute the system files.

Packaging and use of dynamic libraries

Packaging
The packaging of dynamic libraries is a little different than static libraries, but it is roughly the same, so let’s use these four files for packaging demonstration.

Step 1: Let all source files generate the corresponding target files

At this point, you need to carry the -fPIC option when generating target files from source files: -fPIC(position independent code)

-fPIC(position independent code): Generate position independent code.

To clarify.

-fPIC is used in the compilation stage to tell the compiler to generate position independent code, so that the generated code does not have absolute addresses, but uses relative addresses, so that the code can be loaded into any location in memory by the loader and executed correctly. This is exactly what is required by shared libraries, which are not at a fixed location in memory when they are loaded.
If the -fPIC option is not added, when loading a .so file segment, the data object referenced by the segment will need to be relocated, and this relocation will modify the contents of the segment, resulting in each process using the .so file segment generating a copy of the .so file segment in the kernel, and each copy will be different depending on where the .so file segment and the data segment are mapped. so file and data segment memory map.
The .so compiled without -fPIC is repositioned again at load time according to the location it was loaded to, because it contains the BBS location-independent code. If the .so file is shared by multiple applications, then they must each maintain a copy of the .so code (because .so is loaded by each program in a different location, obviously these relocated codes are also different and certainly cannot be shared).
We always use -fPIC to generate .so, but never -fPIC to generate .a. But .so can just as easily be compiled without the -fPIC option, except that such a .so must redirect all table entries when loaded into the address space of the user program.
Step 2: Package all target files as dynamic libraries using the -shared option

Unlike generating static libraries, we don’t have to use the ar command when generating dynamic libraries, we just use gcc’s -shared option.

[cl@VM-0-15-centos dynamic]$ gcc -shared -o libcal.so add.o sub.o

Step 3: Organize the headers and the generated dynamic libraries

As when generating static libraries, for the convenience of others, here we can put the two header files add.h and sub.h into a directory named include, and the generated dynamic library file libcal.so into a directory named lib, and then put both of them under mlib, at which point we can give mlib to others to use.

Using Makefile

Of course, to generate dynamic libraries, you can also write all the above commands into the Makefile, so that when we want to generate dynamic libraries and organize the header and library files, we can do it in one step.

After writing the Makefile, a single make can generate all the target files corresponding to the source files and then generate the dynamic libraries.

A single make output organizes the headers and dynamic libraries.

Using
Let’s use the same main.c we just used to demonstrate the use of dynamic libraries.

#include <stdio.h>
#include <add.h>

int main()
{
int x = 20;
int y = 10;
int z = my_add(x, y);
printf(“%d + %d = %d\n”, x, y, z);
return 0;
}
Now the only thing in that directory is main.c and the dynamic library we just packaged.

We can use the -I, -L and -l options to generate executable programs, or we can copy the header files and library files to the system directory first, and then use the -l option to specify the name of the library to be linked to generate executable programs.

In this case, when you compile main.c with gcc to generate an executable, you need to specify the header file search path with the -I option, the library file search path with the -L option, and finally the -l option to specify which library under the library file path needs to be linked.

[cl@VM-0-15-centos project]$ gcc main.c -I./mlib/include -L./mlib/lib -lcal

Unlike the use of static libraries, the executable we generate at this point does not run directly.

Note that we use the -I, -L, and -l options to tell the compiler where and who the header and library files are during compilation, but when the generated executable is generated it has no relationship with the compiler, and after the executable is run, the operating system cannot find the dynamic libraries that the executable depends on, so we can use the ldd command to check.

As you can see, the dynamic libraries that the executable depends on are not found at this time.

There are three ways to solve this problem.

Method 1: Copy the .so file to the system shared library path

Since the system can’t find our library file, we can copy the library file to the system shared library path directly, so that the system can find the corresponding library file.

[cl@VM-0-15-centos project]$ sudo cp mlib/lib/libcal.so /lib64

The executable can now run without any problems.

Method 2: Change LD_LIBRARY_PATH

LD_LIBRARY_PATH is the path that the program will search for when it runs the dynamic library, we just need to add the path of the dynamic library to the LD_LIBRARY_PATH environment variable.

[cl@VM-0-15-centos project]$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/cl/BasicIO/testlib/project/mlib/lib

At this point, we can look at the executable again with the ldd command and see that the system can now find the dynamic libraries that the executable depends on.

Now we can also run the executable normally.

Method 3: Configure /etc/ld.so.conf.d/

We can solve the problem by configuring /etc/ld.so.conf.d/. All the configuration files stored under the /etc/ld.so.conf.d/ path are .conf suffix configuration files, and the paths stored in these configuration files are all paths, and the system will automatically find all the configuration files under the /etc/ld.so.conf.d/ path. The system will automatically look for all the paths in the /etc/ld.so.conf.d/ path, and then it will look for the libraries you need in each path. If we put the path of our library file in that path, then when the executable runs, the system will be able to find our library file.

First, put the path to the directory where the library file is located into a file with a .conf suffix.

Then copy the .conf file to the /etc/ld.so.conf.d/ directory.

However, when we use the ltd command to view the executable, we find that the system still does not find the dynamic libraries that the executable depends on.

At this point, we need to update the configuration file with the ldconfig command, and then the system will be able to find the dynamic libraries that the executable depends on.

[cl@VM-0-15-centos project]$ sudo ldconfig

And we can now run the executable normally.