Most C compilers allow you to access an array externwith undefined boundaries, for example:

extern int external_array[];
int
array_get (long int index)
{
  return external_array[index];
}

The definition of external_array may be in another translation unit and may look like this:

int external_array[3] = { 1, 2, 3 };

The question is what happens if this separate definition changes like this:

int external_array[4] = { 1, 2, 3, 4 };

Or so:

int external_array[2] = { 1, 2 };

Will the binary interface be preserved (provided that there is a mechanism that allows the application to determine the size of the array at run time)?

Curiously, on many architectures, increasing the size of the array violates binary interface compatibility (ABI). Reducing the size of the array can also cause compatibility issues. In this article, we will take a closer look at ABI compatibility and explain how to avoid problems.

Links in the data section of the executable file

To understand how the size of the array becomes part of the binary interface, we first need to examine the links in the data section of the executable file. Of course, the details depend on the specific architecture, and here we will focus on the x86-64 architecture.

The x86-64 architecture supports addressing relative to the program counter, that is, access to the global array variable, as in the function shown above array_get, can be compiled into one instruction movl:

array_get:
	movl	external_array(,%rdi,4), %eax
	ret

From this, the assembler creates an object file in which the instruction is marked as R_X86_64_32S.

0000000000000000 :
   0:	mov    0x0(,%rdi,4),%eax
			3: R_X86_64_32S	external_array
   7:	retq   

This move tells the linker ( ld) how to populate the corresponding variable location external_arrayduring linking when creating the executable.

This has two important consequences.

• Since the offset of the variable is determined at build time, at run time there is no overhead to determine it. The only price is access to memory itself.
  
• To determine the offset, you need to know the sizes of all the variable data. Otherwise, it would be impossible to calculate the format of the data section during layout.

For C implementations targeting the Executable and Link Format (ELF) , as in GNU / Linux, variable references externdo not contain object sizes. In the example, the array_getsize of the object is unknown even to the compiler. In fact, the entire assembler file looks like this (omitting only the promotion information from c -fno-asynchronous-unwind-tables, which is technically required for psABI compliance):

	.file	"get.c"
	.text
	.p2align 4,,15
	.globl	array_get
	.type	array_get, @function
array_get:
	movl	external_array(,%rdi,4), %eax
	ret
	.size	array_get, .-array_get
	.ident	"GCC: (GNU) 8.3.1 20190223 (Red Hat 8.3.1-2)"
	.section	.note.GNU-stack,"",@progbits

In this assembler file there is generally no information about the size for external_array: the only reference to the symbol is in the line with the instruction movl, and the only numerical data in the instruction is the size of the array element (implied movlby multiplying by 4).

If ELF requires dimensions for undefined variables, then it will even be impossible to compile the function array_get.

How does the linker get the actual character size? He looks at the definition of the symbol and uses the size information that he finds there. This allows the compiler to compute the layout of the data section and fill the data movements with the appropriate offsets.

Common ELF Objects

C implementations for ELF do not require the programmer to add source code markup to indicate whether a function or variable is located in the current object (which may be the library or the main executable) or in another object. The linker and the dynamic loader will take care of this.

At the same time, there was a desire for executable files not to reduce performance by changing the compilation model. This means that when compiling the source code for the main program (that is, without -fPIC, and in this particular case, without -fPIE) the function is array_getcompiled into exactly the same sequence of commands before introducing dynamic shared objects. Also, it doesn't matter if the variable is definedexternal_arrayin the most basic executable file or some shared object is loaded separately at run time. The instructions created by the compiler are the same in both cases.

How is this possible? After all, common ELF objects are position-independent. They are loaded at unpredictable, randomized addresses at runtime. However, the compiler generates a machine code sequence that requires these variables to be located at a fixed offset calculated during linking , long before the program starts.

The fact is that only one loaded object (the main executable file) uses these fixed offsets. All other objects (the dynamic loader itself, C runtime library and any other library used by the program) are compiled and compiled as completely position-independent objects (PICs). For such objects, the compiler loads the actual address of each variable from the global offset table (GOT). We can see this roundabout if we compile an example array_getwith -fPIC, which leads to such assembler code:

array_get:
	movq	external_array@GOTPCREL(%rip), %rax
	movl	(%rax,%rdi,4), %eax
	ret

As a result, the address of the variable is external_arrayno longer hardcoded and can be changed at run time by appropriately initializing the GOT record. This means that at run time, the definition external_arraymay be in the same shared object, another shared object, or the main program. The dynamic loader will find the appropriate definition based on the ELF character search rules and associate the undefined symbol reference with its definition by updating the GOT record to its actual address.

We return to the original example, where the function array_getis in the main program, so the address of the variable is specified directly. The key idea implemented in the linker is that the main program will provide a variable definition external_array,even if it is actually defined in a common object at runtime . Instead of specifying the initial definition of the variable in the shared object, the dynamic loader will select a copy of the variable in the data section of the executable file.

This has two important consequences. First of all, recall that it external_arrayis defined as follows:

int external_array[3] = { 1, 2, 3 };

There is an initializer here that should be applied to the definition in the main executable file. To do this, in the main executable file, a link to the copy copy location of the symbol is placed . The team readelf -rWshows it as a move R_X86_64_COPY.

Relocation section '.rela.dyn' at offset 0x408 contains 3 entries:
    Offset Info Type Symbol's Value Symbol's Name + Addend
0000000000403ff0 0000000100000006 R_X86_64_GLOB_DAT 0000000000000000 __libc_start_main@GLIBC_2.2.5 + 0
0000000000403ff8 0000000200000006 R_X86_64_GLOB_DAT 0000000000000000 __gmon_start__ + 0
0000000000404020 0000000300000005 R_X86_64_COPY 0000000000404020 external_array + 0

Like other movements, copy movement is handled by the dynamic loader. It includes a simple, bitwise copy operation. The target of the copy is determined by the displacement offset ( 0000000000404020in the example). The source is determined at runtime based on the symbol name ( external_array) and its value. When creating a copy, the dynamic loader will also look at the size of the character to get the number of bytes that need to be copied. To make all this possible, a symbol is external_arrayautomatically exported from the executable file as a specific symbol so that it is visible to the dynamic loader at run time. The dynamic symbol table ( .dynsym) reflects this, as shown by the command readelf -sW:

Symbol table '.dynsym' contains 4 entries:
   Num: Value Size Type Bind Vis Ndx Name
     0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND 
     1: 0000000000000000 0 FUNC GLOBAL DEFAULT UND __libc_start_main@GLIBC_2.2.5 (2)
     2: 0000000000000000 0 NOTYPE WEAK DEFAULT UND __gmon_start__
     3: 0000000000404020 12 OBJECT GLOBAL DEFAULT 22 external_array

Where does the information about the size of the object come from (12 bytes, in this example)? The linker opens all common objects, searches for its definition and takes information about the size. As before, this allows the linker to calculate the layout of the data section so that fixed offsets can be used. Again, the size of the definition in the main executable is fixed and cannot be changed at run time.

The dynamic linker also redirects symbolic links in shared objects to the moved copy in the main executable. This ensures that in the entire program there is only one copy of the variable, as required by the semantics of C. Otherwise, if the variable changes after initialization, updates from the main executable will not be visible to dynamic shared objects and vice versa.

Impact on binary compatibility

What happens if we change the definition external_arrayin a common object without linking (or recompiling) the main program? First, consider adding an array element.

int external_array[4] = { 1, 2, 3, 4 };

This will give a warning from the dynamic loader in runtime:

main-program: Symbol `external_array' has different size in shared object, consider re-linking

The main program still contains a definition external_arraywith space for only 12 bytes. This means that the copy is incomplete: only the first three elements of the array are copied. As a result, access to the array element is extern_array[3]not defined. This approach affects not only the main program, but also the entire code in the process, because all references to extern_arraywere redirected to the definition in the main program. This includes a generic object that provides a definition extern_array. He is probably not ready to face a situation where an array element in his own definition has disappeared.

How about changing in the opposite direction, removing an element?

int external_array[2] = { 1, 2 };

If the program avoids access to the array element extern_array[2], because it somehow detects the reduced length of the array, then this will work. After the array, there is some unused memory, but this will not break the program.

This means that we get the following rule:

• Adding elements to a global array variable violates binary compatibility.
  
• Removing items may break compatibility if there is no mechanism that avoids access to deleted items.

Unfortunately, the warning of the dynamic loader looks more harmless than it actually is, and for remote elements there is no warning at all.

How to avoid this situation

Detecting ABI changes is fairly easy with tools like libabigail .

The easiest way to avoid this situation is to implement a function that returns the address of the array:

static int local_array[3] = { 1, 2, 3 };
int *
get_external_array (void)
{
  return local_array;
}

If the definition of the array cannot be made static due to the way it is used in the library, instead we can hide its visibility and also prevent its export and, therefore, avoid the truncation problem:

int local_array[3] __attribute__ ((visibility ("hidden"))) =
  { 1, 2, 3 };

Everything is much more complicated if the array variable is exported for backward compatibility reasons. Since the array from the library is truncated, the old main program with a shorter array definition will not be able to provide access to the full array for the new client code if it is used with the same global array. Instead, the access function may use a separate (static or hidden) array, or perhaps a separate array for added elements at the end. The disadvantage is that it is not possible to store everything in a continuous array if the array variable is exported for backward compatibility. The design of the secondary interface should reflect this.

Using version control of characters, you can export multiple versions with different sizes, never changing the size in a particular version. Using this model, new related programs will always use the latest version, presumably with the largest size. Since the version and size of the symbol are fixed by the link editor at the same time, they are always consistent. The GNU C library uses this approach for historical variables sys_errlistand sys_siglist. However, this still does not provide a single continuous array.

All things considered, an access function (like the function get_external_arrayabove) is the best approach to avoid this ABI compatibility issue.