Build a Shark for iOS and OSX

Yes, such a headline is not very full of meaning, so I will give a few explanations for those who still think "go under the cut or not?".

Shark - a set of C ++ libraries of machine learning (Machine Learning), namely linear and nonlinear optimization, neural networks, learning with and without a teacher, evolutionary algorithms and much more. A more detailed description can be found on the project website .

If you are interested in answers to the following questions

• How to build a static library for iOS?
• How to create a framework for iOS?
• How to publish iOS and OSX framework through CocoaPods?

Then you under the cat.
.

What for?

One could simply answer, “So that there are more libraries good and useful for iOS,” but I had another reason. While working on a board game for iOS, I wanted to use machine learning to develop the computer opponent's Artificial Intelligence (AI).

The plan was simple, first train the AI ​​by driving the neural network through tens of thousands of games, running them on powerful hardware under Mac OS, then take the trained network and use it in the application on iPhone / iPad. The first part of the plan was relatively simple to implement, if only because Shark version 2.3.4 can be installed on the poppy using brew install shark.

Here is the time to note that the article is about the version of the library 2.3.4. The project itself has already gone far from 2.3.4, beta version 3.0 is currently available . The code for the new version has been significantly redesigned, uses boost as a third-party library, so this is a completely different story, especially in the context of iOS.

Let's get back to the second part of the plan. After training, I have in my hands a file with neural network coefficients. The format of this file is not described anywhere (or I searched poorly), so the least expensive way to read and use it is with the same Shark, which means you need to port the library to iOS.

How?

The short answer is to take the sources and compile, a detailed answer is an article on Habré.

Clear business, before writing on Habr, I first assembled the library and designed this whole thing in the form of a repository on GitHub . Therefore, for clarity, I will season the post with pieces of code from the build script.

The whole process can be divided into the following steps

• Download source code
• Patch the source code
• Configure and Build
• Make a "thick" library
• Pack in the framework
• Post to CocoaPods

Yes, namely, we will not just collect a “thick” (aka “fat” or fat) static library for development for iOS devices and a simulator (and separately for OSX), we will also arrange everything as a framework and make it accessible through CocoaPods .

Well, let's get started, step by step.

Download source code

Everything is simple here, I won’t go into too much detail so that it doesn’t work out too chewed, download and unpack the archive.

Patch the source code

Initially, the code was developed using gcc 4.2, but we are trying to compile it using clang version 5.0. Since the release of gcc 4.2, the compilers and the C ++ standard have gone a long way, so it’s not surprising that clangs will not like some pieces of code. There is nothing left but to fix, i.e. patch, problematic code.

Default constructor

For the first time, the compiler will stumble on line 78 in the file ReClaM/EarlyStopping.cpp.

EarlyStopping::EarlyStopping(unsigned sl = 5)

The constructor EarlyStoppinghas a single argument ( unsigned sl) and the default value ( 5) is registered for this argument . Thus, this constructor becomes the default constructor, “well, let it” - gcc 4.2 will say, but clang has a completely different opinion on this. You can find a discussion of this error on StackOverflow . We will correct it “forehead”, ie just remove the default value.

EarlyStopping::EarlyStopping(unsigned sl)

Wait a second! - you say, - what if this constructor is used somewhere in the library code without an argument?
Absolutely fair question. Fortunately, this constructor is not called anywhere else in the code (either explicitly or implicitly), which means there will be no harm from such a change, but you need to be careful when calling it in your code and be sure to pass the value to sl.

finite is not available for iOS

The next error is that the function is finite(x)not available (not included) for iOS, both for devices and for the simulator. You can find references to this problem on the web, for example here .

The solution is to use the function isfinite(x). We redefine finite(x)how isfinite(x)using preprocessor macros. To do this, add the following code toSharkDefs.h


The logic here is the following if we compile for one of the Apple OSes, i.e. defined __APPLE__and __MACH__then include the header file TargetConditionals.h, after which we check the values TARGET_IPHONE_SIMULATOR, TARGET_OS_IPHONEand TARGET_OS_MAC, and redefine finite(x)for the iOS device and simulator.

Exactly the same problem with a function drem(x, y)that needs to be replaced with remainder(x, y). However, Shark drem(x, y)does not use, so this is just a note of information.

Fixes in FileUtil

The following errors are related to the module FileUtil, namely the file FileUtil.h.

At first, clang cannot find type definitions iotypeand constants SetDefault, ScanFromand PrintTo. Let's give the compiler a hint using the namespace (namespace), i.e. perform a simple replacement where necessary

iotype -> FileUtil::iotype
SetDefault -> FileUtil::SetDefault
ScanFrom -> FileUtil::ScanFrom
PrintTo -> FileUtil::PrintTo

The last problem in this file is a function io_strictthat calls functions scanFrom_strictand printTo_strict, declared later.
Decision? Just move io_strictto the end of the file and no problem.

double erf (double) throw () ;

Compilation will Mixture/MixtureOfGaussians.cppmark the following code as erroneous

extern "C" double erf(double) throw();

extern "C"the declaration does not match the original prototype.
In this case, the problem is solved by removal throw().

extern "C" double erf(double);

I am aware that for some corrections I do not give a detailed explanation, such as deletion throw(). I will be glad to comments in the comments.

RandomVector this-> p

The next file Mixture/RandomVector.hin line, line 72. The compiler has no idea what to do with a link to p. Also alarming is the code comment on this line ( // !!!).

for (unsigned k = x.dim(0); k--;) {
    l += log(Shark::max(p(x[ k ]), 1e-100));    // !!!
}

After analyzing the code, I found this one p. RandomVectorinherits RandomVarfrom which the "lost" method is declared p.


The design begs this->p.

for (unsigned k = x.dim(0); k--;) {
    l += log(Shark::max(this->p(x[ k ]), 1e-100));    // !!!
}

CMakeLists.txt

The latest patch has nothing to do with compilation errors.
Since we want to get a library for iOS, this library must be static, but the project will build a dynamic library by default. To get the desired result, replace one line in the file CMakeLists.txt.

ADD_LIBRARY( shark SHARED ${SRCS} )
# заменить на
ADD_LIBRARY( shark STATIC ${SRCS} )

In principle, you can not replace the line with "SHARED", but simply add another with "STATIC", then as a result both dynamic and static libraries will be compiled.

Apply the patch

Of course, if you decide to repeat this process, it makes no sense to pick the code manually. To save time and nerves there is a ready-made patch .
If you are interested, this patch was obtained using the utilitydiff

diff -crB shark_orig shark_patched > shark.patch

And to apply this patch, you need to use the utility, attention, patch

SRC_DIR=src
# ---
patchSource() {
        echo Patching source code...
        patch -d $SRC_DIR/Shark -p1 --forward -r - -i ../../shark.patch
        doneSection
}

Configure and Build

Finally, the patches are over, you can proceed directly to the assembly.
We will assemble the project with the help make, it remains only to configure everything correctly for each platform, the utility cmake(configure make) will help us in this , which will create Makefileother files necessary make. We will have to configure and build the library 3 times, for the following platforms and architectures

• iOS device - Architecture armv7, armv7s,arm64
• iOS simulator - architecture i386andx86_64
• OSX - Architecture x86_64

Yes, an important point, we will use Xcode 5. Architecture support has been armv6officially removed from Xcode 5, so we will not consider it at all.

We will merge the libraries for the iOS device and simulator into one “thick” library, which can be used simultaneously for development on devices and the simulator without unnecessary gestures.

Cmake basics

So cmake. For each platform, we need to properly configure the environment using the following settings

• C ++ compiler ( CMAKE_CXX_COMPILER)
  By default /usr/bin/c++, it is used , but we need to use clang++Xcode from the toolchain.
• Flags of the C ++ compiler ( CMAKE_CXX_FLAGS)
  It is with these flags that we will set the necessary architectures and other important compilation options.
• C compiler ( CMAKE_C_COMPILER)
  Despite the lack of pure C code, you still need to configure this parameter correctly.
• Root system directory ( CMAKE_OSX_SYSROOT)
  Using this setting, we will tell the compiler where to look for all the standard system libraries for the selected platform.
• Installation Prefix ( CMAKE_INSTALL_PREFIX)
  This parameter is optional. It is used by the command make install, but we will not do this.
• Generator of assembly files (flag -G)
  With this flag we can choose one of many build systems. In our case, it will be “Unix Makefiles”, i.e. cmakewill generate the familiar Makefile. It would be logical to use the Xcode project, but I was not able to assemble everything in this way.

Xcode Toolkit

It may not be the most successful translation of the term Toolchain, but certainly better than a “tool chain”. Xcode contains all the tools we need (compilers, libraries for the iPhone OS SDK and iPhone Simulator SDK, etc.) Of course, Xcode must be installed, you also need to install the same tools (Xcode Command Line Tools), you can do this in Xcode settings or using the command xcode-select --install.

Now use xcode-selectto find all the necessary tools and directories.


In the case of the Mac OS X SDK, cmakesmart enough to find the root system directory yourself, without additional instructions.

Next for each platform in order.

iOS devices (iPhone OS SDK)
We will collect in a separate folder, say build/ios.

We already found compilers, now we will define flags for the C ++ compiler. As we will collect for armv7, armv7sand arm64architectures, we will use the special flag -archfor clang++.

All necessary system libraries for ARM architectures are in a SDKs/iPhoneOS7.0.sdksubdirectoryXCODE_ARM_ROOT

CXX_FLAGS="-arch armv7 -arch armv7s -arch arm64"
SYSTEM_ROOT=${XCODE_ARM_ROOT}/SDKs/iPhoneOS7.0.sdk

The call cmakewill look like this

mkdir -p build/ios
cd build/ios
cmake \
  -DCMAKE_CXX_COMPILER=$CXX_COMPILER \
  -DCMAKE_OSX_SYSROOT="$SYSTEM_ROOT" \
  -DCMAKE_C_COMPILER=$C_COMPILER \
  -DCMAKE_CXX_FLAGS="$CXX_FLAGS" \
  -G "Unix Makefiles" \
  ../../src/Shark

By the way, it cmakewill do a lot of additional work for us. It is sufficient to specify only the C and C ++ compilers, cmakeas will all the other tools of the same tool, such as ranlib, lipo, ld, etc.

iOS simulator (iPhone Simulator SDK)
The same compilers.

We need the architectures this time i386and x86_64, the last one is needed for testing on a simulator of devices with 64-bit architecture, for example, “iPhone Retina (4-inch 64-bit)”.

Necessary libraries - in ${XCODE_SIM_ROOT}/SDKs/iPhoneSimulator7.0.sdk.

And another very important point that the code was built specifically for the simulator, and not OS X with the same architecture, the compiler needs to give an additional hint using the flag-mios-simulator-version-min=7.0

mkdir -p build/sim
cd build/sim
CXX_FLAGS="-arch i386 -arch x86_64 -mios-simulator-version-min=7.0"
SYSTEM_ROOT=${XCODE_SIM_ROOT}/SDKs/iPhoneSimulator7.0.sdk
cmake \
  -DCMAKE_CXX_COMPILER=$CXX_COMPILER \
  -DCMAKE_OSX_SYSROOT="$SYSTEM_ROOT" \
  -DCMAKE_C_COMPILER=$C_COMPILER \
  -DCMAKE_CXX_FLAGS="$CXX_FLAGS" \
  -G "Unix Makefiles" \
  ../../src/Shark

Mac OS X
And finally, Mac OS X.
This time it is enough to specify only C and C ++ compilers, cmakehe will find the rest himself.

mkdir -p build/osx
cd build/osx
cmake \
  -DCMAKE_CXX_COMPILER=$CXX_COMPILER \
  -DCMAKE_C_COMPILER=$C_COMPILER \
  -G "Unix Makefiles" \
  ../../src/Shark

Tricks cmake

Well now, come on make! ”, You think, but no.
There cmakeare a couple of features to consider.

Compiler Test
First, at the very first start, it cmakeperforms a test for C and C ++ compilers. Oddly enough, clang and clang ++ successfully fail this test for iOS platforms. There are several recommendations on the Internet how to get around this test, for example, CMakeLists.txtadd NONEto the project description in a file , but that did not help me.

PROJECT( shark NONE )

The way that worked for me is to start first cmakewithout specifying clang and clang ++ compilers and other options. cmakewill generate a file CMakeCache.txtand directory CMakeFiles. If you now start cmakealready with various settings, the compiler test this time will not be performed.

Changing the configuration
Total, cmakeyou need to run at least 2 times, you will think again.
And again, no.

If important parameters, such as C and C ++ compilers, were changed at the next start, the changes will not take effect immediately. cmakeit would give a warning and advise you to run it again. In fact, cmakeit will not say anything, but the utility ccmakewill provide more information.ccmakeThis is a console GUI for cmake.

Generally

• Times cmake, no parameters, to pass the test
• Two cmake, with the right parameters
• Three cmakefor changes to take effect

make

Yes, now you can fulfill the cherished make. To make things faster, you can parallelize the assembly using the flag -j.

make -j16

This process will not take much time, as a result we will have a static library on hand libshark.a. For every fireman, with the help of the utility, filewe will make sure that the resulting libraries support all the necessary architectures.

Make a "thick" library

Libraries for iOS devices and the simulator need to be combined into one "thick" library.

Everything is quite simple here, the utility will cope with this task.lipo

mkdir -p lib/ios
$XCODE_TOOLCHAIN_BIN/lipo -create build/ios/libshark.a build/sim/libshark.a -o lib/ios/libshark.a

Just in case, check the resulting library ( file lib/ios/libshark.a) to make sure that all 5 architectures are in place.

Pack in the framework

It's time to carefully pack the library in the form of a framework. The framework directory is often called a bundle.

Create a bundle

At this stage, create the directory Shark.frameworkand the necessary folder structure inside, with all the necessary symbolic links, using mkdirand ln.

Shark.framework/
├── Documentation -> Versions/Current/Documentation
├── Headers -> Versions/Current/Headers
├── Resources -> Versions/Current/Resources
└── Versions
    ├── A
    │   ├── Documentation
    │   ├── Headers
    │   └── Resources
    └── Current -> A

Copy library

Now copy the static library inside the bundle, while renaming it to Shark.

cp build/ios/libshark.a Shark.framework/Versions/A/Shark

Copy header files

Next, copy all the .hfiles from src/Shark/includeto the Headersfolder inside the bundle. Then delete the unnecessary statistics.h. This file is not needed if only because there is no corresponding INSTALLcommand for it in the file CMakeLists.txt.

cp -r src/Shark/include/* Shark.framework/Headers/
rm Shark.framework/Headers/statistics.h

Patching header files
It would seem that headers copied everything, but again everything is not so obvious. If you try to use the framework right now, some strange errors will pop up related to the paths to the header files. And if in the case of the dynamic library under OS X I managed to get around the problem using the Header Search Path, then in the case of the framework, everything is not so simple.

Since the Shark developers have not taken care of the correct paths for the #includedirectives, we will have to do the work for them.

In search of the right approach, I turned my attention to the example of a well-organized library, namely boost. All paths in #includedirectives for components from boost start with boost/, for example

#include "boost/config.hpp"
#include 

We use the editor sedand patch the files using the following rules for all #includedirectives

• Add the missing space in "#include"and" #include "something" "(yes, there were some)
• Replace SharkDefs.honShark/SharkDefs.h
• Add Shark/paths for all library components

By components we mean subdirectories of the folder Headers, i.e. Array, Rng, LinAlg, FileUtil, EALib, MOO-EALib, ReClaM, Mixture, TimeSeries, Fuzzy.

Create Info.plist

The last step in creating a framework is to create the corresponding Info.plist. Important properties are the name and version of the framework.

До выхода iOS 7, а именно устройств с архитектурой arm64, существовала возможность создания универсальных статических библиотек, а значит и фреймворков, одновременно для iOS и OSX, включая симулятор. Ничто не мешало упаковать armv6, armv7, armv7s, i386 и x86_64 в одну жирную библиотеку. Устройства с 64-битной архитектурой спутали все карты. В «толстой» библиотеке не может быть два слоя с одинаковой архитектурой, а значит не получится запихать x86_64 для iOS симулятора 64-битных устройств и x86_64 для OS X.

Опубликовать на CocoaPods

Right now, you can take a ready-made framework and feel free to add it to your projects, but you want to automate this stage too. This is exactly what CocoaPods is for . If for some reason you are behind the train and are not yet using CocoaPods, it's time to pay attention to this project.

Creating specifications for the hearth is another story. Despite the fact that I practically painted every little thing up to this point, here I will refer to the finished result.


And an example of use.

Summary

In total, if you suddenly come up with your own problem, the result of which, to put it mildly, is not guaranteed, still try to do something. You never know what new skills will be able to pump as a result.