May 10, 2013 at 10:05Apache Thrift RPC Server. Are friends C ++ and Java
Hello colleagues.
In this topic I want to lay out instructions on how to quickly fasten Thrift to my crafts.
Thrift is a technology for organizing interprocess communication between system components. It was developed somewhere in the bowels of Facebook. This is a cross-language framework for creating RPC services, on a binary protocol. With this solution, you can "make friends" components written in different languages C #, C ++, Delphi, Erlang, Go, Java, PHP, Python, Ruby, etc. Description of service and data signatures is carried out using a special IDL language. The technology, in its essence, is similar to COM, but without all this binding with component registration. Also, let's not forget that COM is a Windows-only technology, while Thrift is cross-platform.
In general, I decided to experiment, try to extract some of the load-computational logic from Java to C ++, in the hope that the native C ++ code will be a little more productive, and try Thrift RPC for one, in the hope that it is faster than REST.
As it should be, there were some tambourines and rakes!
And so, first you need to put all this:
1. We put support for Boost, because everything is tied to it
2. download thrift tarball apache.softded.ru/thrift/0.9.0/thrift-0.9.0.tar.gz
unpack, run configure, and then assemble.
It seems to be all ... You can even try to generate code from the tutorial that comes with thrift tarball
The thrift command will generate C ++ wrappers, and carefully put them in the gen-cpp directory. The same thing can be done for Java, PHP, etc. ...
We are trying to compile and compile the supplied sources
Oops, get: error: 'uint32_t' does not name a type
It turns out there is a small cockroach in the thrift libraries associated with uint32_t. Treated by adding "#include"in" Thrift.h ", or (best of all) special compiler options -DHAVE_NETINET_IN_H -DHAVE_INTTYPES_H
Now it looks like this:
That's all, an executable file called something appeared.
We start it and get: error while loading shared libraries: libthrift.so.0: cannot open shared object file: No such file or directory
Maybe there are some elegant methods for solving this problem, but I solved it head-on by copying thrift files from / usr / local / lib in / lib
That's it, the example started. So, everything is in place, and everything works.
Now you can write your RPC server.
Its task is to be a key-value repository. Store long (several hundred thousand) bit masks. Add them (AND), and give the client an array of indices, which turned out units. Yes, Redis can do almost the same thing, but it doesn't suit me.
The full code is here: github.com/2anikulin/fast-hands.git
We describe the signatures of data and services in the thrift definition file :
And generate the wrappers.
Implementation in C ++
This code creates and starts the RPC server
In the FastHandsHandler class - all our applied functionality
is implemented. This is the header file.
We try to collect, and we get another error: c ++ undefined reference to apache :: thrift :: server :: TNonblockingServer
The fact is that, unlike the tutorial, my server is asynchronous and uses the TNonblockingServer class. To build the code, you need to add additional libraries -lthriftnb -levent
i.e. the assembly will now look like this:
Now all is well. The code is compiled, linked, the output file is named something. We
generate wrappers for java, and we write such a client
What is the result.
An interesting technology, and not a bad way to fasten transport functionality to bare C ++ code. But I will not say that it is much faster than REST, binary data is also transmitted beautifully via http. As for the performance of the code taken from Java in C ++, a miracle did not happen, it is 1.2 - 1.4 times faster, because serialization + expenses for the transport level.
In this topic I want to lay out instructions on how to quickly fasten Thrift to my crafts.
Thrift is a technology for organizing interprocess communication between system components. It was developed somewhere in the bowels of Facebook. This is a cross-language framework for creating RPC services, on a binary protocol. With this solution, you can "make friends" components written in different languages C #, C ++, Delphi, Erlang, Go, Java, PHP, Python, Ruby, etc. Description of service and data signatures is carried out using a special IDL language. The technology, in its essence, is similar to COM, but without all this binding with component registration. Also, let's not forget that COM is a Windows-only technology, while Thrift is cross-platform.
In general, I decided to experiment, try to extract some of the load-computational logic from Java to C ++, in the hope that the native C ++ code will be a little more productive, and try Thrift RPC for one, in the hope that it is faster than REST.
As it should be, there were some tambourines and rakes!
And so, first you need to put all this:
1. We put support for Boost, because everything is tied to it
$ sudo apt-get install libboost-dev libboost-test-dev libboost-program-options-dev libevent-dev automake libtool flex bison pkg-config g++ libssl-dev
2. download thrift tarball apache.softded.ru/thrift/0.9.0/thrift-0.9.0.tar.gz
unpack, run configure, and then assemble.
$ ./configure
$ make
$ sudo make install
It seems to be all ... You can even try to generate code from the tutorial that comes with thrift tarball
$ thrift --gen cpp tutorial.thrift
The thrift command will generate C ++ wrappers, and carefully put them in the gen-cpp directory. The same thing can be done for Java, PHP, etc. ...
We are trying to compile and compile the supplied sources
$ g++ -Wall -I/usr/local/include/thrift *.cpp -L/usr/local/lib -lthrift -o something
Oops, get: error: 'uint32_t' does not name a type
It turns out there is a small cockroach in the thrift libraries associated with uint32_t. Treated by adding "#include
Now it looks like this:
$ g++ -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -I/usr/local/include/thrift *.cpp -L/usr/local/lib -lthrift -o something
That's all, an executable file called something appeared.
We start it and get: error while loading shared libraries: libthrift.so.0: cannot open shared object file: No such file or directory
Maybe there are some elegant methods for solving this problem, but I solved it head-on by copying thrift files from / usr / local / lib in / lib
That's it, the example started. So, everything is in place, and everything works.
Now you can write your RPC server.
Its task is to be a key-value repository. Store long (several hundred thousand) bit masks. Add them (AND), and give the client an array of indices, which turned out units. Yes, Redis can do almost the same thing, but it doesn't suit me.
The full code is here: github.com/2anikulin/fast-hands.git
We describe the signatures of data and services in the thrift definition file :
namespace cpp fasthands
namespace java fasthands
namespace php fasthands
namespace perl fasthands
exception InvalidOperation {
1: i32 what,
2: string why
}
service FastHandsService {
i32 put(1:i32 key, 2:binary value),
binary get(1:i32 key),
list bitAnd(1:list keys) throws (1:InvalidOperation ouch)
}
And generate the wrappers.
Implementation in C ++
This code creates and starts the RPC server
#define PORT 9090
#define THREAD_POOL_SIZE 15
int main() {
printf("FastHands Server started\n");
shared_ptr protocolFactory(new TBinaryProtocolFactory());
shared_ptr handler(new FastHandsHandler());
shared_ptr processor(new FastHandsServiceProcessor(handler));
shared_ptr threadManager = ThreadManager::newSimpleThreadManager(THREAD_POOL_SIZE);
shared_ptr threadFactory = shared_ptr(new PosixThreadFactory());
threadManager->threadFactory(threadFactory);
threadManager->start();
TNonblockingServer server(processor, protocolFactory, PORT, threadManager);
server.serve();
printf("done.\n");
return 0;
}
In the FastHandsHandler class - all our applied functionality
is implemented. This is the header file.
class FastHandsHandler : virtual public FastHandsServiceIf {
public:
FastHandsHandler();
int32_t put(const int32_t key, const std::string& value);
void get(std::string& _return, const int32_t key);
void bitAnd(std::vector & _return, const std::vector & keys);
private:
void appendBitPositions(std::vector & positions, unsigned char bits, int offset);
private:
std::map m_store;
};
We try to collect, and we get another error: c ++ undefined reference to apache :: thrift :: server :: TNonblockingServer
The fact is that, unlike the tutorial, my server is asynchronous and uses the TNonblockingServer class. To build the code, you need to add additional libraries -lthriftnb -levent
i.e. the assembly will now look like this:
g++ -DHAVE_INTTYPES_H -DHAVE_NETINET_IN_H -Wall -I/usr/local/include/thrift *.cpp -L/usr/local/lib -lthrift -lthriftnb -levent -o something
Now all is well. The code is compiled, linked, the output file is named something. We
generate wrappers for java, and we write such a client
import fasthands.FastHandsService;
import fasthands.InvalidOperation;
import org.apache.thrift.TException;
import org.apache.thrift.transport.TFramedTransport;
import org.apache.thrift.transport.TTransport;
import org.apache.thrift.transport.TSocket;
import org.apache.thrift.transport.TTransportException;
import org.apache.thrift.protocol.TBinaryProtocol;
import org.apache.thrift.protocol.TProtocol;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.List;
public class JavaClient {
public static void main(String [] args) {
TTransport transport = new TFramedTransport(new TSocket("localhost", 9090));
TProtocol protocol = new TBinaryProtocol(transport);
final FastHandsService.Client client = new FastHandsService.Client(protocol);
final List filters = new ArrayList();
try {
transport.open();
int count = 12500;
byte bt[] = new byte[count];
for (int i =0; i < count; i++) {
bt[i] = (byte)0xFF;
}
for (int i = 0; i < 50; i++) {
client.put(i, ByteBuffer.wrap(bt)) ;
filters.add(i);
}
List list = client.bitAnd(filters);
System.out.println(list.size());
} catch (TTransportException e) {
e.printStackTrace();
} catch (TException e) {
e.printStackTrace();
}
transport.close();
}
}
What is the result.
An interesting technology, and not a bad way to fasten transport functionality to bare C ++ code. But I will not say that it is much faster than REST, binary data is also transmitted beautifully via http. As for the performance of the code taken from Java in C ++, a miracle did not happen, it is 1.2 - 1.4 times faster, because serialization + expenses for the transport level.