Thrift as REST API
A small article about how we are faced with the problems of synchronization between teams of client and server development. How we connected Thrift in order to simplify the interaction between our teams.
Who cares how we did it, and what "side" effects we caught, please look under the cat.
In early 2017, when we started a new project, we chose EmberJS as the front end. Which almost automatically led us to work on the REST scheme in organizing the interaction of the client and server parts of the application. Because EmberData provides a handy tool for separating the work of backend and frontend commands, and using an Adapter allows you to select the “protocol” of interaction.
At first, everything is fine - Ember gave us the opportunity to implement emulation of requests to the server. The data for emulation of server models were put into separate fuxtures. If somewhere we started working, I do not use Ember Data, then Ember allows you to write an endpoint handler emulator next to it and return this data. We had an agreement that backend-developers should make changes to these files to keep the data up-to-date for the frontend developers to work correctly. But as always, when everything is built on “agreements” (and there is no tool for their verification), a moment comes when “something goes wrong.”
New requirements led not only to the appearance of new data on the client, but also to update the old data model. What ultimately led to the fact that maintaining synchronism of models on the server and on its emulation in the client's source code was simply expensive. Now the development of the client part, as a rule, begins after the server stub is ready. And the development is conducted on top of the working server, and this complicates the teamwork and increases the release time of the new functionality.
Now we are abandoning EmberJS in favor of VueJS. and within the framework of the decision on migration, we began to look for solutions to this problem. The following criteria were developed:
Thinking, it was decided to stop at Thrift . This gave us a simple and understandable API description language.
For interaction, we use TMultiplexedProcessor, accessible via TServlet, using the TJSONProtocol. I had to dance a little bit to make Thrift integrate seamlessly with Spring. For this, I had to create and register the Servlet in the ServletContainer programmatically.
What should be noted here. In this code, two service areas are formed. Protected, which is available at / api / v2 / thrift. And open, available at / api / v2 / thrift-ns. For these areas use different filters. In the first case, when accessing the service by a cookie, an object is created that identifies the user who makes the call. If it is impossible to form such an object, a 401 error is thrown, which is correctly processed on the client side. In the second case, the filter skips all service requests, and if it determines that authorization has occurred, then, after performing the operation, it fills in cookies with the necessary information so that you can make requests to the protected area.
To connect a new service, you have to write a little extra code.
And register the processor
The last part of the code can be simplified by attaching an additional interface to all processors, which will allow you to immediately receive a list of processors with one designer’s parameter, and handing the responsibility for the processor access key value to the processor itself.
The work in the mode “without server” has undergone a little change. The developers of the frontend part made an offer that they would work on the stub PHP server. They themselves generate classes for their server that implement the signature for the required protocol version. And implement the server with the necessary data set. All this allows them to work before the server-side developers finish their work.
The main processing point on the client side is the thrift-plugin written by us.
For the correct operation of this plugin, you must connect the generated classes.
Calling server methods on the client looks like this:
Here I don’t go deep into the features of VueJS itself, where it’s right to keep the code calling the server. This code can be used inside the component, inside the route and inside the Vuex-action.
When working with the client side, there are a couple of limitations that need to be taken into account after mental migration with internal thrift integration.
The transition to Thrift allowed us to solve the problems that are present in the interaction between server and client development when working on the old version of the interface. Allowed to make possible the handling of global errors in one place.
At the same time, due to the strict API typing and, consequently, the strict rules of data serialization / deserialization, we received an increase of ~ 30% in the interaction time per client and server for most requests (when comparing the same requests via REST and THRIFT interaction, from the time of sending the request to the server until the response is received)
Who cares how we did it, and what "side" effects we caught, please look under the cat.
Prehistory
In early 2017, when we started a new project, we chose EmberJS as the front end. Which almost automatically led us to work on the REST scheme in organizing the interaction of the client and server parts of the application. Because EmberData provides a handy tool for separating the work of backend and frontend commands, and using an Adapter allows you to select the “protocol” of interaction.
At first, everything is fine - Ember gave us the opportunity to implement emulation of requests to the server. The data for emulation of server models were put into separate fuxtures. If somewhere we started working, I do not use Ember Data, then Ember allows you to write an endpoint handler emulator next to it and return this data. We had an agreement that backend-developers should make changes to these files to keep the data up-to-date for the frontend developers to work correctly. But as always, when everything is built on “agreements” (and there is no tool for their verification), a moment comes when “something goes wrong.”
New requirements led not only to the appearance of new data on the client, but also to update the old data model. What ultimately led to the fact that maintaining synchronism of models on the server and on its emulation in the client's source code was simply expensive. Now the development of the client part, as a rule, begins after the server stub is ready. And the development is conducted on top of the working server, and this complicates the teamwork and increases the release time of the new functionality.
Project development
Now we are abandoning EmberJS in favor of VueJS. and within the framework of the decision on migration, we began to look for solutions to this problem. The following criteria were developed:
- Compatibility with older and newer protocol versions
- Maximum convenience for frontend-developers when working “without server”
- Separation of API description from test data
- Easy to synchronize call signatures
- clear signature description
- easy modification of both frontend and backend developers
- maximum autonomy
- A strongly typed API is desirable. Those. as soon as possible to detect the fact of a protocol change
- Ease of testing server logic
- Integration with Spring on the server side without dancing with tambourines.
Implementation
Thinking, it was decided to stop at Thrift . This gave us a simple and understandable API description language.
namespace java ru.company.api
namespace php ru.company.api
namespace javascrip ru.company.api
conststring DIRECTORY_SERVICE= "directoryService"
exception ObjectNotFoundException{
}
struct AdvBreed {
1: string id,
2: string name,
3: optional string title
}
service DirectoryService {
list<AdvBreed> loadBreeds()
AdsBreed getAdvBreedById(1: string id)
}
For interaction, we use TMultiplexedProcessor, accessible via TServlet, using the TJSONProtocol. I had to dance a little bit to make Thrift integrate seamlessly with Spring. For this, I had to create and register the Servlet in the ServletContainer programmatically.
@ComponentclassThriftRegister : ApplicationListener<ContextRefreshedEvent>,
ApplicationContextAware, ServletContextAware {companionobject {
privateconstval unsecureAreaUrlPattern = "/api/v2/thrift-ns"privateconstval secureAreaUrlPattern = "/api/v2/thrift"
}
privatevar inited = falseprivatelateinitvar appContext:ApplicationContext
privatelateinitvar servletContext:ServletContext
overridefunonApplicationEvent(event: ContextRefreshedEvent) {
if (!inited) {
initServletsAndFilters()
inited = true
}
}
privatefuninitServletsAndFilters() {
registerOpenAreaServletAndFilter()
registerSecureAreaServletAndFilter()
}
privatefunregisterSecureAreaServletAndFilter() {
registerServletAndFilter(SecureAreaServlet::class.java,
SecureAreaThriftFilter::class.java, secureAreaUrlPattern)
}
privatefunregisterOpenAreaServletAndFilter() {
registerServletAndFilter(UnsecureAreaServlet::class.java,
UnsecureAreaThriftFilter::class.java, unsecureAreaUrlPattern)
}
privatefunregisterServletAndFilter(servletClass:Class<outServlet>,
filterClass:Class<outFilter>, pattern:String) {
val servletBean = appContext.getBean(servletClass)
val addServlet = servletContext.addServlet(servletClass.simpleName, servletBean)
addServlet.setLoadOnStartup(1)
addServlet.addMapping(pattern)
val filterBean = appContext.getBean(filterClass)
val addFilter = servletContext.addFilter(filterClass.simpleName, filterBean)
addFilter.addMappingForUrlPatterns(null, true, pattern)
}
overridefunsetApplicationContext(applicationContext: ApplicationContext) {
appContext = applicationContext
}
overridefunsetServletContext(context: ServletContext) {
this.servletContext = context
}
}What should be noted here. In this code, two service areas are formed. Protected, which is available at / api / v2 / thrift. And open, available at / api / v2 / thrift-ns. For these areas use different filters. In the first case, when accessing the service by a cookie, an object is created that identifies the user who makes the call. If it is impossible to form such an object, a 401 error is thrown, which is correctly processed on the client side. In the second case, the filter skips all service requests, and if it determines that authorization has occurred, then, after performing the operation, it fills in cookies with the necessary information so that you can make requests to the protected area.
To connect a new service, you have to write a little extra code.
@Component
class DirectoryServiceProcessor @Autowired constructor(handler: DirectoryService.Iface):
DirectoryService.Processor<DirectoryService.Iface>(handler)And register the processor
@Component
class SecureMultiplexingProcessor @Autowired constructor(dsProcessor: DirectoryServiceProcessor) : TMultiplexedProcessor() {
init {
this.registerProcessor(DIRECTORY_SERVICE, dsProcessor)
...
}
}The last part of the code can be simplified by attaching an additional interface to all processors, which will allow you to immediately receive a list of processors with one designer’s parameter, and handing the responsibility for the processor access key value to the processor itself.
The work in the mode “without server” has undergone a little change. The developers of the frontend part made an offer that they would work on the stub PHP server. They themselves generate classes for their server that implement the signature for the required protocol version. And implement the server with the necessary data set. All this allows them to work before the server-side developers finish their work.
The main processing point on the client side is the thrift-plugin written by us.
import store from'../../store'import { UNAUTHORIZED } from'../../store/actions/auth'const thrift = require('thrift')
exportdefault {
install (Vue, options) {
const DirectoryService = require('./gen-nodejs/DirectoryService')
let _options = {
transport: thrift.TBufferedTransport,
protocol: thrift.TJSONProtocol,
path: '/api/v2/thrift',
https: location.protocol === 'https:'
}
let _optionsOpen = {
...
}
const XHRConnectionError = (_status) => {
if (_status === 0) {
....
} elseif (_status >= 400) {
if (_status === 401) {
store.dispatch(UNAUTHORIZED)
}
...
}
}
let bufers = {}
thrift.XHRConnection.prototype.flush = function () {
var self = thisif (this.url === undefined || this.url === '') {
returnthis.send_buf
}
var xreq = this.getXmlHttpRequestObject()
if (xreq.overrideMimeType) {
xreq.overrideMimeType('application/json')
}
xreq.onreadystatechange = function () {
if (this.readyState === 4) {
if (this.status === 200) {
self.setRecvBuffer(this.responseText)
} else {
if (this.status === 404 || this.status >= 500) {...
} else {...
}
}
}
}
xreq.open('POST', this.url, true)
Object.keys(this.headers).forEach(function (headerKey) {
xreq.setRequestHeader(headerKey, self.headers[headerKey])
})
if (process.env.NODE_ENV === 'development') {
let sendBuf = JSON.parse(this.send_buf)
bufers[sendBuf[3]] = this.send_buf
xreq.seqid = sendBuf[3]
}
xreq.send(this.send_buf)
}
const mp = new thrift.Multiplexer()
const connectionHostName = process.env.THRIFT_HOST ? process.env.THRIFT_HOST : location.hostname
const connectionPort = process.env.THRIFT_PORT ? process.env.THRIFT_PORT : location.port
const connection = thrift.createXHRConnection(connectionHostName, connectionPort, _options)
const connectionOpen = thrift.createXHRConnection(connectionHostName, connectionPort, _optionsOpen)
Vue.prototype.$ThriftPlugin = {
DirectoryService: mp.createClient('directoryService', DirectoryService, connectionOpen),
}
}
}For the correct operation of this plugin, you must connect the generated classes.
Calling server methods on the client looks like this:
thriftPlugin.DirectoryService.loadBreeds()
.then(_response => {
...
})
.catch(error => {
...
})
})
Here I don’t go deep into the features of VueJS itself, where it’s right to keep the code calling the server. This code can be used inside the component, inside the route and inside the Vuex-action.
When working with the client side, there are a couple of limitations that need to be taken into account after mental migration with internal thrift integration.
- Javascript client does not recognize null values. Therefore, for fields that can be null, you must specify the optional flag. In this case, the client will correctly perceive this value.
- Javascript does not know how to work with long values, so all integer identifiers must be cast to the server side string.
findings
The transition to Thrift allowed us to solve the problems that are present in the interaction between server and client development when working on the old version of the interface. Allowed to make possible the handling of global errors in one place.
At the same time, due to the strict API typing and, consequently, the strict rules of data serialization / deserialization, we received an increase of ~ 30% in the interaction time per client and server for most requests (when comparing the same requests via REST and THRIFT interaction, from the time of sending the request to the server until the response is received)