September 12, 2013 at 11:49

RESTful API on Node.js + MongoDB

Being a mobile application developer, I often need backend services for storing user data, authorization, and more. Of course, for such tasks you can use BaaS (Parse, Backendless, etc ...). But your decision is always more convenient and practical.

And yet I decided to study technologies completely unknown to me, which are now very popular and positioned as easily mastered by beginners and not requiring deep knowledge and experience for implementing large-scale projects. So let's check together whether a layman can write his own effective and correct backend.

This article will discuss the construction of the REST API for a mobile application on Node.js using the Express.js framework and the Mongoose.js module for working with MongoDB. To control access, we will resort to OAuth 2.0 technology using the OAuth2orize and Passport.js modules.

I am writing from the perspective of an absolute beginner. Glad of any feedback and corrections on the code and logic!

Content

  1. Node.js + Express.js, a simple web server
  2. Error handling
  3. RESTful API endpoints, CRUD
  4. MongoDB & Mongoose.js
  5. Access control - OAuth 2.0, Passport.js


I work in OSX, IDE - JetBrains WebStorm .

I learned the basics of Node.js in the screencasts of Ilya Kantor , I highly recommend it! ( And here is the post about them on the Habré ) The

finished project at the last stage can be taken on GitHub . To install all modules, run the npm install command in the project folder.

1. Node.js + Express.js, a simple web server


Node.js has non-blocking I / O, which is cool for an API that many clients will access. Express.js is a developed, lightweight framework that allows you to quickly describe all the paths (API endpoints) that we will process. You can also find many useful modules to it.

Create a new project with a single server.js file. Since the application will rely entirely on Express.js , install it. Installation of third-party modules occurs through Node Package Manager by executing a command npm install modulenamein the project folder.

cd NodeAPI
npm i express


Express is installed in the node_modules folder. Connect it to the application:

var express         = require('express');
var app = express();
app.listen(1337, function(){
    console.log('Express server listening on port 1337');
});


Run the application through the IDE or console ( node server.js). This code will create a web server on localhost: 1337. If you try to open it, it will display a message Cannot GET /. This is because we have not yet set a single route. Next, create several paths and make basic Express settings.

var express         = require('express');
var path            = require('path'); // модуль для парсинга пути
var app = express();
app.use(express.favicon()); // отдаем стандартную фавиконку, можем здесь же свою задать
app.use(express.logger('dev')); // выводим все запросы со статусами в консоль
app.use(express.bodyParser()); // стандартный модуль, для парсинга JSON в запросах
app.use(express.methodOverride()); // поддержка put и delete
app.use(app.router); // модуль для простого задания обработчиков путей
app.use(express.static(path.join(__dirname, "public"))); // запуск статического файлового сервера, который смотрит на папку public/ (в нашем случае отдает index.html)
app.get('/api', function (req, res) {
    res.send('API is running');
});
app.listen(1337, function(){
    console.log('Express server listening on port 1337');
});


Now localhost: 1337 / api will return our message. localhost: 1337 will display index.html.

Here we move on to error handling.

2. Error handling


First, connect the convenient Winston logging module . We will use it through our wrapper. Install in the project root npm i winston, then create the libs / folder and the log.js. file there

var winston = require('winston');
function getLogger(module) {
    var path = module.filename.split('/').slice(-2).join('/'); //отобразим метку с именем файла, который выводит сообщение
    return new winston.Logger({
        transports : [
            new winston.transports.Console({
                colorize:   true,
                level:      'debug',
                label:      path
            })
        ]
    });
}
module.exports = getLogger;


We created 1 transport for logs - to the console. We can also sort and save messages separately, for example, in a database or file. Connect the logger to our server.js.

var express         = require('express');
var path            = require('path'); // модуль для парсинга пути
var log             = require('./libs/log')(module);
var app = express();
app.use(express.favicon()); // отдаем стандартную фавиконку, можем здесь же свою задать
app.use(express.logger('dev')); // выводим все запросы со статусами в консоль
app.use(express.bodyParser()); // стандартный модуль, для парсинга JSON в запросах
app.use(express.methodOverride()); // поддержка put и delete
app.use(app.router); // модуль для простого задания обработчиков путей
app.use(express.static(path.join(__dirname, "public"))); // запуск статического файлового сервера, который смотрит на папку public/ (в нашем случае отдает index.html)
app.get('/api', function (req, res) {
    res.send('API is running');
});
app.listen(1337, function(){
    log.info('Express server listening on port 1337');
});


Our newsletter is now beautifully separately displayed in the console. Add error handling 404 and 500.

app.use(function(req, res, next){
    res.status(404);
    log.debug('Not found URL: %s',req.url);
    res.send({ error: 'Not found' });
    return;
});
app.use(function(err, req, res, next){
    res.status(err.status || 500);
    log.error('Internal error(%d): %s',res.statusCode,err.message);
    res.send({ error: err.message });
    return;
});
app.get('/ErrorExample', function(req, res, next){
    next(new Error('Random error!'));
});


Now, if there are no paths available, Express will return our message. With an internal application error, our handler will also work, this can be checked by contacting localhost: 1337 / ErrorExample.

3. RESTful API endpoints, CRUD


Add paths to handle some “articles”. There is an excellent article on the hub explaining how to make a convenient API correctly. We will not fill them with logic yet, we will do this in the next step, with the database connected.

app.get('/api/articles', function(req, res) {
    res.send('This is not implemented now');
});
app.post('/api/articles', function(req, res) {
    res.send('This is not implemented now');
});
app.get('/api/articles/:id', function(req, res) {
    res.send('This is not implemented now');
});
app.put('/api/articles/:id', function (req, res){
    res.send('This is not implemented now');    
});
app.delete('/api/articles/:id', function (req, res){
    res.send('This is not implemented now');
});


For testing post / put / delete, I recommend a wonderful wrapper over cURL - httpie . Further I will give examples of queries using this tool specifically.

4. MongoDB & Mongoose.js


Choosing a DBMS, I was again guided by the desire to learn something new. MongoDB is the most popular NoSQL document-oriented DBMS. Mongoose.js - a wrapper that allows you to create convenient and functional document schemes.

Download and install MongoDB . Set mongoose: npm i mongoose. I selected work with the database in the libs / mongoose.js file.

var mongoose    = require('mongoose');
var log         = require('./log')(module);
mongoose.connect('mongodb://localhost/test1');
var db = mongoose.connection;
db.on('error', function (err) {
    log.error('connection error:', err.message);
});
db.once('open', function callback () {
    log.info("Connected to DB!");
});
var Schema = mongoose.Schema;
// Schemas
var Images = new Schema({
    kind: {
        type: String,
        enum: ['thumbnail', 'detail'],
        required: true
    },
    url: { type: String, required: true }
});
var Article = new Schema({
    title: { type: String, required: true },
    author: { type: String, required: true },
    description: { type: String, required: true },
    images: [Images],
    modified: { type: Date, default: Date.now }
});
// validation
Article.path('title').validate(function (v) {
    return v.length > 5 && v.length < 70;
});
var ArticleModel = mongoose.model('Article', Article);
module.exports.ArticleModel = ArticleModel;


In this file, a connection to the database is made, and object schemes are also declared. Articles will contain image objects. A variety of complex validations can also be described here.

At this stage, I suggest connecting the nconf module to store the path to the database in it. Also in the config, save the port on which the server is created. The module is installed by the team npm i nconf. The wrapper will be libs / config.js

var nconf = require('nconf');
nconf.argv()
    .env()
    .file({ file: './config.json' });
module.exports = nconf;


It follows that we must create config.json in the root of the project.

{
    "port" : 1337,
    "mongoose": {
        "uri": "mongodb://localhost/test1"
    }
}


Changes to mongoose.js (only in the header):

var config      = require('./config');
mongoose.connect(config.get('mongoose:uri'));


Server.js changes:

var config          = require('./libs/config');
app.listen(config.get('port'), function(){
    log.info('Express server listening on port ' + config.get('port'));
});


Now add CRUD actions to our existing paths.

var log             = require('./libs/log')(module);
var ArticleModel    = require('./libs/mongoose').ArticleModel;
app.get('/api/articles', function(req, res) {
    return ArticleModel.find(function (err, articles) {
        if (!err) {
            return res.send(articles);
        } else {
            res.statusCode = 500;
            log.error('Internal error(%d): %s',res.statusCode,err.message);
            return res.send({ error: 'Server error' });
        }
    });
});
app.post('/api/articles', function(req, res) {
    var article = new ArticleModel({
        title: req.body.title,
        author: req.body.author,
        description: req.body.description,
        images: req.body.images
    });
    article.save(function (err) {
        if (!err) {
            log.info("article created");
            return res.send({ status: 'OK', article:article });
        } else {
            console.log(err);
            if(err.name == 'ValidationError') {
                res.statusCode = 400;
                res.send({ error: 'Validation error' });
            } else {
                res.statusCode = 500;
                res.send({ error: 'Server error' });
            }
            log.error('Internal error(%d): %s',res.statusCode,err.message);
        }
    });
});
app.get('/api/articles/:id', function(req, res) {
    return ArticleModel.findById(req.params.id, function (err, article) {
        if(!article) {
            res.statusCode = 404;
            return res.send({ error: 'Not found' });
        }
        if (!err) {
            return res.send({ status: 'OK', article:article });
        } else {
            res.statusCode = 500;
            log.error('Internal error(%d): %s',res.statusCode,err.message);
            return res.send({ error: 'Server error' });
        }
    });
});
app.put('/api/articles/:id', function (req, res){
    return ArticleModel.findById(req.params.id, function (err, article) {
        if(!article) {
            res.statusCode = 404;
            return res.send({ error: 'Not found' });
        }
        article.title = req.body.title;
        article.description = req.body.description;
        article.author = req.body.author;
        article.images = req.body.images;
        return article.save(function (err) {
            if (!err) {
                log.info("article updated");
                return res.send({ status: 'OK', article:article });
            } else {
                if(err.name == 'ValidationError') {
                    res.statusCode = 400;
                    res.send({ error: 'Validation error' });
                } else {
                    res.statusCode = 500;
                    res.send({ error: 'Server error' });
                }
                log.error('Internal error(%d): %s',res.statusCode,err.message);
            }
        });
    });
});
app.delete('/api/articles/:id', function (req, res){
    return ArticleModel.findById(req.params.id, function (err, article) {
        if(!article) {
            res.statusCode = 404;
            return res.send({ error: 'Not found' });
        }
        return article.remove(function (err) {
            if (!err) {
                log.info("article removed");
                return res.send({ status: 'OK' });
            } else {
                res.statusCode = 500;
                log.error('Internal error(%d): %s',res.statusCode,err.message);
                return res.send({ error: 'Server error' });
            }
        });
    });
});


Thanks to Mongoose and the described schemes - all operations are extremely clear. Now, in addition to node.js, you should start the mongoDB command mongod. mongo- utility for working with the database, the service itself mongod. You do not need to create anything previously in the database.

Examples of requests using httpie:

http POST http://localhost:1337/api/articles title=TestArticle author='John Doe' description='lorem ipsum dolar sit amet' images:='[{"kind":"thumbnail", "url":"http://habrahabr.ru/images/write-topic.png"}, {"kind":"detail", "url":"http://habrahabr.ru/images/write-topic.png"}]'
http http://localhost:1337/api/articles
http http://localhost:1337/api/articles/52306b6a0df1064e9d000003
http PUT http://localhost:1337/api/articles/52306b6a0df1064e9d000003 title=TestArticle2 author='John Doe' description='lorem ipsum dolar sit amet' images:='[{"kind":"thumbnail", "url":"http://habrahabr.ru/images/write-topic.png"}, {"kind":"detail", "url":"http://habrahabr.ru/images/write-topic.png"}]'
http DELETE http://localhost:1337/api/articles/52306b6a0df1064e9d000003


The project at this stage can take a look at GitHub .

5. Access control - OAuth 2.0, Passport.js


To control access, I will resort to OAuth 2. Perhaps this is redundant, but in the future this approach facilitates integration with other OAuth providers. Also, I did not find working examples of the user-password OAuth2 flow for Node.js. Passport.js
will directly monitor access control . For the OAuth2 server, a solution from the same author, oauth2orize, is useful . Users, tokens will be stored in MongoDB. First we need to install all the modules that we need:

  • Faker
  • oauth2orize
  • passport
  • passport-http
  • passport-http-bearer
  • passport-oauth2-client-password

Then, in mongoose.js you need to add schemes for users and tokens:

var crypto      = require('crypto');
// User
var User = new Schema({
    username: {
        type: String,
        unique: true,
        required: true
    },
    hashedPassword: {
        type: String,
        required: true
    },
    salt: {
        type: String,
        required: true
    },
    created: {
        type: Date,
        default: Date.now
    }
});
User.methods.encryptPassword = function(password) {
    return crypto.createHmac('sha1', this.salt).update(password).digest('hex');
    //more secure - return crypto.pbkdf2Sync(password, this.salt, 10000, 512);
};
User.virtual('userId')
    .get(function () {
        return this.id;
    });
User.virtual('password')
    .set(function(password) {
        this._plainPassword = password;
        this.salt = crypto.randomBytes(32).toString('base64');
        //more secure - this.salt = crypto.randomBytes(128).toString('base64');
        this.hashedPassword = this.encryptPassword(password);
    })
    .get(function() { return this._plainPassword; });
User.methods.checkPassword = function(password) {
    return this.encryptPassword(password) === this.hashedPassword;
};
var UserModel = mongoose.model('User', User);
// Client
var Client = new Schema({
    name: {
        type: String,
        unique: true,
        required: true
    },
    clientId: {
        type: String,
        unique: true,
        required: true
    },
    clientSecret: {
        type: String,
        required: true
    }
});
var ClientModel = mongoose.model('Client', Client);
// AccessToken
var AccessToken = new Schema({
    userId: {
        type: String,
        required: true
    },
    clientId: {
        type: String,
        required: true
    },
    token: {
        type: String,
        unique: true,
        required: true
    },
    created: {
        type: Date,
        default: Date.now
    }
});
var AccessTokenModel = mongoose.model('AccessToken', AccessToken);
// RefreshToken
var RefreshToken = new Schema({
    userId: {
        type: String,
        required: true
    },
    clientId: {
        type: String,
        required: true
    },
    token: {
        type: String,
        unique: true,
        required: true
    },
    created: {
        type: Date,
        default: Date.now
    }
});
var RefreshTokenModel = mongoose.model('RefreshToken', RefreshToken);
module.exports.UserModel = UserModel;
module.exports.ClientModel = ClientModel;
module.exports.AccessTokenModel = AccessTokenModel;
module.exports.RefreshTokenModel = RefreshTokenModel;


The virtual password property is an example of how mongoose can embed convenient logic in a model. About hashes, algorithms and salt - not this article, we will not go into details of the implementation.

So, the objects in the database:
  1. User - a user who has a name, password hash and salt of his password.
  2. Client - a client application that is granted access on behalf of the user. They have a name and a secret code.
  3. AccessToken - a token (bearer type) issued to client applications is limited in time.
  4. RefreshToken - another type of token, allows you to request a new bearer token without re-requesting a password from the user.


In config.json, add the token lifetime:
{
    "port" : 1337,
    "security": {
        "tokenLife" : 3600
    },
    "mongoose": {
        "uri": "mongodb://localhost/testAPI"
    }
}


We separate the OAuth2 server and the authorization logic into separate modules. Oauth.js describes the “strategies” of passport.js, we connect 3 of them - 2 to OAuth2 username-password flow, 1 to check the token.

var config                  = require('./config');
var passport                = require('passport');
var BasicStrategy           = require('passport-http').BasicStrategy;
var ClientPasswordStrategy  = require('passport-oauth2-client-password').Strategy;
var BearerStrategy          = require('passport-http-bearer').Strategy;
var UserModel               = require('./mongoose').UserModel;
var ClientModel             = require('./mongoose').ClientModel;
var AccessTokenModel        = require('./mongoose').AccessTokenModel;
var RefreshTokenModel       = require('./mongoose').RefreshTokenModel;
passport.use(new BasicStrategy(
    function(username, password, done) {
        ClientModel.findOne({ clientId: username }, function(err, client) {
            if (err) { return done(err); }
            if (!client) { return done(null, false); }
            if (client.clientSecret != password) { return done(null, false); }
            return done(null, client);
        });
    }
));
passport.use(new ClientPasswordStrategy(
    function(clientId, clientSecret, done) {
        ClientModel.findOne({ clientId: clientId }, function(err, client) {
            if (err) { return done(err); }
            if (!client) { return done(null, false); }
            if (client.clientSecret != clientSecret) { return done(null, false); }
            return done(null, client);
        });
    }
));
passport.use(new BearerStrategy(
    function(accessToken, done) {
        AccessTokenModel.findOne({ token: accessToken }, function(err, token) {
            if (err) { return done(err); }
            if (!token) { return done(null, false); }
            if( Math.round((Date.now()-token.created)/1000) > config.get('security:tokenLife') ) {
                AccessTokenModel.remove({ token: accessToken }, function (err) {
                    if (err) return done(err);
                });
                return done(null, false, { message: 'Token expired' });
            }
            UserModel.findById(token.userId, function(err, user) {
                if (err) { return done(err); }
                if (!user) { return done(null, false, { message: 'Unknown user' }); }
                var info = { scope: '*' }
                done(null, user, info);
            });
        });
    }
));


Oauth2.js is responsible for issuing and updating the token. One exchange strategy is to get a token by username-password flow, another one is to exchange refresh_token.

var oauth2orize         = require('oauth2orize');
var passport            = require('passport');
var crypto              = require('crypto');
var config              = require('./config');
var UserModel           = require('./mongoose').UserModel;
var ClientModel         = require('./mongoose').ClientModel;
var AccessTokenModel    = require('./mongoose').AccessTokenModel;
var RefreshTokenModel   = require('./mongoose').RefreshTokenModel;
// create OAuth 2.0 server
var server = oauth2orize.createServer();
// Exchange username & password for access token.
server.exchange(oauth2orize.exchange.password(function(client, username, password, scope, done) {
    UserModel.findOne({ username: username }, function(err, user) {
        if (err) { return done(err); }
        if (!user) { return done(null, false); }
        if (!user.checkPassword(password)) { return done(null, false); }
        RefreshTokenModel.remove({ userId: user.userId, clientId: client.clientId }, function (err) {
            if (err) return done(err);
        });
        AccessTokenModel.remove({ userId: user.userId, clientId: client.clientId }, function (err) {
            if (err) return done(err);
        });
        var tokenValue = crypto.randomBytes(32).toString('base64');
        var refreshTokenValue = crypto.randomBytes(32).toString('base64');
        var token = new AccessTokenModel({ token: tokenValue, clientId: client.clientId, userId: user.userId });
        var refreshToken = new RefreshTokenModel({ token: refreshTokenValue, clientId: client.clientId, userId: user.userId });
        refreshToken.save(function (err) {
            if (err) { return done(err); }
        });
        var info = { scope: '*' }
        token.save(function (err, token) {
            if (err) { return done(err); }
            done(null, tokenValue, refreshTokenValue, { 'expires_in': config.get('security:tokenLife') });
        });
    });
}));
// Exchange refreshToken for access token.
server.exchange(oauth2orize.exchange.refreshToken(function(client, refreshToken, scope, done) {
    RefreshTokenModel.findOne({ token: refreshToken }, function(err, token) {
        if (err) { return done(err); }
        if (!token) { return done(null, false); }
        if (!token) { return done(null, false); }
        UserModel.findById(token.userId, function(err, user) {
            if (err) { return done(err); }
            if (!user) { return done(null, false); }
            RefreshTokenModel.remove({ userId: user.userId, clientId: client.clientId }, function (err) {
                if (err) return done(err);
            });
            AccessTokenModel.remove({ userId: user.userId, clientId: client.clientId }, function (err) {
                if (err) return done(err);
            });
            var tokenValue = crypto.randomBytes(32).toString('base64');
            var refreshTokenValue = crypto.randomBytes(32).toString('base64');
            var token = new AccessTokenModel({ token: tokenValue, clientId: client.clientId, userId: user.userId });
            var refreshToken = new RefreshTokenModel({ token: refreshTokenValue, clientId: client.clientId, userId: user.userId });
            refreshToken.save(function (err) {
                if (err) { return done(err); }
            });
            var info = { scope: '*' }
            token.save(function (err, token) {
                if (err) { return done(err); }
                done(null, tokenValue, refreshTokenValue, { 'expires_in': config.get('security:tokenLife') });
            });
        });
    });
}));
// token endpoint
exports.token = [
    passport.authenticate(['basic', 'oauth2-client-password'], { session: false }),
    server.token(),
    server.errorHandler()
]


To connect these modules, add to server.js:

var oauth2          = require('./libs/oauth2');
app.use(passport.initialize());
require('./libs/auth');
app.post('/oauth/token', oauth2.token);
app.get('/api/userInfo',
    passport.authenticate('bearer', { session: false }),
        function(req, res) {
            // req.authInfo is set using the `info` argument supplied by
            // `BearerStrategy`.  It is typically used to indicate scope of the token,
            // and used in access control checks.  For illustrative purposes, this
            // example simply returns the scope in the response.
            res.json({ user_id: req.user.userId, name: req.user.username, scope: req.authInfo.scope })
        }
);


For example, the protection is at the localhost address: 1337 / api / userInfo.

To check the operation of the authorization mechanism, you should create a user and a client in the database. I will give an application on Node.js, which will create the necessary objects and remove unnecessary ones from the collections. It helps to quickly clear the database of tokens and users during testing, you, I think, will be enough for one launch :)

var log                 = require('./libs/log')(module);
var mongoose            = require('./libs/mongoose').mongoose;
var UserModel           = require('./libs/mongoose').UserModel;
var ClientModel         = require('./libs/mongoose').ClientModel;
var AccessTokenModel    = require('./libs/mongoose').AccessTokenModel;
var RefreshTokenModel   = require('./libs/mongoose').RefreshTokenModel;
var faker               = require('Faker');
UserModel.remove({}, function(err) {
    var user = new UserModel({ username: "andrey", password: "simplepassword" });
    user.save(function(err, user) {
        if(err) return log.error(err);
        else log.info("New user - %s:%s",user.username,user.password);
    });
    for(i=0; i<4; i++) {
        var user = new UserModel({ username: faker.random.first_name().toLowerCase(), password: faker.Lorem.words(1)[0] });
        user.save(function(err, user) {
            if(err) return log.error(err);
            else log.info("New user - %s:%s",user.username,user.password);
        });
    }
});
ClientModel.remove({}, function(err) {
    var client = new ClientModel({ name: "OurService iOS client v1", clientId: "mobileV1", clientSecret:"abc123456" });
    client.save(function(err, client) {
        if(err) return log.error(err);
        else log.info("New client - %s:%s",client.clientId,client.clientSecret);
    });
});
AccessTokenModel.remove({}, function (err) {
    if (err) return log.error(err);
});
RefreshTokenModel.remove({}, function (err) {
    if (err) return log.error(err);
});
setTimeout(function() {
    mongoose.disconnect();
}, 3000);


If you created the data with a script, the following authorization commands will suit you as well. Let me remind you that I am using httpie .

http POST http://localhost:1337/oauth/token grant_type=password client_id=mobileV1 client_secret=abc123456 username=andrey password=simplepassword
http POST http://localhost:1337/oauth/token grant_type=refresh_token client_id=mobileV1 client_secret=abc123456 refresh_token=TOKEN
http http://localhost:1337/api/userinfo Authorization:'Bearer TOKEN'


Attention! Be sure to use HTTPS on the production server, this is implied by the OAuth 2 specification. And do not forget about the correct password hashing. Implementing https with this example is not difficult, there are many examples on the network.
Let me remind you that all the code is contained in the repository on GitHub .
To work, you need to execute npm installin the directory, run mongod, node dataGen.js(wait for execution), and then node server.js.

If any part of the article should be described in more detail, please indicate this in the comments. The material will be processed and updated as reviews become available.

To summarize, I want to say that node.js is a cool, convenient server solution. MongoDB with a document-oriented approach is a very unusual, but undoubtedly useful tool, most of the features of which I have not yet used. Around Node.js is a very large community, where there are many open-source developments. For example, the creator of oauth2orize and passport.js, Jared Hanson, has done wonderful projects that make the implementation of properly protected systems as easy as possible.
Tags:

Also popular now: