Deform: simple database as a service
Imagine a database that can validate JSON data using the JSON Schema , process events, and process data. And if the idea is implemented as a service using mongodb ?
We have developed just such a service. Our side projects use it:
- docast.me - any data sources can be listened to / viewed in your favorite podcasting application
- watchlater - solution to watch later youtube problem
- warhealth - mobile application
Under the cut, a brief description of the main features of the service with examples.
This article contains sections:
- JSON - JSON documents
- Validation - data validation
- Files - work with files
- Processing - data processing
- WebHook - event handling
Json
All entities in the service are JSON documents. Thanks to this, mongodb came up to us perfectly .
You can work with documents through HTTP REST Api, CLI and Python clients.
I prefer the CLI.
Here are some examples of its use:
- Create document in collection
venues
deform document create -c venues -d '{"text":"hello world"}'
- Find out how many users are in the project
$ deform documents count -c my_project_users
108
Validation
I will say right away - the choice fell on JSON Schema draft v4. This diagram provides a lot of useful information.
Let's say that 2 documents come to you:
{
"name": "Товар за 1 руб",
"price": 1.0,
"currency": "RUB"
}
{
"name": "1$ stuff",
"price": 1.99,
"currency": "United States Dollar"
}
And we need the second document not to go to the database - the currency (property currency
) was transferred from him incorrectly .
Let's create a scheme according to which the first document passes, and the second does not.
{
"description": "General goods schema",
"type": "object",
"properties": {
"name": {
"type": "string"
},
"price": {
"type": "number"
},
"currency": {
"type": "string",
"enum": ["RUB", "USD", "CAD", "GBP"]
}
},
"required": ["name", "price", "currency"]
}
Pay attention to currency
its property enum
. Only documents in which the currency matches the lines "RUB", "USD", "CAD", "GBP" will be considered correct.
Files
Deform supports file handling. All your files will be documents in the collection _files
. And in the document itself, where you attached the file, it will also be available as an object.
Let's add a new property to the goods scheme - the product image.
{
"description": "General goods schema",
"type": "object",
"properties": {
"name": {
"type": "string"
},
"price": {
"type": "number"
},
"currency": {
"type": "string",
"enum": ["RUB", "USD", "CAD", "GBP"]
},
"image": {
"type": "file"
}
},
"required": ["name", "price", "currency"]
}
Now we can upload a picture for this product.
Using the CLI, create a document:
$ deform document create -c goods -d '{
"name": "Teapot",
"price": 11.99,
"currency": "GBP",
"image": @"/tmp/teapot.png"
}'
As a result, we get
{
"_id": "576ffce308888f000599ee17",
"name": "Teapot",
"price": 11.99,
"currency": "GBP",
"image": {
"_id": "5772db5308888f000599f095",
"collection_id": "goods",
"content_type": "image/png",
"date_created": "2016-06-25T18:31:12.356Z",
"document_id": "576ec022bd4db46b765ae94a",
"last_access": "2016-06-25T17:32:18.347Z",
"md5": "a8eda376612338e0286ff1c1a725b111",
"name": "teapot.png",
"size": 16214
}
}
You can get the contents of the file both from the collection with files and from the product document itself.
Link to the documentation section with files
Processing
Deform can process data. Full list of handlers .
Let's say that we need to change the size of the image of the product at the time of its creation.
The processor will help us resize
with this (at the moment it works only with images)
The scheme will turn into:
{
"description": "General goods schema",
"type": "object",
"properties": {
"name": {
"type": "string"
},
"price": {
"type": "number"
},
"currency": {
"type": "string",
"enum": ["RUB", "USD", "CAD", "GBP"]
},
"photo": {
"type": "file"
},
"300x150": {
"type": "file",
"processors": [
{
"name": "resize",
"in": {
"original_image": {
"property": "photo"
},
"size": {
"value": [300,150]
}
}
}
]
}
}
}
Link to processing documentation section
Webhook
Our webhooks write history and they have events. They also have headers, methods and their own validation of matching documents.
Suppose we need a notification in slack that the product was created. The hook will look like this:
{
"name": "Slack notification",
"url": "https://hooks.slack.com/services/....",
"method": "POST",
"triggers": ["created"],
"collection": "slack_notifications"
}
Pay attention to the property triggers
- in this case only created documents will trigger the hook.
Now let's say that we only need notifications about goods where the price is higher 1000
and the currency USD
:
{
"name": "Slack notification",
"url": "https://hooks.slack.com/services/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",
"method": "POST",
"triggers": ["created"],
"collection": "slack_notifications",
"condition": {
"type": "object",
"additionalProperties": true,
"price": {
"type": "number",
"minimum": 999
},
"currency": {
"type": "string",
"enum": ["USD"]
}
}
}
If the document is correct for the scheme inside condition
, the hook will work out.
Link to the documentation section with hooks
Conclusion
This article is a short overview of the main features of the project.
If you are interested in it - I will be just any feedback, including objective criticism: D
A few links on the project:
- CLI - CLI for the service
- Python - Python library for use in your projects
- Documentation