March 24, 2015 at 15:00Wolfram Data Drop - New Wolfram Research Service
- Transfer
Translation post by Stephen Wolfram (Stephen by Wolfram) " of The Drop by Wolfram the Data Is Live! ". I express my deep gratitude to Kirill Guzenko for help with the translation.
Where should the data from the Internet of things go ? We have excellent technology at Wolfram Language for interpreting, visualizing, analyzing queries and other interesting things. But the question is, how should all this data from connected devices and all other sources get to where all of the above can be done with them? Today we are launching what, in my opinion, is an excellent solution to this problem: Wolfram Data Drop .
When I first started thinking about Data Drop, I saw it mainly as a convenient means of moving various data. But now that Data Drop has been created, I understand that it is much more than just a service for moving data. In fact, this is an important step in our ongoing journey of integrating computing and the real world.
So what is Wolfram Data Drop? At the functional level, it is a universal data aggregator designed to receive and organize data received from various sensors, devices, programs, people, or whatever. Moreover, the data is stored in the cloud in such a way that it can be used for calculation and processing as if it were stored on the device itself (seamless integration of calculations and data).
Our goal is to make Wolfram Data Drop data accessible as easily as possible from anywhere. You can use web API , email , Twitter , web form , Arduino , Raspberry Pi and the like. How are we going to gradually add more and more different opportunities for communication with other data collection systems - both software and hardware. It doesn't matter where the data comes from. The idea is that Wolfram Data Drop collects this data using the same algorithm: in a “databin” with a specific ID.
Below is an example of how this works. On my desk now lies this little device:
Every 30 seconds it receives data from tiny sensors that are on the right end, and then sends data via wifi and web API to the Wolfram Data Drop databin with ID “3pw3N73Q”. Like all databin, this databin has its own page on the Internet: wolfr.am/3pw3N73Q .
This page is a control center , thanks to which you can, for example, download the source data. But the most interesting fact is that the databin is fully integrated into the Wolfram Language. The main concept of Wolfram Language is that it is based on knowledge, contains within itself a huge amount of knowledge about the world, computing; about how that works.
For example, Wolfram Language has current stock price data ,earthquakes and many other things . But now he can know about things like the environmental conditions on my desk that were obtained through Wolfram Data Drop. In our case, we get data from the small device shown above.
Here's how it works. The following is a Wolfram Language symbol object called databin: You
can perform various actions on it. For example, below are graphs of the time series of data from the databin:
And here are histograms of values:
But here are the source data, presented in the form of a table:
What's really cool: a databin that can hold data from anywhere is only part of the language. And we can make calculations with this data as with anything else.
Here, for example, the minimum and maximum temperature functions on my table are given (MinMax - a new Wolfram Language function):
You can convert to other units of measurement ( % - corresponds to the result calculated in the previous step):
Let's now imagine the pressure function from time to time. Here it is:
Of course, Wolfram Knowledgebase has an archive of weather data. So, in Wolfram Language, we can simply request pressure at a point of our location for the period of time that is contained in the databin, and the results will be surprisingly similar:
If we want, we can do many interesting things with data: analyze time series , use machine algorithms training , create a model and much more. We can visualize the data in some sophisticated way . You can create structured queries, or queries in a natural language .
It should be noted that the data that we get from the databin comes with the units. Here is an example of a fundamental feature of Wolfram Data Drop: it is not just storing source data; the data is stored and contains the real meaning that is attached to it. This means that you can immediately understand where this data should be used.
To do this, we use our high-tech system: Wolfram Data Framework ( WDF ). Developed in close association with Wolfram | Alpha , we have developed our system of standards for the symbolic representation of data from the real world. And each databin in Wolfram Data Drop can use WDF to determine the meaning of the data (“ data semantics signature ”), which determines how this data will be interpreted, as well as how our automatic import systemand a natural language recognition system will process the new data that comes in.
The beauty of all this is that since the data ended up in Wolfram Data Drop, it can be universally accessible and interpreted for Wolfram Language and any system that uses this language. So, for example, any public databin in Wolfram Data Drop is available at any time through Wolfram | Alpha, just like through any intelligent assistants that use Wolfram | Alpha. Tell Wolfram | Alpha the name databin, and the system will create an automatic report with the analysis of the data contained: Using WDF, Wolfram Data Drop instantly works with 10,000 types of various physical dimensions
. But Data Drop is not limited to numbers and numerical values. You can download anything you want into the system. And since Wolfram Language is a symbolic language, it can handle all this with the same methods.
The Wolfram Data Drop automatically enters time data and, when possible, location data . In both cases, there are specific standards for presenting this data to the WDF. As well as chemical compounds , cities , classes , networks , or thousands of other things . You can also work with images uploaded to Wolfram Data Drop.
Somewhere in our quality assurance department there is a Raspberry Pi with a connected camera that takes pictures of our two fish every 10 minutes and writes it to the databin in Wolfram Data Drop:
Using the Wolfram Language, it's easy to assemble all these images into an interactive 3D image cube :
Or process images to get a map of where the fish spent their time:
We can do any kind of data analysis in Wolfram Language. But for me, the most exciting fact is how easy it is to put data from the real world into the system through Wolfram Data Drop.
In our office, various databin are commonly used. They are very easy to create and integrate with existing control and monitoring systems. We now have a databin for the server room (HVAC), for temperature sensors on the roof of our headquarters, for refrigerators in the rest room, for control of the Internet network, and for Data Drop itself. Every day, some new solutions come and go.
Many databin have been created for personal use. For me, for example, collecting and analyzing data is a long-standing hobby. And, in fact, I collected personal data about myselffor more than a quarter century. I can also say that in March 2015 there will be a step forward on a historical scale. Indeed, using Data Drop, it has become much easier to collect data, as a result of which the amount of information that I will collect will increase dramatically. I will be a man with at least twenty-five databin`ami ... and this is not the limit.
A very important fact is that everything in Wolfram Data Drop is stored in WDF , that is, all data is presented in a certain form, which means it is possible to work with different databin, compare, combine different values and get meaningful calculation results.
As long as you work with small amounts of data, Wolfram Data Drop is completely free and open, that is, anyone or anything caninstantly upload data there . Official users will be able to work with much larger amounts of data at a speed that, we believe, will constantly increase.
Wolfram Data Drop databin can be either public or private. It may also be open for changes or require authorization. Anyone can access the Wolfram Data Drop with our main cloud, the Wolfram Cloud . However, organizations that have their own Wolfram Private Cloud will soon be able to have their own private Data Drop, which works inside their own infrastructure.
So what does a typical Wolfram Data Drop workflow look like? It depends on what you do. And even with a single databin, judging from my experience, several different workflows are required.
It is very convenient to be able to take some kind of databin and immediately make calculations online in the Wolfram Language session , examine the data, record the results of work and create reports in a separate file.
However, it is often much more convenient when working with databin in automatic mode. For example, you can create a scheduled task, and the system will send a databin to the email. You can post data to Wolfram Cloud so that someone using CloudCDF can work with the data received over the Internet. You can make the report automatically generated and sent every time someone visits the page.
But the scope is not limited to the Internet. From the moment the report arrives at Wolfram Cloud, it immediately becomes available on mobile and wearable gadgets. The same applies to desktop computers.
You do not have to make a report yourself. Instead, you just need to have a program in Wolfram Language that will monitor the databin, then, for example, send a notification / alarm / perform any other action in case some combination of conditions is met.
You can make the databin public, thereby effectively publishing some data in this way. Or you can make the databin private, accessible only to the data source or to some third party you have designated. You can create an API that will receive data from the databin in its original / processed form, and you can call all this not only through the Internet, but also through any programming language or any software system.
A certain databin can receive data from only one source, or from one device, or collect data from different sources and act as an aggregator. Since each piece of data is provided with detailed metadata, you can always tell where it came from.
For several years, we have been pretty closely associated with companies that deal with various connected devices, in particular through our Connected Devices Project.. And many times I had a similar dialogue: the company talks about the wonderful device that it makes, which measures something very interesting. Then I ask - what happens to the data that this device registers? And, often, they replied that this was a matter of great concern to them, but they did not plan to hire separate teams to create cloud servers, applications, or other infrastructure for working with data.
One of the reasons we created Wolfram Data Drop was to share the best solution with these companies. They receive the data, then simply send it to the Data Drop, and from there the data already arrives on our cloud (or on a private cloud), where it can be easily analyzed, visualized, received and distributed through web pages, APIs, applications, or through no matter what.
I believe a large number of device manufacturers will use Wolfram Data Drop. They will be able to receive data in many ways, sometimes through the web API. Sometimes through a direct connection to the Wolfram Language system, for example, on the Raspberry Pi . Sometimes through Arduino , or through Electric Imp, or through any other hardware platform compatible with Data Drop. Data can be transmitted via telephones or other mobile devices. And sometimes already recorded data from another cloud can serve as a source.
We are not working with the lowest-level problems of receiving data from a device via wires, wifi, Bluetooth, or anything else.
There are various ways to access data between connected devices. Developers and researchers can access directly the Wolfram Cloud , or through the Wolfram Language desktop / online environment . Customer-oriented device-related companies can work, for example, through their own Wolfram Private Cloud. They can also access Data Drop using the Wolfram mobile app or their own program. Or through some kind of wearable device.
It happens that a company needs to collect data from different devices. Let's say to monitor the network or conduct some research. And again, their users may want to work directly with Wolfram Language, or through a portal / application.
When I first began to think over the concept of Wolfram Data Drop, I assumed that the data will mainly come from some automated devices. But now that we have Data Drop, I realized that it will be very useful to work with data that comes from people. This is a great way to collect answers, say, in a classroom, or collect feedback for a crowdsourcing project, store various personal notes, and the like. Once the data semantics signature for the databin is defined, Wolfram Data Drop can automatically generate a data presentation form that can be uploaded to a network or mobile device.
Forms can work with images, text, and many other things. In the case of text, our natural language interpretation systemcan receive data and interpret it in WDF and lead to a standard form.
Now that we have Wolfram Data Drop, I continue to find various applications for this system and I don’t understand at all - how could I do without this system before? As with everything in Wolfram Language, this is the story of automation: Wolfram Data Drop automates the entire messy set of actions that involves collecting and processing real-world data from various sources.
The ultimate goal for me is the ability to collect and analyze data of any kind without the need to create / use any other systems. For example, last week I ended up using Wolfram Data Drop to collect performance data on our cloud. In a normal situation, this task would be too complicated and cumbersome, I would not even consider the possibility of doing it myself. But with the help of Data Drop, installation and configuration took me only a few minutes, and now I can get the data that interests me.
I look forward to the implementation of the things that I intend to implement in Wolfram Data Drop, just as I see how people use this system. You can already try out the beta version that we launched today and send us feedback(coming with the Data Drop databin, of course). I hope it will be a little time before the databins are woven into the infrastructure of the world: another step forward in our long-term mission is to make the world computable ...
Where should the data from the Internet of things go ? We have excellent technology at Wolfram Language for interpreting, visualizing, analyzing queries and other interesting things. But the question is, how should all this data from connected devices and all other sources get to where all of the above can be done with them? Today we are launching what, in my opinion, is an excellent solution to this problem: Wolfram Data Drop .
When I first started thinking about Data Drop, I saw it mainly as a convenient means of moving various data. But now that Data Drop has been created, I understand that it is much more than just a service for moving data. In fact, this is an important step in our ongoing journey of integrating computing and the real world.
So what is Wolfram Data Drop? At the functional level, it is a universal data aggregator designed to receive and organize data received from various sensors, devices, programs, people, or whatever. Moreover, the data is stored in the cloud in such a way that it can be used for calculation and processing as if it were stored on the device itself (seamless integration of calculations and data).
Our goal is to make Wolfram Data Drop data accessible as easily as possible from anywhere. You can use web API , email , Twitter , web form , Arduino , Raspberry Pi and the like. How are we going to gradually add more and more different opportunities for communication with other data collection systems - both software and hardware. It doesn't matter where the data comes from. The idea is that Wolfram Data Drop collects this data using the same algorithm: in a “databin” with a specific ID.
Below is an example of how this works. On my desk now lies this little device:
Every 30 seconds it receives data from tiny sensors that are on the right end, and then sends data via wifi and web API to the Wolfram Data Drop databin with ID “3pw3N73Q”. Like all databin, this databin has its own page on the Internet: wolfr.am/3pw3N73Q .
This page is a control center , thanks to which you can, for example, download the source data. But the most interesting fact is that the databin is fully integrated into the Wolfram Language. The main concept of Wolfram Language is that it is based on knowledge, contains within itself a huge amount of knowledge about the world, computing; about how that works.
For example, Wolfram Language has current stock price data ,earthquakes and many other things . But now he can know about things like the environmental conditions on my desk that were obtained through Wolfram Data Drop. In our case, we get data from the small device shown above.
Here's how it works. The following is a Wolfram Language symbol object called databin: You
can perform various actions on it. For example, below are graphs of the time series of data from the databin:
And here are histograms of values:
But here are the source data, presented in the form of a table:
What's really cool: a databin that can hold data from anywhere is only part of the language. And we can make calculations with this data as with anything else.
Here, for example, the minimum and maximum temperature functions on my table are given (MinMax - a new Wolfram Language function):
You can convert to other units of measurement ( % - corresponds to the result calculated in the previous step):
Let's now imagine the pressure function from time to time. Here it is:
Of course, Wolfram Knowledgebase has an archive of weather data. So, in Wolfram Language, we can simply request pressure at a point of our location for the period of time that is contained in the databin, and the results will be surprisingly similar:
If we want, we can do many interesting things with data: analyze time series , use machine algorithms training , create a model and much more. We can visualize the data in some sophisticated way . You can create structured queries, or queries in a natural language .
It should be noted that the data that we get from the databin comes with the units. Here is an example of a fundamental feature of Wolfram Data Drop: it is not just storing source data; the data is stored and contains the real meaning that is attached to it. This means that you can immediately understand where this data should be used.
To do this, we use our high-tech system: Wolfram Data Framework ( WDF ). Developed in close association with Wolfram | Alpha , we have developed our system of standards for the symbolic representation of data from the real world. And each databin in Wolfram Data Drop can use WDF to determine the meaning of the data (“ data semantics signature ”), which determines how this data will be interpreted, as well as how our automatic import systemand a natural language recognition system will process the new data that comes in.
The beauty of all this is that since the data ended up in Wolfram Data Drop, it can be universally accessible and interpreted for Wolfram Language and any system that uses this language. So, for example, any public databin in Wolfram Data Drop is available at any time through Wolfram | Alpha, just like through any intelligent assistants that use Wolfram | Alpha. Tell Wolfram | Alpha the name databin, and the system will create an automatic report with the analysis of the data contained: Using WDF, Wolfram Data Drop instantly works with 10,000 types of various physical dimensions
. But Data Drop is not limited to numbers and numerical values. You can download anything you want into the system. And since Wolfram Language is a symbolic language, it can handle all this with the same methods.
The Wolfram Data Drop automatically enters time data and, when possible, location data . In both cases, there are specific standards for presenting this data to the WDF. As well as chemical compounds , cities , classes , networks , or thousands of other things . You can also work with images uploaded to Wolfram Data Drop.
Somewhere in our quality assurance department there is a Raspberry Pi with a connected camera that takes pictures of our two fish every 10 minutes and writes it to the databin in Wolfram Data Drop:
Using the Wolfram Language, it's easy to assemble all these images into an interactive 3D image cube :
Or process images to get a map of where the fish spent their time:
We can do any kind of data analysis in Wolfram Language. But for me, the most exciting fact is how easy it is to put data from the real world into the system through Wolfram Data Drop.
In our office, various databin are commonly used. They are very easy to create and integrate with existing control and monitoring systems. We now have a databin for the server room (HVAC), for temperature sensors on the roof of our headquarters, for refrigerators in the rest room, for control of the Internet network, and for Data Drop itself. Every day, some new solutions come and go.
Many databin have been created for personal use. For me, for example, collecting and analyzing data is a long-standing hobby. And, in fact, I collected personal data about myselffor more than a quarter century. I can also say that in March 2015 there will be a step forward on a historical scale. Indeed, using Data Drop, it has become much easier to collect data, as a result of which the amount of information that I will collect will increase dramatically. I will be a man with at least twenty-five databin`ami ... and this is not the limit.
A very important fact is that everything in Wolfram Data Drop is stored in WDF , that is, all data is presented in a certain form, which means it is possible to work with different databin, compare, combine different values and get meaningful calculation results.
As long as you work with small amounts of data, Wolfram Data Drop is completely free and open, that is, anyone or anything caninstantly upload data there . Official users will be able to work with much larger amounts of data at a speed that, we believe, will constantly increase.
Wolfram Data Drop databin can be either public or private. It may also be open for changes or require authorization. Anyone can access the Wolfram Data Drop with our main cloud, the Wolfram Cloud . However, organizations that have their own Wolfram Private Cloud will soon be able to have their own private Data Drop, which works inside their own infrastructure.
So what does a typical Wolfram Data Drop workflow look like? It depends on what you do. And even with a single databin, judging from my experience, several different workflows are required.
It is very convenient to be able to take some kind of databin and immediately make calculations online in the Wolfram Language session , examine the data, record the results of work and create reports in a separate file.
However, it is often much more convenient when working with databin in automatic mode. For example, you can create a scheduled task, and the system will send a databin to the email. You can post data to Wolfram Cloud so that someone using CloudCDF can work with the data received over the Internet. You can make the report automatically generated and sent every time someone visits the page.
But the scope is not limited to the Internet. From the moment the report arrives at Wolfram Cloud, it immediately becomes available on mobile and wearable gadgets. The same applies to desktop computers.
You do not have to make a report yourself. Instead, you just need to have a program in Wolfram Language that will monitor the databin, then, for example, send a notification / alarm / perform any other action in case some combination of conditions is met.
You can make the databin public, thereby effectively publishing some data in this way. Or you can make the databin private, accessible only to the data source or to some third party you have designated. You can create an API that will receive data from the databin in its original / processed form, and you can call all this not only through the Internet, but also through any programming language or any software system.
A certain databin can receive data from only one source, or from one device, or collect data from different sources and act as an aggregator. Since each piece of data is provided with detailed metadata, you can always tell where it came from.
For several years, we have been pretty closely associated with companies that deal with various connected devices, in particular through our Connected Devices Project.. And many times I had a similar dialogue: the company talks about the wonderful device that it makes, which measures something very interesting. Then I ask - what happens to the data that this device registers? And, often, they replied that this was a matter of great concern to them, but they did not plan to hire separate teams to create cloud servers, applications, or other infrastructure for working with data.
One of the reasons we created Wolfram Data Drop was to share the best solution with these companies. They receive the data, then simply send it to the Data Drop, and from there the data already arrives on our cloud (or on a private cloud), where it can be easily analyzed, visualized, received and distributed through web pages, APIs, applications, or through no matter what.
I believe a large number of device manufacturers will use Wolfram Data Drop. They will be able to receive data in many ways, sometimes through the web API. Sometimes through a direct connection to the Wolfram Language system, for example, on the Raspberry Pi . Sometimes through Arduino , or through Electric Imp, or through any other hardware platform compatible with Data Drop. Data can be transmitted via telephones or other mobile devices. And sometimes already recorded data from another cloud can serve as a source.
We are not working with the lowest-level problems of receiving data from a device via wires, wifi, Bluetooth, or anything else.
There are various ways to access data between connected devices. Developers and researchers can access directly the Wolfram Cloud , or through the Wolfram Language desktop / online environment . Customer-oriented device-related companies can work, for example, through their own Wolfram Private Cloud. They can also access Data Drop using the Wolfram mobile app or their own program. Or through some kind of wearable device.
It happens that a company needs to collect data from different devices. Let's say to monitor the network or conduct some research. And again, their users may want to work directly with Wolfram Language, or through a portal / application.
When I first began to think over the concept of Wolfram Data Drop, I assumed that the data will mainly come from some automated devices. But now that we have Data Drop, I realized that it will be very useful to work with data that comes from people. This is a great way to collect answers, say, in a classroom, or collect feedback for a crowdsourcing project, store various personal notes, and the like. Once the data semantics signature for the databin is defined, Wolfram Data Drop can automatically generate a data presentation form that can be uploaded to a network or mobile device.
Forms can work with images, text, and many other things. In the case of text, our natural language interpretation systemcan receive data and interpret it in WDF and lead to a standard form.
Now that we have Wolfram Data Drop, I continue to find various applications for this system and I don’t understand at all - how could I do without this system before? As with everything in Wolfram Language, this is the story of automation: Wolfram Data Drop automates the entire messy set of actions that involves collecting and processing real-world data from various sources.
The ultimate goal for me is the ability to collect and analyze data of any kind without the need to create / use any other systems. For example, last week I ended up using Wolfram Data Drop to collect performance data on our cloud. In a normal situation, this task would be too complicated and cumbersome, I would not even consider the possibility of doing it myself. But with the help of Data Drop, installation and configuration took me only a few minutes, and now I can get the data that interests me.
I look forward to the implementation of the things that I intend to implement in Wolfram Data Drop, just as I see how people use this system. You can already try out the beta version that we launched today and send us feedback(coming with the Data Drop databin, of course). I hope it will be a little time before the databins are woven into the infrastructure of the world: another step forward in our long-term mission is to make the world computable ...