OpenSensorData.net


OpenSensorData.net is a hosting service where you can store streams of data that come from any sensor or sensor networks. The use of OpenSensorData.net is free (like in free beer). And the software is Free (like in freedom), under the GPLv3. You can download the code and install it on your own servers (requirements: Apache/PHP/MySQL).

Storing your data on OpenSensorData.net

  1. First you must know your rights.
  2. Then you must create an account at http://opensensordata.net/createaccount.php
  3. The account creation will generate an API key. Write it down. You will need it to access the services.
  4. Make sure you understand the basic concepts.
  5. Define your datastreams and groups of datastreams.
  6. Upload your data.
  7. Download your data.
  8. View your data.

Know your rights

The first thing that is important to know is that we store all the data that you upload to OpenSensorData.net (that’s what it’s for) AND that we make all of this data freely available to any one under the Open Database License. The full text of the Open Database License can be found at http://opendatacommons.org/licenses/odbl/. Any rights in individual contents of the database are licensed under the Database Contents License whose text can be found http://opendatacommons.org/licenses/dbcl/. If you find it difficult to comply with this, then you should set up your own data servers.

The second important thing is that at any time you can ask us to remove ALL of the data associated with your account. We will make it possible for you to do this yourself in the future. For now, however, you have to send your request by email to the following address: someone AT opensensordata.net.

The source code of OpenSensorData.net is available under version 3 of the GNU General Public License. That means you can use, study, and modify it, and that you redistribute modified copies of the source code. The lastest version of the code is available here.

The basic concepts

A datapoint consists of the time at which the measurement took place and its output value. For example, on Sunday, March 3rd 2013, at 11.55 am, the temperature sensor at my home measured 8.63 degrees celsius. The data point consists of the date (2013-03-10 11:55:00, for the precise format of the date, see details below), and its value (8.63).

A datastream is a series of datapoints. To add a datapoint to a datastream, you must first create the datastream. The minimum information you have to specify is a name, but a longer discription is welcome to. In addition, you can add information such as the geographical location of the sensor, and the timezone in which it is situated. All datastreams have a number that uniquely identifies it. To add a datapoint you must specify the datastream in which you want to insert it. You can pass along the datastream id, but there are other ways too (see “Upload your data”).

A group is a set of datastreams that you would like to keep together. Nearly all sensor boxes or data loggers will generate several datastreams and it obvious that you want to work with these datastreams together. Groups are flexible and it is also possible to create a group to bundle datastreams from different locations and from different users.

The creation of datastreams and groups is handled in the next section.

Define your datastreams and groups of datastreams

In most cases, you will not have to create groups and datastreams yourself. If the sensor box that user OpenSensorData is well designed, you should only provide the API key and a group name, and the device or software will create the group and its datastreams automagically.

In the short future, we will provide a online editor to manipulate groups and datastreams, however.

To create a groups and its datastreams, you must send a JSON description to the server over HTTP. Note that you must always specify the API kay in the X-OpenSensorData-Key header. The group definition contains a name and a description. A name can only be used once per account to label a group. So if I have a sensor box and it uses group name ’greenhouse’, if I install a second greenhouse I will have to choose another name.

The set of datagroups are coded in a JSON array, and for each you must provide its name and description. The datagroups will be automatically created for you.

The following example shows the complete HTTP request, with the headers, following by and empty line, and then the JSON description in the body.

PUT http://opensensordata.net/groups/ HTTP/1.1  
Host: opensensordata.net  
Content-Type: application/json  
Content-Length: 100000  
X-OpenSensorData-Key: 12345678-xxxx-yyyy-zzzz-123456789012  
 
{  
    "name": "greenhouse",  
    "description": "The sensorbox of the greenhouse on my terrace",  
    "datastreams": [  
        {  
            "name": "temperature",  
            "description": "Temperature"  
        },  
        {  
            "name": "Humidity",  
            "description": "Humidity"  
        }  
    ]  
}

A simple way to create the group, is to use telnet, connect to opensensordata.net on port 80, and then copy/paste the example above.

It is possible to add geolocation data to the datastream description. This allows the datastream to be positioned on a map. two extra fields are used, the longitude and latitude. You can specify a timezone. This can be useful for devices without a real-time clock. Lastly, you can specify a the unit in which the value is measured. For a complete list of currently supported units, see the list at the end of the documentation.

    ...  
    "datastreams": [  
        {  
            "name": "temperature",  
            "description": "Temperature"  
            "longitude": "2.363471",  
            "latitude": "48.917536",  
            "timezone": "+1",  
            "unit": "Celsius"  
        },  
    ...

All the error handling in OpenSensorData.net is handled through the HTTP status codes. This makes it easier to communicate to the server in scripts. Status code 200 indicates that the request was handled successfully. Pretty much anything else will indicate an error. The body of the response will then contain a simple plain text error message.

If all goes well, the request above should have returned:

HTTP/1.1 200 OK  
Content-Length: xxx  
Content-Type: text/html  
 
{  
    "id": "387",  
    "name": "greenhouse",  
    "description": "The sensorbox of the greenhouse on my terrace",  
    "datastreams": [  
        {  
            "id": "9347",  
            "name": "temperature",  
            "description": "Temperature"  
        },  
        {  
            "id": "9348",  
            "name": "Humidity",  
            "description": "Humidity"  
        }  
    ]  
}

As you can see, the server returns the same JSON structure, but with an additional ID field for the group and the new datastreams. These IDs uniquely define the objects and can be used in other requests, to upload data, for example.

Variations of the PUT request can be used to modify the definition of an existing group. To update a group, or datastream, you must explicitely specify the IDs. In result of the following request, the name of the group will be changed from ’greenhouse’ to ’SmallGreenhouse’. Note that no definition of the datastreams is given. As a result, no changes to the set of datastreams of the group will be made. The server response will contain the complete description of the group in JSON, like above.

PUT http://opensensordata.net/groups/ HTTP/1.1  
Host: opensensordata.net  
Content-Type: application/json  
Content-Length: 100000  
X-OpenSensorData-Key: 12345678-xxxx-yyyy-zzzz-123456789012  
 
{  
    "id": "387",  
    "name": "SmallGreenhouse",  
}

Had we given a description of the group’s datstreams, then these would have been updated, too. For example, you can remove all the datastreams from the group using the request below, in which the datastreams field is explicitely set to an empty array. The datastreams themselves will not be deleted. They will simply be dissociated from the group.

PUT http://opensensordata.net/groups/ HTTP/1.1  
Host: opensensordata.net  
Content-Type: application/json  
Content-Length: 100000  
X-OpenSensorData-Key: 12345678-xxxx-yyyy-zzzz-123456789012  
 
{  
    "id": "387",  
    "name": "greenhouse",  
    "description": "The sensorbox of the greenhouse on my terrace",  
    "datastreams": []  
}

Define your datastreams directly

It is not necessary to create a group to create datastreams. Datastreams can be created directly using a PUT request similar to the one for groups. The URL to which you must send your request is http://opensensordata.net/datastreams/. Here’s an example:

PUT http://opensensordata.net/datastreams/ HTTP/1.1  
Host: opensensordata.net  
Content-Type: application/json  
Content-Length: 100000  
X-OpenSensorData-Key: 12345678-xxxx-yyyy-zzzz-123456789012  
 
{  
    "name": "temperature-4",  
    "description": "Temperature",  
    "longitude": "2.363471",  
    "latitude": "48.917536",  
    "timezone": "+1"  
}

The name of the datastream is unique for every user, i.e.. a user X can only create one datastream called Y. The response of the request above will return the JSON description of the new datastream, including its unique, global ID number. When the datastream is created, any follow-up PUT request will update the description of the datastream, except its ID.

Upload your data

You can upload a set of datapoints through the following URL: http://opensensordata.net/upload. The upload interface is designed to accept many datapoints that have been produced by a sensor device and that are uploaded through software, not by humans manually.

To upload data, you must send an HTTP POST request to http://opensensordata.net/upload, with all the datapoints in the body of the request. The format of the datapoints is the CSV file format (Comma Separated Values). This is a textual file format in which each line represents one datapoint. Each line should contain at least three fields: the ID of the datastream, the date, and the sensor value. All field values of the datapoint are separated by commas.

The following example makes this clear. The first five lines in the example are the HTTP request headers. You can pass the key of your account in the X-OpenSensorData-Key header.

POST /opensensordata.net/upload HTTP/1.1  
Host: opensensordata.net  
Content-Type: text/csv  
Content-Length: 100000  
X-OpenSensorData-Key: 12345678-xxxx-yyyy-zzzz-123456789012  
 
1,2013-03-03T18:29:43,6.875  
2,2013-03-03T18:29:43,6.4375  
3,2013-03-03T18:29:43,61.054  
4,2013-03-03T18:29:43,61.054  
5,2013-03-03T18:29:43,70.4128  
6,2013-03-03T18:29:43,44.8116  
7,2013-03-03T18:29:43,1021.48  
8,2013-03-03T18:29:43,131.768  
1,2013-03-03T18:30:01,6.875  
2,2013-03-03T18:30:01,6.4375  
3,2013-03-03T18:30:01,61.054  
4,2013-03-03T18:30:01,61.054  
5,2013-03-03T18:30:01,70.4128  
6,2013-03-03T18:30:01,44.8116  
7,2013-03-03T18:30:01,1021.48  
8,2013-03-03T18:30:01,131.768

The first column identifies the datastream. It is possible to mix datapoints of several datastreams together in the same request. The second column is the timestamp. OpenSensorData.net uses the ISO_8601 standard to encode the date and time. The last column is the value.

It is also possible to add geolocation data, for those of you who move around with the sensor device. The fourth column should contain the longitude, and the fifth the latitude:

1, 2013-03-03T18:29:43, 6.875, 2.363471, 48.917536

The key of your account can be passed in the X-OpenSensorData-Key header. It is also possible to pass the key as a query string. The following example uploads a CSV file directly using the wget program:

wget --post-file=data.csv \  
  --output-document=data.out \  
  "http://opensensordata.net/upload?key=12345678-xxxx-yyyy-zzzz-123456789012"

More on the datastream identification

The unique, numerical datastream identifier is the shortest way to specify the datastream. There is a second way to identify the datastream that is more human-readable. You can combine the account name, the group name, and the datastream name to identify the stream, as in the following example:

peter/greenhouse/temperature, 2013-03-03T18:29:43, 6.875, 2.363471, 48.917536

Default geolocation and timezone

If the geolocation or the timezone is not specified in a datapoint, then the geolocation and timezone of the datastream will be used.

Download your data

To obtain all the data of a datastream with ID [id], use the following URL: http://opensensordata.net/datastreams/[id].csv. We mentioned above that a datastream can also be identified using the account name and so on. This can be used here as well: http://opensensordata.net/datastreams/peter/greenhouse/temperature.csv.

If you want to select a specific time slice, you can extend the URL as follows: http://opensensordata.net/datastreams/[id]/[year]/[month]/[day]/[hour]/[minute]/[second].csv. It is not necessary to specify down to the second. Any intermediate range value will be exepted. For example, to get the data from March 2013, you can use: http://opensensordata.net/datastreams/peter/greenhouse/temperature/2013/03.csv.

View your data

If you use OpenSensorData.net for an application, you will probably have you own visualisation tools. Your app obtains the data through the URL explained above and displays the results in whatever way that makes sense. It is still handy to be able to inspect the data online at OpenSensorData.net. also so other people can view your data.

We currently use dygraph to make the figures. The URLs are quite similar to the ones above: http://opensensordata.net/viewdata/[id].csv. For example, http://opensensordata.net/viewdata/peter/greenhouse/temperature/2013/06.csv.

It is also possible to view all the datastreams in a group: http://opensensordata.net/viewgroup/peter/greenhouse/2013/06.csv.

Photostreams

Besides datastreams, you can also create photostreams to upload the output of image sensors. Photostreams are very similar to datastreams, except for uploading the data which must be done one photo at a time (see below).

The URL to define or modify a photostream description is http://opensensordata.net/photostreams/. Here’s an example:

PUT http://opensensordata.net/photostreams/ HTTP/1.1  
Host: opensensordata.net  
Content-Type: application/json  
Content-Length: 100000  
X-OpenSensorData-Key: 12345678-xxxx-yyyy-zzzz-123456789012  
 
{  
    "name": "webcam",  
    "description": "The webcam inside the greenhouse",  
    "longitude": "2.363471",  
    "latitude": "48.917536",  
    "timezone": "+1"  
}

The name of the photostream must be unique for every user, i.e.. a user X can only create one datastream called Y. The response of the request above will return the JSON description of the new photostream, including its unique, global ID number. When the photostream is created, any follow-up PUT request will update the description of the photostream, except its ID.

Like datastreams, photostreams can be created and modified as part of a group. The processing of the group description, looks for the ’photostreams’ field, which must be an array:

PUT http://opensensordata.net/groups/ HTTP/1.1  
Host: opensensordata.net  
Content-Type: application/json  
Content-Length: 100000  
X-OpenSensorData-Key: 12345678-xxxx-yyyy-zzzz-123456789012  
 
{  
    "name": "greenhouse",  
    "description": "The sensorbox of the greenhouse on my terrace",  
    "datastreams": [  
        {  
            "name": "temperature",  
            "description": "Temperature"  
        },  
        {  
            "name": "Humidity",  
            "description": "Humidity"  
        }  
    ],  
    "photostreams": [  
        {  
            "name": "webcam",  
            "description": "The webcam inside the greenhouse"  
        }  
    ]  
}

The JSON description of a particular photostream can be obtained using the URL http://opensensordata.net/photostreams/id.json. The ID is the unique numerical ID of the photostream, or the combination of the username, the name of a group, and the name of the photostream separated by slashes. For example, http://opensensordata.net/photostreams/peter/greenhouse/webcam.json, or http://opensensordata.net/photostreams/1.json.

For upload a JPEG photo, use a PUT or a POST request to the address http://opensensordata.net/photo/stream-id/timestamp.jpg. The stream-id is, like above, the numerical ID or the combination of user, group, and stream name. The timestamp is a concatenation of the date and time. For example, 20130511-170002.jpg. The year is coded in 4 digits, all the other fields use 2 digits.

The body of the request should contain the image data, without any additional information. This makes it easier for small clients to upload images. To push images from a shell script to the server, you can use, for example, the following command:

wget --post-file=image.jpg \  
  --header "X-OpenSensorData-Key: 12345678-xxxx-yyyy-zzzz-123456789012" \  
  http://opensensordata.net/photo/1/20130524-154300.jpg

Currently only JPEG photos are accepted. We have also introduced some upload limits to reduce the exposure to software glitches or abuse of our service. The maximum image size is limited to 2MB and the maximum number of uploads that you can do per week per account is 100. If these limits are too constraint for your project, you should contact us and we will relax them.

You can retreive an image using the same URL as in the upload. For example, the image uploaded above is available at http://opensensordata.net/photo/1/20130524-154300.jpg. The server also makes a small (320x240) and a thumbnail (100x75) version of the image available. They have the “_s” and “_t” postfix, respectively: 20130524-154300_s.jpg for the small image and 20130524-154300_t.jpg for the thumbnail. The name 20130524-154300_o.jpg can also be used for the original image.

The list of images in a photostream can be retreived using http://opensensordata.net/photos/stream-id.json,txt. The first options returns a JSON array; the second the flat text with one photo URL per line.

The URL can contain a specific date, similar to getting the datapoints from a datastream. To get the list of photos from May 24th 2013, use http://opensensordata.net/photos/1/2013/05/24.txt.

Error handling

For all requests, you should check the HTTP status code of the response. If the server returns ”HTTP/1.1 200 OK”, all went fine. Otherwise, the HTTP status class will give you idea of what went wrong. Most likely, one of the following three errors occured: unauthorized access, a bad request, or an internal server error. More details about the error can be found in the body of the response, which contains a plain text message, like in the example below:

HTTP/1.1 400 Bad Request  
Content-Length: xxx  
Content-Type: text/plain  
 
Invalid datastream ID: 1932

Summary of the API

Create/update datastream
Method: PUT
URL: http://opensensordata.net/datastreams/
Format: JSON
Returns: datastream description, or HTTP error code

Upload data
Method: PUT
URL: http://opensensordata.net/upload
Format: CSV
Returns: HTTP status code

Get the data of the datastream
Method: GET
URL: http://opensensordata.net/datastreams/[id].csv
URL: http://opensensordata.net/datastreams/[id]/year.csv
URL: http://opensensordata.net/datastreams/[id]/year/month/.../second.csv
ID: the datastream’s unique numerical ID, or the combination of username/groupname/datastreamname
Format: CSV
Returns: the data

Get the datastream description
Method: GET
URL: http://opensensordata.net/datastreams/[id].json
ID: datastream unique ID, or username/groupname/datastreamname
Format: JSON
Returns: the datastream description, or HTTP error code

Create/update group
Method: PUT
URL: http://opensensordata.net/groups/
Format: JSON

Get the group description
Method: GET
URL: http://opensensordata.net/groups/[id].json
ID: the group’s unique numerical ID or username/groupname.
Format: JSON

Create/update photostream
Method: PUT
URL: http://opensensordata.net/photostreams/
Format: JSON
Returns: The new photostream description, or HTTP error code

Get the photostream description
Method: GET
URL: http://opensensordata.net/photostreams/id.json
with id: the photostream’s unique ID, or username/group-name/photostream-name
Format: JSON
Returns: The photostream description, or HTTP error code

Upload a photo
Method: PUT
URL: http://opensensordata.net/photo/id/yearmonthday-hourminutesecond.jpg
with id: the photostream’s unique ID, or username/group-name/photostream-name
Format: JPG
Returns: HTTP status code

Get a photo
Method: GET
URL: (orig version): http://opensensordata.net/photo/id/yearmonthday-hourminutesecond.jpg
or (orig version, explicite): http://opensensordata.net/photo/id/yearmonthday-hourminutesecond˙o.jpg
or (small version, 240x180): http://opensensordata.net/photo/id/yearmonthday-hourminutesecond˙s.jpg
or (thumbnail version, 100x75): http://opensensordata.net/photo/id/yearmonthday-hourminutesecond˙t.jpg
Format: JPG
Returns: HTTP status code

Get the list of photos in a photostream
Method: GET
URL: http://opensensordata.net/photostreams/id.json,txt
URL: http://opensensordata.net/photostreams/id/year.json,txt
URL: http://opensensordata.net/photostreams/id/year/month/.../second.json,txt
ID: the photostream’s unique numerical ID, or the combination of username/groupname/photostreamname
Format: JSON or text
Returns: JSON: An array of the photo descriptions; text: one photo URL per line

The currently available units

The specify the unit of the datapoints, set the ’unit’ field in the in a datastream using the short names of the table below.

DimensionShort nameLong name Comment




none none none
time s second
time ms millisecond
time us microsecond
time min minute
time h hour
time day day
time year year
length m meter
length mm millimeter
length cm centimeter
length km kilometer
length in inch
length ft foot
length yd yard
length mi mile
length ly lightyear
mass g gram
mass mg milligram
mass kg kilogram
mass ton ton imperial/long ton
mass t tonne metric/short tonne
mass lb pound
temperature Celsius degree Celsius
temperature Fahrenheitdegree Fahrenheit
temperature K kelvin
force N newton
energy J joule
energy kWh kilowatt-hour
power W watt
electrical current A ampere
electromotive forceV volt
area m2 square meter
area mm2 square millimeter
area cm2 square centimeter
area km2 square kilometer
area in2 square inch
area ft2 square foot
area mi2 square mile
volume m3 cubic meter
volume cm3 cubic centimeter
volume L liter
volume gal gallon
pressure Pa pascal
pressure hPa hectopascal
pressure mbar millibar
plane angle rad radian
solid angle sr steradian
frequency Hz hertz
humidity g/kg absolute humidity
humidity RH% relative humidity
percentage % percent
intensity dB decibel
loudness dB SPLdecibel sound pressure level
loudness dBA A-weighted decibel
electric charge C coulomb
electric charge Ah ampere-hour
capacitance F farad
electric resistanceohm ohm
electric conductance S siemens
magnetic flux Wbweber
magnetic flux densityT tesla
inductance H henry
luminous intensity cd candela
luminous flux lm lumen
illuminance lx lux
catalytic activity kat catalytic activity

Copyright

Copyright 2013, Sony Computer Science Laboratory Paris. This documentation is available under the GNU Free Documentation License.