Palantir – Part 4
February 7, 2011
Under a cloud
This section is subject to change as new server side systems get developed. But it’s helpful to test that Palantir is correctly communicating to the server and logging data to the database before going further ahead with the hardware. We’ll keep it brief though!
Setting up a host
Palantir refers to all connected Arduinos by a hostname. This is the short (and hopefully unique) key that you entered in for ARDUINO_ID in the sketch. Before you go any further you need to have this host name and your chosen password registered on the system – get in contact with Michael to do this. Once this is done, you can access the set up page here, giving your host for your username and the password you chose. Note down your authentication key, then open the Palantir sketch and change #define HTTPAUTH “xxx” so that xxx is that key. Load this to the Arduino and set it running. Assuming you’ve got a working Ethernet connection you should be able to find your hostname on the reports page and if you click on it, see the incoming data. If you don’t then something has gone wrong; i) reset the Arduino & Ethershield; ii) check you have lights on the shield socket; iii) check that the Arduino is talking to the world (maybe try reloading the web client example).
JSON
Nearly all data in Palantir is transmitted and stored in JSON format. This is a text-based, extensible format similar in concept to XML but less strictly structured. Unfortunately, it’s not the easiest for us poor humans to read, since the bracketing structure is a tad confusing but it’s terse, easily built by the Arduino or read by the server and can handle all the data we want to throw around. So, lets start with an example:
{ yourdino: [
{ id : "28E44FBB20027", vl : "25.62" },
{ id : "28B282BB20075", vl : "25.87" },
{ id : "L0", vl : "770" },
{ id : "P0", vl : "71" },
{ id : "H0", vl : "51" },
{ id : "AL", vl : "66" }
] }This is the structure of the data blocks your Arduino should be sending to the server every 20s or so. It’s been ‘prettified’ with additional white space to make it easier to read and you can see that it consist of first the hostname, then a series of name : value pairs for the sensor data. Sensors are referred to using the unique ID in the case of temperature or P0, 1, 2 etc. for the others. This is the same as being reported to the Serial output. Pair ‘AL’ refers to the saved set of alerts and can be ignored for now.
Once the server receives each data block, it stores it in a database (currently MySQL but other things are also being experimented with) along with a unique ID for that block, the hostname it came from and a timestamp. This data can then be accessed either directly by scripts (PHP, Javascript) running on the server, or via standardised interface.
Needing a REST
This isn’t the place to get too into the philosophy of Representational State Transfer aka ‘REST’ but enough to say that it’s an approach for sending and requesting data via the normal web POST/GET protocols. It allows anything that can talk ‘web’ to access the Palantir database. For example a GET request (just click on it!) to: http://www.fluffydragons.org/api/data/575236 gives:
{"glom":[{"id":"28E44FBB20027","vl":"25.75"},{"id":"28B282BB20075","vl":"26.37"},{"id":"28B65EBB200D6","vl":"23.00"},{"id":"L0","vl":"759"},{"id":"P0","vl":"147"},{"id":"P1","vl":"247"},{"id":"H0","vl":"53"},{"id":"AL","vl":"53"}]}
which is the (non-prettified) data being stored under unique ID ‘575236’. You’ll notice that this doesn’t give any additional meta information, it’s just the data as originally supplied by the Arduino.
A slightly different request, without the ID but including a parameter – /api/data?arduinoid=parrot will return the last data point for hostname ‘parrot’. You’ll notice that this time it includes some metadata, including the timestamp of the saved data block. Including further parameters – /api/data?arduinoid=parrot&before=2011-02-01%2000:00:00&limit=5 gives us (in this example) the five most recent data blocks before midnight on the 1st of February.
While GET requests (aka browser page requests) are the way to access the data, POST requests (as used in web forms) are used to save things to the database. This is how the Arduino sends its data and how a web or other software interface can make changes to the settings. We won’t get further into that since it involves security and other boring stuff but if you’re interested in the REST API, there’s live documentation of it here.
Sensors & alerts
So data is stored as it comes in from the Arduino, but how do we know what it means? If you looked at the API, you’ll have noticed there’s actually four types of information being stored: data, sensors, alerts and hosts. We understand host name and data but how do we define a set of sensors and alerts for them?
Sensors
If we go back to configuration page, click on your hostname and then on ‘sensors’, we should get a blank form page (after entering the authentication details of course!) Here we need to define the set of sensors that your Arduino is using, including any relevant metadata etc. An example of suitable JSON is:
{ yourdino: [
{ id: "2850E28320077", type: "temperature", description: "in the tank", unit: "degC" },
{ id: "289C7EC20097", type: "temperature", description: "ambient room", unit: "degC" },
{ id: "L0", type: "light", unit: "AU" },
{ id: "P0", type: "pressure", description: "internal filter", unit: "kPa", baseline: "39", scaling: "10.85" },
{ id: "P1", type: "pressure", description: "air pump", unit: "kPa", baseline: "31", scaling: "54.23" },
{ id: "H0", type: "hall effect", description: "external TetraTec filter", unit: "AU", baseline: "64" }
] }
There’s quite a lot of metadata here but the only required ones are ‘id’ and ‘type’ (which is one of: temperature | light | pressure | hall effect). The optional ‘description’ field is useful for adding a short human readable label, ‘unit’ is obvious (and may be AU for ‘arbitrary unit’), ‘baseline’ refers to the minimum value you get off the sensor when no physical signal is being measured, while ‘scaling’ is a multiple used to give an approximate conversion between the sensor value and the measurement in the unit given. These last two require some derivation, for ‘baseline’ the easiest approach is to watch the Serial data coming in with the appropriate sensors attached but the equipment they are monitoring turned off. We’re only bothering with ‘scaling’ for the pressure sensors and the values are roughly calculated from the sensor specifications as 10.85 (MPX5010DP) and 54.23 (MPX5050DP).
So copy the above into the blank form, change the hostname, IDs and descriptions as you like, then save it. The page should now show you the configured sensors and allow you to see the last saved values for them. If this didn’t work or if you have unconfigured sensors reported, then you probably got your JSON wrong! Correct it (using a text editor may help) and re-save.
Random aside: you’ll notice that it appears as if you are overwriting the old sensor JSON block each time you save. That’s not actually the case and instead every new block is saved separately in the database. This allows us to access previously defined sensor sets should we desire to – compare /api/sensors?arduinoid=glom&before=2011-02-01%2000:00:00 with /api/sensors?arduinoid=glom&after=2011-02-01%2000:00:00.
Alerts
We now need to do the equivalent job for the alerts JSON:
{ yourdino: [
{ id: "2850E28320077", test: "<", threshold: "24.4", alert: "low",
message: "Warning: temperature is too low!", buzzer: "true" },
{ id: "2850E28320077", test: ">", threshold: "27.6", alert: "high",
message: "Warning: temperature is too high!", buzzer: "true" },
{ id: "2828BC8320079", test: "<", threshold: "18.5", alert: "low",
message: "Warning: temperature is too low!", buzzer: "true", email: "true" },
{ id: "2828BC8320079", test: ">", threshold: "24.5", alert: "high",
message: "Warning: temperature is too high!", buzzer: "true", email: "true" },
{ id: "P0", test: "<", threshold: "100", alert: "low",
message: "Warning: pressure is too low!", buzzer: "true", email: "true" },
{ id: "H0", test: "<", threshold: "70", alert: "low",
message: "Warning: impeller has stalled!", buzzer: "true", email: "true" }
] }which is a touch longer… Going through the parameters: ‘id’ should match that defined in the sensors, ‘test’, ‘threshold’ and ‘alert’ define the machine readable test that needs to pass before the alert is triggered. So in the above case, if temperature sensor 2850E28320077 gets hotter than 27.6oC then a ‘high’ alert will go off. Where that alert triggers depends on the remaining parameters, buzzer: “true” means that this alert will be triggered on the Arduino and set off a buzzer ‘chirp’. An email: “true” flag will (when this function is actually added!) cause the server to send a warning email including the defined warning message. So the above JSON translates as: audible alarm if 24.4oC > 2850E28320077 > 27.6oC, 18.5oC > 2828BC8320079 > 24.5oC, P0 < 100 or H0 < 70. Also send email warnings in the latter two case. It also indicates that we don't care about setting an alert on the P1 sensors (for whatever reason).
So far, so complicated. Oh yeah and the baseline/scaling maths isn’t taken into account yet – we’ll up date this when it is. Anyway, copy the above, change it to match your equipments and IDs and save into the configuration page. Assuming everything worked, you should now have all the sensors with data coming in and all the alerts with a matching sensor. If not, it’s bonking time again…
What’s next?
Assuming you’ve vanquished the Cloud of Doom, you should have a Palantir happily reporting junk data to our server. Nifty. So how about we get off this silly breadboard and onto something more solid?
Leave Your Comment