- This topic has 48 replies, 4 voices, and was last updated 10 years, 2 months ago by
.
-
Posts
-
I have an RPi based Emoncms system logging energy consumption. I am using MQTT and Node-Red to collect the data and distribute it to Emoncms and other applications (eg I can send alerts to email, or to XBMC etc). It thought it would be nice to be able to use Emoncms to log the data from my Opensprinkler-pi system, as the Emoncms system has a hard drive. MQTT and Node-Red make this easy to do. I have done a test and I can publish messages from my Opensprinkler Pi system to the MQTT broker running on the Emoncms system using a very simple python script. Node-Red then allows these messages to be reformatted and passed on to other destinations eg email, sms, twitter, Emoncms, XBMC, and so on. This could easily be two way. eg control Opensprinkler from other systems..
The code to publish MQTT messages is as simple as:
import paho.mqtt.publish as publish
publish.single(“OSPI/topic”, “My Message”, hostname=”192.168.x.x”)
This is all that is needed (hostname is the host name or ip address of the MQTT broker). Node Red then just subscribes to this topic and routes the message (with some reformatting if necessary) to other systems.
I am wondering about the best way to integrate this into Opensprinkler. A plugin seems to be desirable so that I don’t have to modify the basic code. If I make a plugin, is this called once off during initialisation, or is it called every time through the main loop of the interval program? Can I intercept the log messages in my plugin and reformat them for MQTT?
Some useful links
Node Red.
https://learn.adafruit.com/raspberry-pi-hosting-node-red/what-is-node-red Adafruit have a tutorial on installing node red.
Installing MQTT. Note – you only need to complete the basic install instructions here as the security certificates etc is for another project.
http://jpmens.net/2013/09/01/installing-mosquitto-on-a-raspberry-pi/
EmonCMS enenergymonitor.org/
MQTT Client
https://pypi.python.org/pypi/paho-mqtt#multiple (if you download the source the examples directory ahs some sample code).
Node Red —
node-red.org
Ian
I have a “proof of concept” working. Whenever entries are placed in the log, a message is published on an MQTT broker, and Node-Red sends this to a Raspberry Pi running XBMC. This causes a popup with the log entry to appear on my TV. Although I doubt I want to have this popup appearing in the long term it is useful when testing! Node-Red can send it elsewhere just as easily.
The code is
– import the MQTT library into helpers.py (1 line),
– in log_run reformatting the log entry as an MQTT message (1 line),
– publishing the message in log_run (1 line)
– and in Node Red I had to reformat the message to a format suitable for the XBMC addon I use (a couple of lines).
If there is interest I will provide more detail.
To do this I had to modify log_run in helpers.py so that when it inserts a log entry it publishes it to MQTT. I would prefer not to modify core code and have this option as a separate plugin. Is this possible?
Also I noticed that when using the Android Opensprinkler application, manually running a station does not result in a log entry. Is this a bug?
Hi Ian,
This sounds very interesting.
You should be able to make a plugin that does what you want without modifying the log_run function.
The email_adj.py plugin contains some code that could give you some ideas. It checks each station once a second to determine when a change happens. Actually there is an easier way. The list gv.srvals holds the state of the stations/zones (see gv_reference.txt in the OSPi directory for more.). You could copy that list then use a loop to check when a change happens:
import time from gv import srvals, lrun zone_state = srvals[:] while zone_state == srvals: time.sleep(1)The list gv.lrun holds the data that is used to create log entries in log_run(). You could use that even if logging is disabled.
As far as the Android app goes, I think it uses it’s own logging functions separate from the one in ospi. There are some issues with the logging in ospi that will be addressed soon so log_run() will likely change in the near future.
Dan
Thanks Dan
Your response started me thinking. That plus the fact that I have been reading the web api document has led me to realise I dont need to modify Opensprinkler at all! I can get the station status via the web api from node-red. I have tested this and it works.
So now I must think through what I want do with this. Node-red accessing Opensprinkler once a second is also testing that Opensprinkler is still running. If I run node-red on the Opensprinkler Pi, it just tests the application is running. However if I am running node-red on the Pi running Emoncms it tests that the whole Opensprinkler system is running, and can alert me!
So these are my initial thoughts. This is all in node-red.
Initially get the station names and store them. Store the master valve. Also the hostname and port of the Opensprinkler system.
Detect when a station starts and display its name as a message popup on my TV (because I can!)
Detect when it stops and log the run time to Emoncms if it is not a master valve (Emoncms will timestamp it).
Detect if opensprinkler is not running and send an email (twitter, or sms?) If it is not running after 10 seconds. Send reminders after 1 hour, 1 day, and the weekly until it responds again.I will be using the copy of node-red running on my Eoncms Pi as it provides a better warning of failure of Opensprinkler that way. I only need MQTT to display the popup message on my TV.
However I am also thinking to have this copy of node-red publish a regular MQTT meßsage and have node-red on the Opensprinkler Pi check this “heartbeat” and alert me if the Emoncms Pi fails!Most of this I have already prototyped.
If anyone has any further thoughts, or suggestions for additional functions I would be grateful.
Ian,
This is opening up a lot of possibilities. Now that I have looked into Node-red I have a better understanding of what you are doing.
Question: you mention you have been reading “the web api document”. Could you be a bit more specific? I see there is a generic API client node for node-red.
Have you looked at the mobile-app plugin? It has a set of URLs which provide an API for working with the ospi, e.g. [ospi URL]/jn rerutns station information. There is also an API from the built in UI, [ospi URL]/api/status which returns a bunch of status information.
Thanks.
Dan
Dan – your last message was marked as private. Did you mean this to happen? I was sent an automatic email copy, so I have read it.
The “web api document” was a pdf titled “Opensprinkler Firmware 2.1.0 API document (18th Oct, 2014. I found it somewhere on the Opensprinkler website. I think it is the one you referred to.
eg It documents sending http requests such as http://hostname:port/jc?pw=password to get the controller variables. The filename was fw210_api-1.pdf.
Node red has the capability to send such http requests and process the response often with not more to do than drag and drop, and fill in a few parameters. So it is almost trivial to send a request for station status data, and receive the response. Then I just use a javascript function node to massage the format of the returned data so it is suitable for wherever I am sending it.
It does open up a lot of possibilities. For example I have some of these cheap water meters (see http://www.seeedstudio.com/depot/G12-Water-Flow-Sensor-p-635.html although I bought mine from element14 some years ago, and never got around to using it). I have an arduino reading pulses from one on my desk, and calculating flow rates. (I simulate water flow by blowing in it occasionally to check it works!). I am thinking to place it just upstream of my master valve, and read it from either Opensprinkler or the EmonTX (ie an arduino reading my power consumption) node I have next to it. It doesn’t matter much where I read it, I will send it to Emoncms via Node-red (which will be able to allocate the flow to a station). Emoncms can then calculate daily. weekly totals etc. These totals could easily be sent to an Opensprinkler plugin for display, or I could display them in an Emoncms dashboard.
Even where there are no specific node-red nodes to interface with an application, often the application will have an MQTT interface. XBMC is one case, Openhab is another that I can think of quickly.
Regards
Ian
In a day or so I will have a working prototype of a node-red application interacting with Opensprinkler. It won’t have much error checking initially, and won’t handle things such as adding stations, changing the master etc. I will post it when it is working.
Hi Ian,
I marked the previous message private to test that feature of the new forum software. I’m still exploring what it can do.
The API for the Arduino based OpenSprinkler is partly supported on the Pi version by the mobile_app.py plugin as far as reading the status. The functions for controlling the program can be added.
Keep us posted on your progress.
Dan
Dan
I have a version working, getting data from OSPI, and detecting when a station is switched on/off. It sends a message to my TV when a station switches on eg “Front lawn sprinklers started”. When the station switches off it logs the duration to EmonCMS (Emoncms time stamps it). It needs some tidying up, extension to more than one board, and lots of error checking. In the process of developing it I learnt more about Node-red. Some of the tidying up needed came from that learning. eg One thing I will do is change it so that instead of logging straight to Emoncms it will publish to an MQTT topic. A separate flow will subscribe to that topic and log to Emoncms. This will make it easier to log to different databases and will also result in more reuseable code.
However I learnt another capability of Node-red that offers an interesting possibility. Node-red has global context variables that are available to all functions running. I am using the station names, the station number of the master valve, the number of stations, and the station status, so I am thinking to develop a plugin for OSPi that publishes these values to an MQTT topic on startup, and whenever there is a change. These values don’t change that often, so the MQTT traffic will be minimal.
In Node-red I will develop a flow that subscribes to this topic and places these values into global context variables. (It will be necessary in Node-red to have a startup initialisation via the JSON calls I used previously as Node-red and OSPi can’t be relied on to startup sequentially!). This effectively exposes near real time values of OSPi variables within Node-red! I will then complete the flow that populated the global variables. This will send the station swithon and off messages as before. Other flows could be developed to use the same variables to achieve different results if required.
The same could be done for other related sets of variables from OSPi eg weather related variables for adjusting watering times.
This approach will be more efficient, and is scaleable as more applications are developed (the way I have done it involves my application doing JSON calls to the OSPi web server every second or so, so it is not very scaleable. I haven’t yet tested it at 1 second intervals for example).
One thing I need to do is to develop a structure for the MQTT topics eg Garden/OSPI/stationrun, Garden/OSPI/stationrun/s1,2,3,… ,Garden/OSPI/stationrun/error, Garden/OSPI/stationrun/error/API/errornumbers (such as 400, etc) and so on. Stationrun would refer to this application – if someone developed a similar weather related application, they could use Garden/OSPI/weather for example.
What do you think?
Ian
Hey Ian…
Thanks for pointing me to Node Red. It’s been a lot of fun exploring it, as well as thinking of how it might be extended. I’m poking around OSPI now to figure out how to publish MQTT publishing from within OSPI. I have it working from within Python now, and played around with a fork of Dan’s Github gv.py, adding:
[“MQTT Host”, “string”, “mhost”,”Address of MQTT server for reporting events.”, “System”]
[“MQTT Topic”, “string”, “mtopic” , “Topic to which this OSPI server will post”, “System”]
[“MQTT Payload”,”string”, “mpay”, “What logging to perform”,”System”]
[“MQTT pub?”, “boolean”, “mpub”,”Whether to publish events via MQTT”,”System”]
re: your comments about needing to develop a structure for topics… I agree 100%. This is a very deep subject, and a problem that is rarely handled well (I spent 40 years in medical informatics, and this problem is everywhere). One approach that might be applicable is Topic Maps http://en.wikipedia.org/wiki/Topic_MapsThis is also related to Semantic Web http://en.wikipedia.org/wiki/Semantic_Web and Linked Data http://en.wikipedia.org/wiki/Linked_data
I think that this is one of the most powerful things the open source community can do: converging on an open, extensible definition of MQTT topics rather than Apple pushing “iTopics” or Google pushing “Gtopics” and having corresponding topic-wars.
I am also interested in “Promise” software technology: http://en.wikipedia.org/wiki/Futures_and_promises This is kind of related to an MQTT’s “last will and testament” of a server, to handle fail-over. (each OSPI could be a failover server for MQTT). An OSPI client, for example, could promise to send a “heartbeat” message every 10 minutes, which would trigger activity only when the promise was broken. Sprinkling programs could be communicated as “promises” which would run silently unless the promise was broken.
I’m just starting with OSPI and Github stuff. I used to be able to leap tall buildings with a single line of code, but that was many years ago, I’m afraid I’m a bit creaky. I hope to get up to speed with Python, Node.js, Node JS, and other tools real soon now 🙂
Ian,
This is really exciting. I too was not aware of node-red until reading your first post although I have had a bit of experience with flow based programming:
http://www.fireflyexperiments.com/#home
@munnecke, your comments add even more interesting avenues to explore.My original goal when I started working with Python and OpenSprinkler was to develop a platform for experimenting, and something that can be easily customized. I am now looking for ways to integrate node-red into the ospi distribution which has the possibility of allowing even more users to create custom functionality without having to learn all the details of hand coding a programming language.
Dan
munnecke
I haven’t had much of a look at gv.py yet. I assume your modifications will create data items that can be input via the Opensprinkler web interface?
Some comments
– we probably need hostname and port for the MQTT broker (although we could just enter it as part of the hostname).
– I am thinking of having several topics to post to eg Garden/OSPI/stationrun, Garden/OSPI/weather, and so on and being able to enable/disable each separately. Topics will essentially be based on logical groupings of data items. For example in my logging application I need the Station names, the station number of the master valve, the number of stations, and the status information for each station.
However this mixes some static information and dynamic, and I may change my mind! I am confident though that I will wind up publishing to more than one topic from OSPi.
– I am not sure what your mpay variable is for (in my mind a topic would have a fixed set of data items which would define the payload). Are you thinking to have a more generic system and mpay defines the data items (from gv)? That would be very flexible.
Your links provided some interesting reading. I too am a bit creaky with my programming skills.
Dan
Yes – Node-red is fun. I am nearly finished an application which uses the web API to get data from OSPi, and logs the information to Emoncms and to XBMC. I will post the flow when it is ready. However as I mentioned earlier I am now think that sending data from OSPi to Node-red and exposing the data via global variables is the way to go long term so I won’t be developing this further eg to add more error checking etc.
The data that is sent must also be able to be retrieved by Node-red via the web interface so that Node-red can initialise the globals. An example would be the station status information. You can retrieve nstations and sn via the web interface, and I would have node-red globals context.global.nstations and context.globals.sn[]. These would be initialised by Node-red using the web interface on startup, and thereafter updated via the MQTT interface. I can then process the data in Node-red in any way I wish. Obviously in some applications it will be necessary to send data back to OSPi via MQTT.
This is therefore a way to achieve your goal – a platform for customisation and experimenting. This could be done in Node-red or any other environment simply subscribing to the appropriate MQTT topics.
Regards
Ian
Ok – here is a node red application that gets the station names and number of the master valve from OSPI, then polls the status of the stations. When a station turns on, it sends a message to the MQTT broker, and saves the start time. When a station turns off, it calculates the run time, and posts another message to MQTT. Errors such as failure of the OSPI to respond, or other http errors result in a MQTT message.
The start and stop messages are processed. The start messages go to XBMC and appear on my TV as a notification, The stop messages go to Emoncms to record the run times.
There is still some work to do – tidying up MQTT topics, developing some configuration nodes, and better handling errors. It may also be useful to periodically repoll the station names in case they change.
There are comments in the code, and in comment nodes. Instructions for configuring and running it are in the comment nodes. There is a “debug” environment included so you can start and stop the flows (otherwise the debug messages go too fast!).
This is more of a demonstration. I now intend to develop a plugin (or give input to someone else who will write it!) that will watch OSPI variables and send changes via MQTT, rather than have node-red continuously polling OSPI. Some of this code will then be reused in that new application. I suspect the best way to do this in OSPi is to mirror the web API. ie the an OSPI/jn topic will mirror the values sent in response to a http://host:port/jn?pw=password request, and so on.
Here is the node-red code.
[{“id”:”d7f98065.28068″,”type”:”function”,”name”:”fnBuildurl”,”func”:”// The received message is stored in ‘msg’\n// It will have at least a ‘payload’ property:\n\n// The ‘context’ object is available to store state\n// between invocations of the function\n// context = {};\n\n// The payload of the injected message contains the hostname and port\n// of the OSPI as csv’s\n\n// global context variables are preset in settings.js\n// countresp is updated in fnProcesshttpresp\n// It is used to count the number of times through fnProcesshttpresponse\n// to ensure we have had as many responses as requests \n//Initialise it here\n// The first time through the number of responses will be 0\n\n// context.global.countstart and context.global.countresp are set to 0 in fn Set Global\n\n\n// check to see that the last http request was responded to\n// context.global.countresp is incremented while processing the response\n// so if it is not equal to context.countstart there was no response last time\n// doesn’t apply first time through\nmsgo = msg;\n\n context.host=msg.payload.split(\”,\”);\n\n\nif (context.global.countstart > 0) {\n\n// As we have sent a request, then the no of responses should equal the number of requests\n\n if ((context.global.countstart == context.global.countresp) ) {\n\n// If it was responded to they will be equal. Send success diagnostic\n\n msgo.topic = \”Garden/OSPI/stationrun/api-responses\”;\n msgo.payload = \”Reached server – fnBuildurl \” ;\n msgo.payload = msgo.payload + context.global.countstart.toString() + ‘ ‘ + context.global.countresp.toString();\n }\n\n\n// First time through there was no previous response to have\n\n if ((context.global.countstart != context.global.countresp) ) {\n\n msgo.topic = \”Garden/OSPI/stationrun/api-response\”;\n msgo.payload = \”Error reaching server – fnBuildurl \” ;\n// add some diagnostics\n msgo.payload = msgo.payload + context.global.countstart.toString() + ‘ ‘ + context.global.countresp.toString();\n \n// set them equal so next time the check will work\n\n context.global.countresp = context.global.countstart;\n }\n}\n//if first time get station names.\nif (context.global.countstart == 0) {\n\n// only do this the first time\n\n msgo.url = ‘http://’ + context.host[0] +’:’ + context.host[1] + ‘/jn?pw=’ + context.host[2];\n// add some diagnostics\n msgo.payload = msgo.url+ \” Get Station names \”;\n msgo.payload = msgo.payload + context.global.countstart.toString() + ‘ ‘ + context.global.countresp.toString();\n\n }\n\n//if second time get master valve number\n\nif (context.global.countstart == 1) {\n msgo.url = ‘http://’ + context.host[0] +’:’ + context.host[1] + ‘/jo?pw=’ + context.host[2];\n msgo.payload = msgo.url+ \” Get Master Valve no \”;\n msgo.payload = msgo.payload + context.global.countstart.toString() + ‘ ‘ + context.global.countresp.toString();\n\n }\n// All subsequent times just get station status\nif (context.global.countstart > 1) {\n msgo.url = ‘http://’ + context.host[0] +’:’ + context.host[1] + ‘/js?pw=’ + context.host[2];\n msgo.payload = msgo.url+ \” Get Status \”;\n msgo.payload = msgo.payload + context.global.countstart.toString() + ‘ ‘ + context.global.countresp.toString();\n\n}\ncontext.global.countstart = context.global.countstart + 1;\nreturn [msgo];”,”outputs”:”1″,”x”:163.04559326171875,”y”:255.93939971923828,”z”:”c3e73b60.3c18c8″,”wires”:[[“4b28d045.b4d73″,”2c641a61.d39be6”]]}]
Oops – I don’t seem to have copied the whole worksheet. Here is another try.
[{“id”:”39eb56aa.c614aa”,”type”:”emoncms-server”,”server”:”http://192.168.0.58/emoncms”,”name”:”Locally Hosted Emoncms”},{“id”:”ba386057.845d3″,”type”:”mqtt-broker”,”broker”:”192.168.0.58″,”port”:”1883″,”clientid”:””},{“id”:”d7f98065.28068″,”type”:”function”,”name”:”fnBuildurl”,”func”:”// The received message is stored in ‘msg’\n// It will have at least a ‘payload’ property:\n\n// The ‘context’ object is available to store state\n// between invocations of the function\n// context = {};\n\n// The payload of the injected message contains the hostname and port\n// of the OSPI as csv’s\n\n// global context variables are preset in settings.js\n// countresp is updated in fnProcesshttpresp\n// It is used to count the number of times through fnProcesshttpresponse\n// to ensure we have had as many responses as requests \n//Initialise it here\n// The first time through the number of responses will be 0\n\n// context.global.countstart and context.global.countresp are set to 0 in fn Set Global\n\n\n// check to see that the last http request was responded to\n// context.global.countresp is incremented while processing the response\n// so if it is not equal to context.countstart there was no response last time\n// doesn’t apply first time through\nmsgo = msg;\n\n context.host=msg.payload.split(\”,\”);\n\n\nif (context.global.countstart > 0) {\n\n// As we have sent a request, then the no of responses should equal the number of requests\n\n if ((context.global.countstart == context.global.countresp) ) {\n\n// If it was responded to they will be equal. Send success diagnostic\n\n msgo.topic = \”Garden/OSPI/stationrun/api-responses\”;\n msgo.payload = \”Reached server – fnBuildurl \” ;\n msgo.payload = msgo.payload + context.global.countstart.toString() + ‘ ‘ + context.global.countresp.toString();\n }\n\n\n// First time through there was no previous response to have\n\n if ((context.global.countstart != context.global.countresp) ) {\n\n msgo.topic = \”Garden/OSPI/stationrun/api-response\”;\n msgo.payload = \”Error reaching server – fnBuildurl \” ;\n// add some diagnostics\n msgo.payload = msgo.payload + context.global.countstart.toString() + ‘ ‘ + context.global.countresp.toString();\n \n// set them equal so next time the check will work\n\n context.global.countresp = context.global.countstart;\n }\n}\n//if first time get station names.\nif (context.global.countstart == 0) {\n\n// only do this the first time\n\n msgo.url = ‘http://’ + context.host[0] +’:’ + context.host[1] + ‘/jn?pw=’ + context.host[2];\n// add some diagnostics\n msgo.payload = msgo.url+ \” Get Station names \”;\n msgo.payload = msgo.payload + context.global.countstart.toString() + ‘ ‘ + context.global.countresp.toString();\n\n }\n\n//if second time get master valve number\n\nif (context.global.countstart == 1) {\n msgo.url = ‘http://’ + context.host[0] +’:’ + context.host[1] + ‘/jo?pw=’ + context.host[2];\n msgo.payload = msgo.url+ \” Get Master Valve no \”;\n msgo.payload = msgo.payload + context.global.countstart.toString() + ‘ ‘ + context.global.countresp.toString();\n\n }\n// All subsequent times just get station status\nif (context.global.countstart > 1) {\n msgo.url = ‘http://’ + context.host[0] +’:’ + context.host[1] + ‘/js?pw=’ + context.host[2];\n msgo.payload = msgo.url+ \” Get Status \”;\n msgo.payload = msgo.payload + context.global.countstart.toString() + ‘ ‘ + context.global.countresp.toString();\n\n}\ncontext.global.countstart = context.global.countstart + 1;\nreturn [msgo];”,”outputs”:”1″,”x”:163.04559326171875,”y”:255.93939971923828,”z”:”c3e73b60.3c18c8″,”wires”:[[“4b28d045.b4d73″,”2c641a61.d39be6”]]},{“id”:”47b336b9.b84cc8″,”type”:”inject”,”name”:”Start”,”topic”:”Garden/OSPI/stationrun/processing”,”payload”:”192.168.0.182,8080,opendoor”,”payloadType”:”string”,”repeat”:””,”crontab”:””,”once”:false,”x”:77.77275085449219,”y”:47.11363220214844,”z”:”c3e73b60.3c18c8″,”wires”:[[“a59bbed.f5a644”]]},{“id”:”4b28d045.b4d73″,”type”:”debug”,”name”:”fn Build URL”,”active”:true,”console”:”false”,”complete”:”false”,”x”:538.1591339111328,”y”:253.74998474121094,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”d24cc8ef.2db338″,”type”:”debug”,”name”:”Http Request”,”active”:false,”console”:”false”,”complete”:”false”,”x”:538.4545135498047,”y”:284.25001525878906,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”3a62be14.c59d42″,”type”:”function”,”name”:”fnProcesshttpresponse”,”func”:”// Messages returned \n// – Station start\n// – station stops (duration is in payload, station number in topic)\n// – errors\n\n// First response processed is from OSPi (to /jn?pw=… – get station names\n// – we must save station names\n// First save the msg (including payload) so we can build a new one\n// without wrecking the old one\n\n// This function just produces MQTT messages – Other flows will subscribe to these messages and send them to \n// XBMC, Emoncms or wherever. This allows a more modular approach\n\n// MQTT topics produced by this function are \n// Garden/OSPI/stationrun/api-noresponse\n// Garden/OSPI/stationrun/start/\n// Garden/OSPI/stationrun/stop/\n\n// For start and stop the station number is in the payload\n \nomsg = msg;\ntopic = msg.topic;\nstatusCode = msg.statusCode;\n\n\nif (statusCode != 200 ) {\n\n// This error processing does not handle the case when there is no response from OSPI\n\n// Still need to get the first two headers, so retry \n\n// will add more error processing. For now just return and try again\n// the error message will go to MQTT\n\n msg.topic = \”Garden/OSPI/stationrun/api-response\”;\n msg.payload = \”Error reaching server – code \” + statusCode;\n\n return [omsg];\n} \n\n// set up a date object with now as the date/time\ntempdate = new Date() ;\n\n//\n//———————————————-\n// check if this is the first succesful time through\n//———————————————-\n\nif ( context.global.countresp == 0) {\n\ncontext.snames = [\”1\”,\”2\”,\”3\”,\”4\”,\”5\”,\”6\”,\”7\”,\”8\”]; // Initialise\ncontext.starttime = [ 0, 0, 0, 0, 0, 0, 0, 0 ];\ncontext.lastsn = [\”0\”,\”0\”,\”0\”,\”0\”,\”0\”,\”0\”,\”0\”,\”0\”];\n for (i=0;i<8;i++) {\n \n context.snames[i] = omsg.payload.snames[i];\n\n// set up start times and status (sn). Initially set status to off (0). \n// These are global so this function so will persist \n\n context.starttime[i] = new Date();\n context.lastsn[i] = 0;\n context.global.countresp = 1;\n \n }\n\n// Don’t want output at this stage – next time we will get more information from OSPi \n\n return [null];\n }\n//\n//—————————————-\n// Second time through\n// —————————————\n//\n//response (to /jo?pw=.. – get options\n\nif (context.global.countresp == 1) {\n\n// – save Master station number (mas) (0 means none)\n// stations are 0 – 7, mas is 1-8. Subtract 1 from mas \n\n context.mas = msg.payload.mas -1;\n context.noofstns = 8 + msg.payload.ext * 8;\n\n// Don’t want ouput at this stage – next time we will start getting station status data from OSPi \n return [null];\n }\n//\n//————————————-\n// All subsequent times through (Must be > 1)\n//————————————–\n//\n\nif (context.global.countresp > 1) {\n\n// Subsequent responses (to /js?pw=…\n// – process the station status. \n// If a station is on check if it has just been turned on\n// and it is not the master\n// If this is true\n// – save the start time (per station as it may not be in sequential mode), \n// – and construct a message for MQTT saying \”Station xxxxxx started\”\n//\n// If a station is off check if it has just been turned off\n// If it has\n// – calculate the duration in milliseconds\n// – construct a payload for EMoncms which has the station and duration\n// This version only caters for one station turning on at a time (a todo item)\n// context.noofstns = omsg.payload.nstations;\n\nfor (i=0;i<8;i++) {\n//omsg.payload = omsg.payload + \” \” + omsg.payload.sn[5] + \” \” +context.lastsn[5] + \” \” +context.mas;\n//return omsg;\n \n if ((omsg.payload.sn[i] == 1) && (context.lastsn[i] == 0) && (i != context.mas)) {\n\n// Save station status and time it started\n\n context.lastsn[i] = omsg.payload.sn[i];\n context.starttime[i] = tempdate;\n\n// send MQTT message that will get sent to the TV\n\n omsg.topic = ‘Garden/OSPI/stationrun/start’;\n omsg.payload = \”Station – \”+ i.toString() +context.snames[i] + \” started at \” + context.starttime[i].toString();\n context.global.countresp = context.global.countresp + 1;\n return[omsg];\n }\n if ((omsg.payload.sn[i] == 0) && (context.lastsn[i] == 1) && (i != context.mas)) {\n context.lastsn[i] = omsg.payload.sn[i];\n duration = Math.abs(tempdate-context.starttime[i]);\n \n omsg.topic = ‘Garden/OSPI/stationrun/stop’;\n \n// in the payload the / will be used to split the payload at the duration \n\n omsg.payload = \”Station – \”+ i.toString() + \” /\” + duration.toString() + \”/\”;\n// add some debug information\n omsg.payload = omsg.payload + \” \” + tempdate.toString() + \” \” + context.starttime[i].toString();\n context.global.countresp = context.global.countresp + 1;\n \n return [omsg];\n\n }\n\n// If we get here it is not an error, or start or stop\n// increment context.global.countresp so fnBuildurl will know we got a response\n\ncontext.global.countresp = context.global.countresp + 1;\n\n// Save the current status and time\n\n context.starttime[i] = tempdate;\n context.lastsn[i] = omsg.payload.sn[i];\n\n }\n}\nreturn [null];\n”,”outputs”:”1″,”x”:296.9501037597656,”y”:347.20008277893066,”z”:”c3e73b60.3c18c8″,”wires”:[[“76f8f02f.89071″,”b74328be.48bcd8”]]},{“id”:”76f8f02f.89071″,”type”:”debug”,”name”:”Errors,start,stop”,”active”:true,”console”:”true”,”complete”:”true”,”x”:544.7000885009766,”y”:374.9500274658203,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”1da70e4a.e258f2″,”type”:”debug”,”name”:”Json out”,”active”:false,”console”:”false”,”complete”:”true”,”x”:523.7000274658203,”y”:314.9500274658203,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”8d7cea09.728318″,”type”:”json”,”name”:”Json “,”x”:221.25012969970703,”y”:317.000039100647,”z”:”c3e73b60.3c18c8″,”wires”:[[“1da70e4a.e258f2″,”3a62be14.c59d42”]]},{“id”:”2c641a61.d39be6″,”type”:”http request”,”name”:”http request”,”method”:”GET”,”url”:””,”x”:202.70008087158203,”y”:285.9500207901001,”z”:”c3e73b60.3c18c8″,”wires”:[[“d24cc8ef.2db338″,”8d7cea09.728318”]]},{“id”:”b74328be.48bcd8″,”type”:”mqtt out”,”name”:”Output from processing status “,”topic”:””,”qos”:”0″,”retain”:”false”,”broker”:”ba386057.845d3″,”x”:587.9499664306641,”y”:344.9500274658203,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”25ad1419.da52ec”,”type”:”mqtt in”,”name”:”Returns/errors from API calls”,”topic”:”Garden/OSPI/stationrun/api-response”,”broker”:”ba386057.845d3″,”x”:120,”y”:490.1999969482422,”z”:”c3e73b60.3c18c8″,”wires”:[[“74a6653d.8b599c”,”6ce3076d.931cf8″]]},{“id”:”74a6653d.8b599c”,”type”:”function”,”name”:”fnSendtoXBMC”,”func”:”var paylmqtt = msg.payload;\nvar topicstr = msg.topic;\n// This is a lvl 2 topic\noutmsg = msg;\npayloadmqtto = ‘\”{\”lvl\”:\”2\”,\”sub\”:\”‘;\npayloadmqtto = payloadmqtto + topicstr + ‘\”,’;\npayloadmqtto = payloadmqtto + ‘\”txt\”:\”‘+paylmqtt+’\”,\”img\”:\”/home/pi/.xbmc/userdata/Thumbnails/background.png\”,\”delay\”: \”20000\”}’;\noutmsg.payload = payloadmqtto;\noutmsg.topic = \”Home/xbmc1\”;\nreturn [outmsg];”,”outputs”:1,”x”:370.2000274658203,”y”:492.45001792907715,”z”:”c3e73b60.3c18c8″,”wires”:[[“4767e15b.b8982″,”2ed02f35.d12fd”]]},{“id”:”4767e15b.b8982″,”type”:”mqtt out”,”name”:”Send errors to XBMC”,”topic”:”Home/ospi”,”qos”:”0″,”retain”:”false”,”broker”:”ba386057.845d3″,”x”:653.7000846862793,”y”:492.44996643066406,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”ea53ba2c.15ac48″,”type”:”mqtt in”,”name”:”Station stop messages”,”topic”:”Garden/OSPI/stationrun/stop”,”broker”:”ba386057.845d3″,”x”:140.45000457763672,”y”:461.1999969482422,”z”:”c3e73b60.3c18c8″,”wires”:[[“e27d1f82.1d82e”,”6f0e8ea6.90f17″]]},{“id”:”e27d1f82.1d82e”,”type”:”function”,”name”:”fnSendtoEmoncms”,”func”:”paylmqtt = msg.payload;\ntopicstr = msg.topic;\npaylmqtto = \” fnSendtoEmoncms\”;\nsubtopics = [\” \”,\” \”,\” \”,\” \”];\n// Emoncms is expecting a payload like [0,0,0,78,0,0,0,0] \n// where this means a duration of 78 seconds for station 3\n\n// The station number is in the payload \n\n// The topic is of the form\n// Garden/OSPI/stationrun/start\n// or\n// Garden/OSPI/stationrun/stop\n\n//Only Stop messages go to Emoncms\n\n// Get the station number\nsubtopics = msg.payload.split(‘ ‘);\nsn = subtopics[2];\n\n// I am sure this can be done better\n\n var spl = [0,0,0,0,0,0,0,0];\n// The / was put in payload so the duration could be found\n splitpayload = paylmqtt.split(‘/’);\n\n duration = splitpayload[1];\n\n spl[sn] = duration;\n paylmqtto = \”[\”+spl[0]+\”,\”+spl[1]+\”,\”+spl[2]+\”,\”+spl[3]+\”,\”+spl[4]+\”,\”+spl[5]+\”,\”+spl[6]+\”,\”+spl[7]+\”]\”;\n msg.payload = paylmqtto;\n return [msg];\n\n”,”outputs”:1,”x”:378.45005798339844,”y”:462.95001792907715,”z”:”c3e73b60.3c18c8″,”wires”:[[“bdb27557.424d88″,”3da80bf.fc257f4”]]},{“id”:”bdb27557.424d88″,”type”:”emoncms”,”name”:”Emoncms”,”emonServer”:”39eb56aa.c614aa”,”nodegroup”:”25″,”x”:618.9501457214355,”y”:463.9499969482422,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”7d982275.8267dc”,”type”:”mqtt in”,”name”:”Station start messages”,”topic”:”Garden/OSPI/stationrun/start”,”broker”:”ba386057.845d3″,”x”:139.9499969482422,”y”:520.1999969482422,”z”:”c3e73b60.3c18c8″,”wires”:[[“f37c342e.0c83c8″,”14aae646.eb551a”]]},{“id”:”f37c342e.0c83c8″,”type”:”function”,”name”:”fnSendtoXBMC”,”func”:”var paylmqtt = msg.payload;\nvar topicstr = msg.topic;\n// This is a lvl 1 topic\noutmsg = msg;\n\noutmsg.topic = \”Home/xbmc1\”;\npayloadmqtto = ‘{\”lvl\”:\”2\”,\”sub\”:’;\npayloadmqtto = payloadmqtto + ‘\”‘ + topicstr + ‘\”,’;\npayloadmqtto = payloadmqtto + ‘\”txt\”:\”‘+paylmqtt+’\”,\”img\”:\”/home/pi/.xbmc/userdata/Thumbnails/background.png\”,\”delay\”: \”20000\”}’;\noutmsg.payload = payloadmqtto;\nreturn [outmsg];”,”outputs”:1,”x”:371.2000198364258,”y”:521.200023651123,”z”:”c3e73b60.3c18c8″,”wires”:[[“c9005a4e.36ffa8″,”2ed02f35.d12fd”]]},{“id”:”c9005a4e.36ffa8″,”type”:”mqtt out”,”name”:”Send start messages to XBMC”,”topic”:”Home/ospi”,”qos”:”0″,”retain”:”false”,”broker”:”ba386057.845d3″,”x”:680.4500846862793,”y”:523.1999969482422,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”6ce3076d.931cf8″,”type”:”debug”,”name”:”Errors from API”,”active”:true,”console”:”false”,”complete”:”true”,”x”:371.2000274658203,”y”:552.1999969482422,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”6f0e8ea6.90f17″,”type”:”debug”,”name”:”Stop messages”,”active”:true,”console”:”false”,”complete”:”true”,”x”:368.7000274658203,”y”:433.20001792907715,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”14aae646.eb551a”,”type”:”debug”,”name”:”Stat Start messages”,”active”:true,”console”:”false”,”complete”:”true”,”x”:384.9500274658203,”y”:582.7000122070312,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”2ed02f35.d12fd”,”type”:”debug”,”name”:”sent to xbmc”,”active”:true,”console”:”false”,”complete”:”payload”,”x”:625.7000846862793,”y”:583.6999969482422,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”b7319d2e.48ce6″,”type”:”inject”,”name”:”Toggle on/off”,”topic”:”Garden/OSPI/stationrun/processing”,”payload”:”192.168.0.182,8080,opendoor”,”payloadType”:”string”,”repeat”:””,”crontab”:””,”once”:false,”x”:93.74999237060547,”y”:76.99999237060547,”z”:”c3e73b60.3c18c8″,”wires”:[[“1d95ac5f.e26a54″,”e05f89e5.1fa078”]]},{“id”:”1d95ac5f.e26a54″,”type”:”function”,”name”:”Set loop Global”,”func”:”\n\nif (context.global.state == ‘off’ ) {\n\tcontext.global.state = ‘on’;\n\n} else {\n\tcontext.global.state = ‘off’;\n\treturn null;\n}\n\nreturn msg;”,”outputs”:1,”x”:253.75003051757812,”y”:80.0000228881836,”z”:”c3e73b60.3c18c8″,”wires”:[[“bb897ba1.447688”]]},{“id”:”bb897ba1.447688″,”type”:”function”,”name”:”Control Loop”,”func”:”// The received message is stored in ‘msg’\n// It will have at least a ‘payload’ property:\n// console.log(msg.payload);\n// The ‘context’ object is available to store state\n// between invocations of the function\n// context = {};\n\nif (context.global.state == ‘off’ ) {\n\tmsg.loop = ‘off’;\n} else {\n\tmsg.loop = ‘on’;\n}\n\nreturn msg;”,”outputs”:1,”x”:443.52783203125,”y”:47.833343505859375,”z”:”c3e73b60.3c18c8″,”wires”:[[“1628a645.e9d75a”]]},{“id”:”1628a645.e9d75a”,”type”:”delay”,”name”:””,”pauseType”:”delay”,”timeout”:”5″,”timeoutUnits”:”seconds”,”rate”:”1″,”rateUnits”:”second”,”randomFirst”:”1″,”randomLast”:”5″,”randomUnits”:”seconds”,”drop”:false,”x”:588.3611450195312,”y”:47.83334732055664,”z”:”c3e73b60.3c18c8″,”wires”:[[“b172480a.4e8db8”]]},{“id”:”b172480a.4e8db8″,”type”:”function”,”name”:”Loop”,”func”:”// The received message is stored in ‘msg’\n// It will have at least a ‘payload’ property:\n// console.log(msg.payload);\n// The ‘context’ object is available to store state\n// between invocations of the function\n// context = {};\nconsole.log(‘Entering loop’);\nconsole.log(msg.loop);\nmsg.payload = context.global.saveparams;\nif (msg.loop == ‘off’ ) {\n\treturn [msg,null];\n} else {\n\tconsole.log(‘Looping’);\n\treturn [msg,msg];\n}”,”outputs”:”2″,”x”:499.9444580078125,”y”:129.02779388427734,”z”:”c3e73b60.3c18c8″,”wires”:[[“bb897ba1.447688”],[“d7f98065.28068”]]},{“id”:”e05f89e5.1fa078″,”type”:”debug”,”name”:””,”active”:true,”console”:”false”,”complete”:”true”,”x”:228.75003051757812,”y”:110.5,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”3da80bf.fc257f4″,”type”:”debug”,”name”:”Send to Emoncms”,”active”:true,”console”:”false”,”complete”:”payload”,”x”:643.1110954284668,”y”:552.7777557373047,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”a59bbed.f5a644″,”type”:”function”,”name”:”fnSetappGlobal”,”func”:”context.global.countstart = 0;\ncontext.global.countresp = 0;\ncontext.global.saveparams = msg.payload;\nreturn msg;”,”outputs”:1,”x”:255.77780151367188,”y”:47.888885498046875,”z”:”c3e73b60.3c18c8″,”wires”:[[“bb897ba1.447688”]]},{“id”:”1e0dc7d9.e1f238″,”type”:”comment”,”name”:”Control”,”info”:”This section enables you to start and stop/reset the process.\n\nUseful for debugging. Start/reset counters to 0 by clicking on Start. \nThen click on Toggle on/off to start it, and pause it. \n\nConfigure Start and Toggle with the host,port,password (of OSPI). These should \nbe comma separated. (I haven’t worked out configuration nodes yet!).\n\nAdjust the timing by changing the delay. 5 seconds works well for debugging.\n\nI have added debug messages to the message payload. For example if you see 15 15 \nthis is the count of the no of requests to OSPI, and the number of responses.\n\nWhen count = 0 the first request for station names is sent.\nWhen count = 1 the second request for master station number is sent\nWhen count > 1 requests for valve status are sent. Once this is happening \nyou can start and stop sprinklers and the start and stop will be processed. \n\nWhen debugging you can then pause the process and inspect the debug output \nat your leisure.\n\nWhen debugging, putting a return [msg]; statement in a function effectively \ncreates a breakpoint of sorts. Useful for tracking down runtime errors that \njavascript won’t say where they are! eg \”index of undefined\”. \n\nIn production a separate start node should be added that fires off at a set interval. \nThis is so that it will not require someone to click on the start and toggle buttons every time\nnode-red restarts.\n\nTip for the future.\nInstead of adding debug information to the msg body, I will add it to a new message property.\neg instead of adding variable xyz to the message body, I would add it to msg.xyz\ncontext.xyz would be added to msg.cxyz and global.context.xyz would be added to msg.gcxyz\nAdditional text could be added as well. \n\nAcknowledgement:\nThis looping functionality came from \nhttps://www.ibm.com/developerworks/community/blogs/hickmat/entry/updated_version_of_node_red_loop?lang=en\n\n\n\n\n \n”,”x”:78.19999694824219,”y”:20,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”155d2723.eaa2d9″,”type”:”comment”,”name”:”Main processing”,”info”:”The production start inject node must be changed to inject the string \nevery x seconds. This node will then automatically fire whenever node-red restarts\nand continue firing. For testing this node is turned off. The control loop can be left \nin place as it will do nothing if not clicked. \n\nHere a http://host:port/jn?pw=password is sent to get the station names\nThen http://host:port/jo?pw=password is sent to get the master valve no\nThen repeated http://host:port/js?password messages are sent to get the valve status’s\n\nThese are monitored for changes for starts and stops. \nStarts are sent to XBMC via MQTT and the start time noted\nStops are sent to Emoncms via MQTT after calculating the duration.\nErrors in communication with OSPI (no response or return code ne 200)\nare sent to XBMC via MQTT\n\nThe process is \nBuild the URL\nSend the HTTP request\nPut the response through the JSON Node \n (I have no idea what this does but the next function crashes if I dont).\nProcess the response.\n\nThis main processing section results in MQTT messages only. It does not\nsend messages directly to the end recipients eg XBMC).”,”x”:120,”y”:174.1999969482422,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”eb9e2e17.1461d”,”type”:”comment”,”name”:”Process results”,”info”:”Here the MQTT messages that are sent from the main processing are subscribed to, \nreformatted for the appropriate recipient, and sent off.\n\nTo add eg an email message whenever a station starts, simply add another\nflow, subscribing to the start message, create a function to reformat the message \nfor email and pass the message to the email node. Same for twitter etc.\n”,”x”:118.19999694824219,”y”:428.2000274658203,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”1ed2ed83.e12d12″,”type”:”comment”,”name”:”Debug nodes”,”info”:”These can be removed, but this simply would make the flow look simpler. \n\nI have used separate debug nodes instead of one common one so that the debug node name \nhelps identify where the debug message comes from.”,”x”:101.19999694824219,”y”:599.2000122070312,”z”:”c3e73b60.3c18c8″,”wires”:[]},{“id”:”2d4434ba.d2bbcc”,”type”:”inject”,”name”:”Production start”,”topic”:””,”payload”:”host,port,password”,”payloadType”:”string”,”repeat”:””,”crontab”:””,”once”:false,”x”:117,”y”:205.1999969482422,”z”:”c3e73b60.3c18c8″,”wires”:[[“d7f98065.28068”]]},{“id”:”b5cc645.f4a3398″,”type”:”comment”,”name”:”Deployment”,”info”:”To modiify these flows for another site, the inject nodes must all be \neditted so the string injected is host,port,password of your system. \nHost is the OSPI host.\n\nThe MQTT nodes must be editted to set the address of the MQTT broker.\nThis is only done once as they all share the same configuration node.\nSame for the Emoncms node.\n\nTo use the XBMC notifications, see https://github.com/matbor/mqtt2xbmc-notifications\nYou will need to edit autoexec.py to subscribe to the correct MQTT topic\nAll that is required to implement it on an XBMC system is to place\nautoexec.py and mosquitto.py in the /home/pi/.xbmc/userdata directory.\n\nThe topics I have used may need to be modified for your needs. Unfortunately\nthese topics are spread all over the place. In future perhaps a set of globals \nmay be useful so that this configuration can be done in one function.\n”,”x”:85.19999694824219,”y”:670.2000122070312,”z”:”c3e73b60.3c18c8″,”wires”:[]}]
For sending JSON data and program code it might be better to attach a file as the data can change/corrupt when using the forum body (encoding of characters etc). It also helps keep things neat and usable on the forums.
Thanks, Ian. I got the single node fnBuildurl.. I noticed that you used two different topics – one with plural “api-responses” on lines 48 and 38. I get an “Unexpected Token” error when I try to deploy the node…. not very helpful; it just gives the message and no other info. Maybe it was garbled in the email.
I don’t understand the full data flow of how this all works…
It would seem simpler to me to put the publisher logic in the main OSPI routine, synchronized with the state change of the sprinkler. Polling seems to be a lot more complicated, and subject to some missed events if the poll happens to happen during a state change in OSPI. I haven’t done real time systems like this before, but it seems to me that the whole value of having a event/message architecture is to get around all the problems of polling. Plus, polling doesn’t seem to be scalable; as we add more detail and events, it would seem that the polling process would get overwhelmed.
How do you display the messages in XBMC?
Maybe this is worth moving off to a Github repo?This is a lot of fun.
Hi munnecke
In the Node Red editor after you import the flow I sent earlier you should see the screen as per the jpg I have attached. The unexpected token message must mean something has become garbled, so I will do as suggested earlier – attach a file. node-red.txt is a notepad file. Open it, select all and copy to the clipboard. In the Node-red editor use the import from clipboard function. I just tried it with no errors, so hopefully this will work better. I have fixed the issue with the mis-spelt topic in this new version.
There are a lot of comments in the comment nodes that describe the function of each node.There are a lot of comments in the code of the functions as well.
Most people will probably need to add the emoncms node to Node red – I think sudo npm install emoncms will do it. If you don’t have emoncms, then perhaps swapping that with an email node would be better for those people.
The information about getting data to XBMC is in the comment node labelled deployment. This provides some information about changes necessary to run on another system eg where to edit IP addresses etc.
Your comments regarding polling etc mirror the conclusions I have reached. So that is the direction I will be going in. I was thinking of putting the publisher logic in a separate plugin. Unless this was to become “core” functionality, and maintained as such I would prefer not to be modifying the main OSPI code. That is why I have referred to this as a “demonstration” rather than a full application. It is a working demonstration though, at least on my system. It needs some tidying up, and more error handling though.
It does illustrate the ease of which these tools can be used to develop this applications and push messages around to different systems.
Ian
Attachments:
Ian,
I couldn’t get the text you first posted to import into Node-red but the file you uploaded worked like a charm. You mentioned that the text file was a notepad file so I’m guessing you use a Windows system. Are you familiar with WinSCP?
http://winscp.net/eng/docs/introduction
With it you can easily copy files between the Raspi and your windows system. You could export a Node-red flow to the “library” in Node-red then copy the resulting .json file from /home/pi/node-red/lib/flows on the Pi to your Windows system for transfer to the forum or send via email etc. It would work in the other direction as well.
Your plan to develop a plugin for ospi is the way to go. It would allow someone to add these features to an installation without worrying about which distribution of the main program to use. I have not had time to document the plugin system yet but I plan to work on that in the next few weeks. in the mean time there is a little info on the wiki:
http://rayshobby.net/mediawiki/index.php/Python_Interval_Program_for_OSPi
And of course you can always ask questions here and I will try to help. That will also help me with the documentation.
Dan
Thanks Dan.
I’m glad it worked for you.
I use Winscp and Putty, but it never occurred to use it to copy the JSON file!
I noticed that the Node-red application doesn’t work too well if you start the sprinklers first, then Deploy the application! However as I said, I will treat this as a demonstration, and learning exercise. Although it would be possible to finish it off so it handles all these unusual cases and errors, I think polling OSPi every second for this data is likely to cause problems.
I assume a plugin that gets imported into the main application runs in a manner similar to a separate “thread” but with access to the variables of the main program (provided I access them as shown in the documentation!). So I can write a plugin that monitors the variables I am interested in, and publish them to MQTT when they change, and not worry too much what the main program is doing.
I will of course also need to get some configuration data from a web page. (MQTT broker address for example, and an MQTT enable flag for each topic..
At this stage my thinking is to mirror the API calls. eg I will publish to topic OSPI/jc all the variables that the http://host/jc?pw=password call would retrieve. This will make it easier in Node-red to initialise variables (although the retain feature of MQTT may make this unnecessary!). At the early stage I will concentrate on the jo, jn, and js calls because they are the ones I need to log the data to Emoncms.
Regards
Ian-in-WA (Western Australia – hence the times of posts!)
So many intertwined topics… let me respond to some of them raised on this thread:
Dan: re: “My original goal when I started working with Python and OpenSprinkler was to develop a platform for experimenting, and something that can be easily customized.” Wonderful idea, and thank you for acting on it. Hopefully, I can build on your platform – and use my own home as a experimental site. I have a huge water bill and am in the middle of California’s drought (San Diego), so this is quite timely for me.
Ian: re: “I haven’t had much of a look at gv.py yet. I assume your modifications will create data items that can be input via the Opensprinkler web interface?” And thank you Ian for pointing out these neat new technologies. I’ll let Dan comment on this, but my understanding is that gv.py is the home for establishing the global variables used by the other OSPI.py software and plugins.
I want the capability to track ALL state changes in my sprinkler system(s). I want to know when a program changes, when the rain sensor comes on and goes off, when someone changes the minutes per station within a program, how long a valve has been actuated, how much water flowed through my irrigation meter during the actuation, if there were any unexpectedly fast flows (a gusher), or low flows (bad valve). I would like to add micro-climate sensors to adjust my zones (the north side of my home is much cooler than the south side, for example, and varies with the season, leaves on or off the trees, etc.) Obviously, I don’t intend to do all of this at the onset, but my design style is to design for the general and then narrow down to the specific. I would like to design for the general framework, and then populate it with one instance. It’s a bit like creating a wiki (which is really a very simple technology) and letting Wikipedia emerge rather than collecting standards committees and experts to write the “correct” encyclopedia. So, what are the simple initial conditions we can create to allow other innovation to emerge?
It seems to me that we need to trigger the messages at the time of the state change in the main program. I was thinking of the mpay global variable as a string that would control which messages to be published. Maybe others have a better idea, but my first thought was to assign character codes to each type of messaging: “s” would represent sprinkler state, “p” would be program state, “r” would be rain state, for example. mpay=”srp” would cause OSPI to publish all events, mpay=”s” would publish only sprinkler events, etc.
munnecke,
You are right that the gv module is a repository for global variables which can be shared among the main program and other modules such as plugins just by importing it. You can assign new vars to it if needed.
There are some existing sources of state data. For example, the mobile_app plugin provides the “/jx” urls such as ‘/js’ which returns the current state of stations. There is also an /’api/status’ URL which returns some state info but these return the information only when requested via an HTTP Get request. I am looking into ways to broadcast status data when a change occurs. Any ideas in this regard are welcome. Your list of things you want to track is very helpful.
I ‘m also experiencing the drought (Sacramento area). There are dead lawns everywhere. A recent study released by (I think) U.C. Riverside concluded that one of the most effective water conservation measures is tiered pricing. The state of CA even has an Urban-Drought-Guidebook that recommends “large per billing unit price increases” to water suppliers to help reduce usage. I am thinking about writing a plugin that will allow a user to set a cost/volume restriction on irrigation. It would require the irrigation system to be calibrated by e.g. running each zone for a specific time and determining the amount of water used by taking water meter readings before and after. This should become useful as the cost of water goes up.
Another project on my workbench is an Arduino based system for connecting soil moisture sensors to the OSPi. My plan is to use the modbus protocol over RS-485 serial. It could be used for other types of sensors as well. The Raspberry Pi can easily communicate with an Arduino via serial using a USB cable.
Anyway, this is a fun and useful project.
Dan
<div>
<div>
<div>Thanks, Dan.</div>
I’m looking at interfacing this water meter http://www.dwyer-inst.com/Product/Flow/WaterMeters/SeriesWMT2 which would require a process or microcontroller to count the pulses. It could update a global variable in gv.py as well as publish it via MQTT… Don’t know anything about modbus, I have about a 150 foot run from the meter to the controller. I was thinking about a solar powered zigbee transmission link. I was also wondering if it might be possible to transmit the pulse information over one of the sprinkler zone wires. The wires would still have to work as the 24v actuator for the zone’s valve, but would also carry the meter pulse info back to the controller, saving me from having to run a long cable. Maybe wireless is the way to go. I’m happy to not reinvent the wheel, if there is something out there already.</div>
Having status APIs is nice, but I’m particularly interested in state <i>changes</i>. I want know when someone has changed a program – say, a new daily schedule schedule. Or adjusted the zone time. It would be good to see time/week per zone, which be a good overall metric. And I want to go back and see what I was doing last year at this time… minutes/week (or gal/week) would probably be the best metric for this.</div>
<div></div>
<div>re: soil moisture sensing. This is a very complicated thing to measure. Simply measuring resistance is more a function of salt content than moisture. It is also dependent on the type of soil, plant, temperature, etc. Check out http://www.allianceforwaterefficiency.org/soil_moisture_sensor-intro.aspx</div>
<div></div>
<div>I happened to have attended the University of California, Riverside, that you mentioned. And as a further coincidence, I got my first job at the USDA Salinity laboratory, and my first taste of computer programming, doing soil moisture determination with the use of soil psychrometers. (see fig 13). This was a ceramic bulb (that simulated the root’s access to the soil) within which a thermocouple was placed. I’m a little fuzzy as to how it worked (this was 1968 🙂 There might be some interesting opportunity to do some microcontroller control of the thermocouple, and 3d printing for the psychrometer body, as well.</div>
<div></div>
<div>gotta run now, but I’ll get back on the ideas for the MQTT publishing points.</div>Hi Dan, munnecke
I have hacked the email_adj plugin (as Dan suggested), and have it sending MQTT messages as follows
– to topic OSPI/system when the system starts (presumably when the ospi service is restarted
– to topic OSPI/start when a station turns on
– to topic OSPI/run when a station turns off
– to topic OSPI/rain when the rain sensor detects rain
There is still some more work to tidy it up, but essentially the plugin has a loop running every second. It detects changes in state by looking at variables in gv and comparing them against the previous values. Almost any variable can be monitored providing it is accessible within my plugin (called surprisingly MQTT!). Doing the detection of events from within the one module by monitoring the content of the variables is going to be far easier than trying to modify the routines that make the changes to the variables. To do that you would need to scan code in all modules, and also rely on authors of as yet unwritten modules to add the necessary hooks in their plugins – very tricky.
There is a web page accessible from the plugins menu item that allows you to set parameters such as the hostname, and the port of the MQTT broker. There are check boxes to select which of these messages you want. I will probably allow the user to input the topic for each of these events, but that is not in place yet.
It would be easy to check for changes in any of the variables in gv. So detecting changes to station names, programs, watering percentage etc are all possible.
This method has much less load on the OSPi. However if the OSPi goes down (or the service stops and doesn’t restart), then you would get no messages. I will look at a mechanism for detecting this, as I am concerned that my sprinklers may not go on if the OSPi doesn’t recover properly say from a power failure.
I am interested in hearing any comments as to ideas for improvements/ things I have done wrong, suggestions for alternatives or anything relevant, etc etc.
One warning for people regarding the MQTT broker – don’t install it from the Raspbian repository -it will just hang when called from a Raspberry Pi. Go to the link I ooriginally provided in my initial post and install it from there.
Regards
Ian
I think that the MQTT Retain feature will automatically provide Node-red with the latest state, making life much simpler.
munnecke – I have put station run times into Emoncms (running on another Pi). I will also be collecting water metering data and placing this in the Emoncms database. From there it is simple to use the facilities of Emoncms to calculate running totals, convert units, apply calculations (including combining data from other feeds) etc. It is however only suitable for analog data (including pulse counter data) ie it is not suitable for status information eg enabled/disabled, and so on.
Once I have got the plugin to a more advanced stage I will make it available for testing and feedback.
In node-red I will take this data and direct it to different systems, such as Emoncms, XBMC, email etc. I will also develop flows that will effectively give node-red a set of globals that mirror the OSPI variables. That should allow people to come up with some inventive uses of the data.
Have you considered a general state-change detector that looks at the whole Export JSON? That could log everything that changes in the controller, kind of like the Recent Changes log in a wiki page. At each interval it would compare the current JSON to the one from the past, and then note the differences. This wouldn’t have to run so frequently, but it would be more general. It would also allow a user to roll back to a previous setting, say to settings of the same month from the previous year. I guess a simple version of this would be a simple cron job that would Export to a file each night. Get the data recorded, and let the future figure out how to detect the changes, if they want to 🙂 In any case, this could be valuable information for going back to try to figure out what was happening/when.
re: “If the OSPi goes down (or the service stops and doesn’t restart), then you would get no messages. I will look at a mechanism for detecting this, as I am concerned that my sprinklers may not go on if the OSPi doesn’t recover properly say from a power failure.” Yes, this is a critical problem. For starters, maybe each controller could publish a “heartbeat” message, saying “I’m alive” with various parameters, e.g. Time since reboot, CPU temperature, voltage, battery status, etc. Maybe this could be every hour. The first time after reboot it could publish another “wake up” message.
p.s. I’m planning to run at least two OSPI controllers, and may adapt one to control my pool, as well. Maybe the publishing topics should be based on the assigned name of the controller, not just “OSPI”
I’m playing with some ideas about using “Promise/Futures” logic for this. Basically, a controller has a set of promises that it has made and is working to fulfill. (“Deliver program
of watering by dawn,” “publish a <Heartbeat> message every hour,” “export the all of the controller’s variables at midnight”), etc. When it boots up, it publish a
For a more general solution, I’ve been looking at Promises/A http://wiki.commonjs.org/wiki/Promises/A and Twisted Twisted: http://twistedmatrix.com/documents/current/core/howto/defer-intro.html. A controller would publish (or commit to) a promise, which the network would then see as a future event. (e.g. running a program by dawn). The controller would then flag the promise as completed when done, or the network could detect that the promise was never completed (or the controller could admit failure). I think, (but am not sure 🙂 that this could simplify the scheduling process when it comes to rain detection, exception days due to drought restrictions, microclimate sensors tweaking a zone, etc. It also builds a foundation for doing alerts when things don’t happen as expected. We can’t expect the failed task to announce that it didn’t work, so there has to be some external promise/completion process to drive this.
I’ve got to get my sprinklers installed and running. I haven’t been able to get the SSH key pair generation to work right… I’ll probably set up two OSPI controllers, an XMBC (or two), and and MQTT/Emoncms machine, too.
Ian: what metering hardware/communication system are you using for your Emoncms setup?
Just an FYI.
I have started testing the Blinker Python module:
https://pythonhosted.org/blinker/
It enables message sending when an event occurs rather than by polling. It can even include a data payload with the message. I have a working proof of concept plugin that prints a message to the python console whenever a zone is turned on or off. It is only 5 lines of code plus 3 lines in the gpio_pins.py file:
from blinker import signal zone_change = signal('zone_change') and at the end of the set_output() function: zone_change.send()This requires installing a non-standard Python module, something I have avoided in the main program code. I was able to install Blinker on both my OSPi test rig and my Windows development system with
"pip install blinker"
It looks like there are about 6-8 places in the main program code where changes are handled. these could have messaging code added that would be ignored if the blinker module was not present.
One possibility for detecting if the Pi is up and running is to simply ping the URL. A heartbeat would be trivial to implement in the ospi software.
Dan
- You must be logged in to reply to this topic.