- This topic has 48 replies, 4 voices, and was last updated 10 years ago by
.
-
Posts
-
Hi Dan
I am thinking it would be a good idea to have the MQTT module become a general change of state publisher via MQTT. If the blinker python module works and Dan is prepared to build it into the core OSPi code, then my MQTT plugin could subscribe to these events and send MQTT messages. I will develop a Node-red application to log the data.
One reason for thinking this way is that I noticed the Android Opensprinkler app polls the OSPi three times every three seconds for change of state via http://host/jc?pw=xxx, http://host/jo?pw=xxx and http://host/js?pw=xxx calls. It obviously needs to keep its knowledge of the OSPI status up to date. I had noticed that my OSPI seemed to be struggling at times when I was polling it in a similar way, and it became unresponsive at times. Clearly it is not possible to run more similar applications on the Android and IOS world, as I found out. It would be best if these type of apps did not have to poll OSPi for events.
So your investigation of the Blinker module is timely! Let me know how you get on, and when it is likely I could have some source code to test.
munnecke
Let me describe my setup. I started with Mythtv and have an old desktop running this. It has a large hard drive and records any programs we are likely to watch (and autoexpires them when the disk is nearly full. I have 2 TVs and each have a Pi running XBMC. One is networked via wifi, and the other by ethernet over powerline using D-Link powerline AV 500. The one using wifi is on a different circuit so I can’t use the D-links. Hence the wifi, but it works well. I also have XBMC on my Nexus 7 tablet and my wifes iPad. I use the Yatse application on the Nexus 7, and Nexus 4 phone to control XBMC. This is not a new house- we bought it in February and I have used wifi and powerline to avoid retro fitting Cat6e cable all over the house).
I have installed Emoncms on another Pi. This one has a small hard drive (even the small ones these days are massive overkill). I did this because the SD cards have a limited life (ie they have a limited number of write cycles and the Emoncms data logging application is expected to run into issues on SD cards. I use an Emontx at the main meter (fitted with 4 CTs) to monitor power consumption. This is essentially an arduino and it transmits by 433MHz radio using the RFM12b radio module. This system is partly to investigate whether a solar power system is worthwhile. I have also installed two EmonTH’s (arduinos)- monitoring temperature. These also transmit via 433MHz radio (which apparently has a range of several hundred metres). One thing that is non standard in my setup is that Node-red intercepts the incoming radio signals (via the serial port) and forwards the data to Emoncms. This is because Emoncms seems to have little or no query capability, although there is an export CSV function. The Emontx and EmonTH are well engineered and work flawlessly.
So as summer is approaching I needed to get my garden reticulation working. The old controller that came with the hosue needed replacing. Given the above infrastructure it OSPi seemed a good fit. The OSPi is connected to the network via the WiPi module. Node-red and MQTT have allowed me to integrate all these systems and potentially customise them without modifying their core code.
My immediate plan is to introduce some sort of logging capability for OSPi. With your and munnecke’s input the scope has grown! The next plan is to measure the water consumption using the small water meters I provided a link to. These provide a pulse output. I have a number of options to integrate these – I could connect them to the EmonTX, or to OSPi, or use a dedicated Arduino. The distance between the meters and the Emontx and OSPi is only a meter or so.
Ian
Ian and munnecke,
There is a new branch on the OSPi GitHub repository named Blinker_test. You can pull an update from the repository then
git checkout Blinker_test
to switch to the new branch.
It includes a new plugin named notify_change.py. You may need to set the permissions to executable to make it load. You will also need to have the blinker Python module installed (see previous post). If you run the ospi program from the command line and make some changes to the program in the UI, such as running a run-once program you will see the changes reported in the console.
This is just a test of Blinker’s signalling but it reports most changes a user can make in the main program. Take a look at the code in notify_change.py for more information. The other changes are in webpages.py and gpio_pins.py.
I think it would be good to have this capability as a standard feature of the ospi program. Especially if we can get it talking to Node-red through MQTT. I have been watching some intro MQTT videos on YouTube and some people claim it is the state of the art for the internet of things.
Thanks for starting this thread.
Dan
Thanks Dan. I agree this would be good to have as standard functionality. I will update my plugin to reflect this new capability. My plugin will now simply be a configuration plugin that enables different messages to MQTT and allowing the user to specify the topics for these messages. My understanding is that notify_change.py would be modified to send the MQTT message (instead of printing on the console or perhaps as well as? What do you think?). I think that it is possible to send the whole of gv in the message body together with a time stamp. I will also develop some Node-red flows that initialise global variables corresponding to items in gv, and update them when they change.
The topics I will have as default will be something like OSPI/login, OSPI/zones-changed, OSPI/controller-changed, OSPI/options-changed, OSPI/reboot, OSPI/station-names-changed, OSPI/program-changed, OSPI/programs-deleted, OSPI/programs-toggled. Users can changed them as they wish. The Node-red flows will need to incorporate some mechanism to notify when the global variables change, and this would then trigger any user defined flows to process the change.
Once we have tested all this we need to provide documentation for plugin developers so they can incorporate similar functionality. eg my plugin would need to have a mechanism for notifying changes to the Topic names, and provide global variables in Node-red that correspond to these variables. Other plugins would need to do the same.
From a performance perspective, changes would be very infrequent (maybe daily, or bi-weekly) – a very significant improvement over the polling approach. I am confident that there will be some easy way to get the updated gv values from Node-red to Android and IOS apps such as the mobile OSPi app.
munnecke: your wish to have a record of changes to your system could then simply be achieved by archiving the updates to gv to file every time one of these messages arrived. You would need to develop your own application to play back/ recover these changes from the archive.
Dan – I think MQTT is getting more and more acceptance in this whole home automation world. So by placing the core OSPi variables in MQTT, it open up OSPi. Some users may not want to use Node-red, and that is OK as they can get the data directly from the MQTT broker. If an alternative emerges in the future to MQTT it would be trivial to allow it as an option in my plugin.
I think that you would be better off using MQTT to send data from your arduino rather than Modbus. (I have 35 years experience in the SCADA field – this is much easier!)
Ian
wow… things are happening fast… Great work, Ian and Dan. Hopefully, I’ll get to this tomorrow to test out. In the spirit of “eating my own dog food,” I plan to install two OSPI controllers to control my real-world sprinklers, so I expect to see the reality of what we are coding here. I played with this:
I’m wondering whether there might be some interesting things in the zeroconfig mDNS stuff… (I really like the “zero config” name). It adds the ability to discover services dynamically (“domains” such as printers, servers, music libraries, etc.). Maybe we should add a domain “sprinkler” that then has topics for all the interesting things sprinklers do. The MQTT server(s) could become a domain(s) within the rDNS service, which would name the “last will and testament” fail-over server if the primary one failed.
here are some links:
http://www.ryukent.com/2013/09/a-local-url-instead-of-an-ip-address-for-your-raspberry-pi/
https://code.google.com/p/pybonjour/ – python-based framework (I got it to work a bit, even though it is very old code.)
The functions in the notify_change.py plugin are really just stubs. They print a message to the console in order to demonstrate that they work. They can/should be extended to send messages to an MQTT broker, do data manipulation etc. The python-mosquitto client modules seem like they could work here.
The blinker signals can also include a data payload. When a user performs an action such as selecting an option, the UI sends a GET request with a query string containing the selected option to a URL which triggers some processing. The blinker signal from such an event could include a Python dictionary object containing the elements of the query string, providing more specific data about the change. I ‘m just learning abut MQTT but it seems like sending the entire contents of the gv.sd dictionary with each change is a bit of overkill. It would require further processing to figure out what the change actually was was.
This new signalling feature will certainly make writing plugins a lot easier and eliminate most of the looping/polling that is now necessary to get data about the state of the system. It has the potential to enhance performance by eliminating HTTP REST calls:
https://mobilebit.wordpress.com/2013/05/03/rest-is-for-sleeping-mqtt-is-for-mobile/
Ian,
The idea of using MQTT for the sensor network crossed my mind but there are some reasons I want to at least try modprobe. There is a separate thread on that so I won’t go into it here.
https://opensprinkler.com/forums/topic/who-accept-a-challenge-to-write-plugin-with-soil-moisture/
Dan
Thanks Dan.
At the moment I am just getting a plugin working that has a web page to allow the user to say which of the notifications they want to receive, and what the MQTT topic should be for each notification. I am also extending your stubs to send a plain text message. When that works I will then look at the payload will be for each type of notification. So it is good we discuss it now.
Sending a python dictionary object containing the elements covered would make it easy to use that and send those elements in the payload. Maintaining the dictionary could be a problem, unless it could be automatically generated. I wonder if that is possible?
I wasn’t too concerned about the traffic as these events are relatively rare. eg Once OSPI is configured, the main events would be stations turning on and off, and these are a few per week, or in the worst case per day. If the python gv object was sent, anyone using it could simply replace all values. This is easy to write and maintain at both ends. However when I get to that stage I will do some tests.
Of course this will only improve performance if the apps making HTTP calls are modified to take advantage of it.
I have taken Dan’s version of OSPi with the notify_change.py plugin and modified it (Sorry – I am not familiar with Git, I tried, but eventually just downloaded a zip file and extracted the files. When I have some more time I will read more about Git!).
I have a working version (call it early alpha!) that uses the notifications Dan introduced to send MQTT messages whenever these events occur. There is a web page that allows you to select which notifications you want published to MQTT, and what the topic will be for each. You can also set the MQTT host and port. There are defaults for all these parameters.
I haven’t developed any Node-Red flows, but I have used an Android app called myMQTT to check the MQTT messages are being sent. They are set with the retain feature of MQTT, so that if you subscribe after the event, you will get the latest message in the topic. They appear almost instantly on my phone.
Please excuse the code (whats wrong with Fortran anyway!) and feel free to make any suggestions. I am very much a Python novice, and struggled with some aspects – getting numerous errors. You will note some routines duplicated in the modules mqtt.py and notify_change.py – it was easiest that way and I am sure someone can let me know how to avoid it! The module mqtt.py is mostly about getting the parameters input by the user, whereas notify_change.py uses the stubs Dan created to process events.
This needs a lot of thought as to what data is to be sent for each event. I am quite sure I have got it wrong in many cases, and will be spending more time looking at that aspect. The message that is sent can be a string (“Someone logged in”, or a python object eg {“snames”:[“s1”.”s2″,”s3″,”s4″,”s5″,”s6″,”s7″,”s8″]}. The topic should indicate which event the message is in response to.
mqtt.py and notify_change.py must go in the plugins directory (and be set executable). mqtt.html goes in the templates directory.
Node-red can be used to process these messages, if necessary reformatting them and sending them on to other applications such as email, storing in a database, and so on. The example I posted earlier provides examples that would be a good starting point.
I see that I can’t upload files with extensions of .py and .html. So I have renamed the files to be mqtt.py.txt, mqtt.html.txt, and notify_change.py.txt. Hopefully that will work. You will just have to rename them when you download them.
Thanks, Ian… I’ve been busy with other issues recently, and haven’t had time to keep up with all the stuff happening here. re: Fortran: yes, those were the good old days, punching cards, dropping decks, waiting for the computer operator to run your job, waiting 2 hours to find out that you missed a comma somewhere and blew the whole job. But I also learned how to make cool paper airplanes by stapling together punched cards. Just a quick calculation: my Raspberry Pi has 43 million times the “disk” storage and 80,000x the main memory, of the IBM 360/50 I learned on.
Dan: I did try to convert my controller over to OSPI from my working Arduino-based version, but couldn’t figure out how to program the individual zones to specific times… they were all fixed to the same time (15 min). Am I missing something here?
P.S. my water district is now talking about doubling the water rates for the highest tier usage, which will really put a focus on more efficient watering…
@ Ian,
Thanks for the files. I will take a look as soon as I have a chance.
The best source of info on Git is the Pro Git book available for free download:
On my windows system I find tortoisegit very helpful:
https://code.google.com/p/tortoisegit/
Strange that it is on Google Code.
Some good Python resources are:
http://www.tutorialspoint.com/python/
http://www.python-course.eu/course.php
and if you are doing a google search for Python… be sure to look for StackOverflow links.
If you are moving from a recent rev of the Arduino OpenSprinkler, it has a new feature that allows individual run times to be set within a single program. It is on the todo list for ospi but is not available yet. You can make a program for each zone which actually gives you more flexibility. For example you can more easily change all the settings for a zone and enable/disable individual zones on the Programs page of the native UI.
Wow! Doubling top tier rates is a big jump. A vloume/$ limiting plugin seems like a worthwhile project to start ASAP. Or at least one that would display what you are using. It will probably take some fiddling to get to something that is really practical but if we don’t start…
Does hollerith code ring a bell? I learned system 360 programming in 1969 but spent the next 16 years in the (plant) nursery business.
Dan
An update.
I have decided to allow the user to input the string that will be sent with each message, so it can be customised. I will set some defaults. That effectively resolves one of the most difficult issues I faced. So the input screen will change.
With this change each user can decide which events will trigger an MQTT message, what the topic will be, and what the payload of the message will be. I may as well allow them to set the QOS and Retain values of the MQTT messages as the optimum values for these may depend on how the messages are to be used.
Ian
I have a version that is close to being ready for wider testing. This consists of two plugins – mqtt.py and notify_changes.py. mqtt.py is the data entry part where you define the events you are interested in, the topics, and the content of the messages. notify_changes.py does the actual work, and is an updated version of Dan-in-CA’s version that fills out the “stubs” he provided.
All topics and message content is user definable, either from the data entry screen, or by defining the message content in a file.
I have attached a Word document which documents what I have done, and provides a screen shot of the data entry form. Please feel free to make any suggestions (or complaints).
I tried to install the latest update, per the instructions for “upgrade.” The GIT logic all worked OK, but it just gives me an “Internal Server Error” when I try to access it via HTTP.
I’m confused as to what version folks are working with, and how to get back to an operating version.
munnecke
As I have now quickly skimmed the e-book on Git, I may be able to help. I am guessing you follwed Dan’s instructions to get the Blinker_test version of OSPi.
If you go to the OSPi directory (or whichever directory you have as your working directory), and type git branch it should show you two branches – Blinker_test and master. Blinker_test should have an asterisk next to it. If this is so, then if you now type git checkout master you should be back to your original configuration.
Before you do it, (and if you are running the Blinker_test branch) could you go to the OSPi directory and type
sudo service ospi stop
sudo python ospi.py
Then you should see the console messages. Fire up your web browser and try to get the Internal Server Error message again. You will probably see a python error message on the screen. That would be helpful information to see why your system got this error (I am running the Blinker_test branch currently with no issues!)
Ian
Ian,
Thanks for the update and documentation.
In order to test MQTT I have installed both IA92
http://www-01.ibm.com/support/docview.wss?uid=swg24006006
Old but still useful like me and also MQTT-spy
https://code.google.com/p/mqtt-spy/
on my Windows system. They are both java based and should run on any OS with a JRE installed. This enables me to monitor the messages from the MQTT broker.
I have been testing the addition of the Blinker module to the OSPi distribution. It looks like both Blinker and the Mosquitto client can be added without licensing problems. This will eliminate the need for users to install the modules as a separate step.
I also plan to add a data payload to the signals that Blinker provides which can be used in the MQTT message to provide a bit more specific information about the state of the system after a change. there may also be some additional activity in the core program that would be worth reporting via Blinker.
Thanks for your contributions to the project. This will make plugin creation much easier.
Dan
Thanks Dan
If the variable for the Blinker data payload is added to gv.py, then I would not have to do anything. The user could configure the MQTT message payload to include this information. It would probably be a good idea though to include this variable in the default configurations supplied.
I have posted on another topic a problem I encountered with the weather_level_adj plugin. This plugin encountered a serious error which it could not resolve, and printed a message (which in normal operation I would not see), then repeated this constantly. It seems that this could be another use for MQTT – to enable a plugin to get an error (or success?) message out to where it can be seen?
Request for input.
I have most of the functionality of the MQTT plugin working. This plugin allows you to configure the topic and content of MQTT messages sent in response to events (logged in, reboot, options change, controller change, station names change, programs change, zones change).
To configure the message content, you define a set of parameters as comma separated variables. The parameters can be a text string, a variable name from module gv, or a defined keyword such as Time, CPUtemp, CPUusage, RAMtotal, RAMused, RAMfree, Getlogin, and Uptime. The gv variables are as defined in gv.reference.txt available in the /home/pi/OSPi directory on your system.
It is possible to define additional keywords that allow you to specify some other values not available via the gv module. These could be values derived from other variables, or obtained from other plugins. I would like to hear from anyone who has any suggestions along these lines (it would be helpful if you could explain how to calculate or otherwise obtain such values).
I still have to work on the Node Red example to process these messages, although the example will build on the earlier examples I posted.
Regards
Ian
Hey Ian… I haven’t forgotten you… I just haven’t been able to get my system to work since I tried to do the upgrade. I have done the Git Pull and the Blinker upgrade – and all is up to date… but whenever I try to run the software I get an “invalid server error”
here’s my error listing:
munnecke@ospilab /home/pi/OSPi $ sudo python ospi.py
plugins loaded:
mobile_app
monthly_adj
relay
weather_adj
weather_level_adj
Starting timing loophttp://0.0.0.0:8080/
Monthly Adjust: Setting water level to 100%
Traceback (most recent call last):
File “/home/pi/OSPi/web/application.py”, line 239, in process
return self.handle()
File “/home/pi/OSPi/web/application.py”, line 230, in handle
return self._delegate(fn, self.fvars, args)
File “/home/pi/OSPi/web/application.py”, line 420, in _delegate
return handle_class(cls)
File “/home/pi/OSPi/web/application.py”, line 396, in handle_class
return tocall(*args)
File “/home/pi/OSPi/webpages.py”, line 94, in GET
return template_render.home()
File “/home/pi/OSPi/web/template.py”, line 1020, in template
return self._base(t(*a, **kw))
File “/home/pi/OSPi/web/template.py”, line 881, in __call__
return BaseTemplate.__call__(self, *a, **kw)
File “/home/pi/OSPi/web/template.py”, line 808, in __call__
return self.t(*a, **kw)
File “templates/home.html”, line 311, in __template__
<td class=”scheduleTick” data=”2″></td>
IndexError: list index out of range192.168.1.5:59442 – – [15/Nov/2014 19:50:23] “HTTP/1.1 GET /” – 500 Internal
munnecke
I am not familiar with that part of the system which is the basic web templating part. However it looks like it is trying to load the home page and getting confused when listing the stations. I assume from your earlier posts you have more than 1 board ie more than 8 stations so it could be part of the multiboard logic, (which I dont use).
Although you have that error you can access other web pages. So do a sudo service ospi stop, then deactivate all plugins (mobile_app, monthly_app, relay, weather_adj, weather_level_adj. To do this go to the plugins directory and do a sudo chmod -x mobile_app.py etc.
Then restart ospi with sudo python ospi.py from the /home/pi/OSPi directory and try to load the home page. If it was one of the plugins causing it (I doubt it – but it is good to remove as many variables as possible) the problem should have disappeared.
If you still have the internal server error, you can still load other pages, so load http://ospi ip address:8080/vo where ospi ip address is whatever IP address your OSPI system has.
You should see the options page. Click on the Stations handling option and check your system is configured properly, especially that the number of extension boards is correct. You could also set this to 0 as this should work.
If this checks out then try http:// ospi ip address:8080/vs and see if you get the right number of stations displayed.
Let me know how you get on. I suspect that there is somehow an inconsistency in these options introduced when you installed the different versions.
Regards
Ian
@munnecke I doubt this is a plugin issue. The most likely thing is the settings inside the /data folder are corrupt. The easiest fix with these types of OSPi issues, in my experience, is to wipe the entire folder and re-clone it. Try that and see if it helps.
The MQTT plugin is ready for wider testing. You will need the Blinker Test version (see https://opensprinkler.com/forums/topic/using-mqtt-to-log-data/page/2/#post-34412) for instructions – at this stage I am not up to date enough with Git to upload it to the repository, so you will have to use the attached files.
To use it, you need the files mqtt.py and notify_change.py installed in your /home/pi/OSPi/plugins directory, and the mqtt.html file in your /home/pi/OSPi/templates directory. The mqtscfile.dat, mqtlgfile.dat and mqtzcfile.dat are optional and if used should be placed in your /home/pi/OSPi/data directory. The attached notify_change.py file replaces the one provided in the Blinker test branch. The mqtt.py module is new. If you don’t want to install the .dat files, just change the message specification for the zone change, station name change and logged in events so they don’t use the “file” option.
The mqtt.py and notify_change.py files need to be made executable to enable the plugin. These files are attached, but due to forum restrictions I have had to name them mqtt.py.txt and notify_change.py.txt. You will need to rename them before using them.
When using it for the first time use the plugins menu item to go to the MQTT configuration screen. Enter the host address and port of your MQTT broker here as a minimum. The attached word document should be self explanatory, and replaces the version I provided earlier.
I am working on a Node Red application that will process these MQTT messages. In the interim period until it is finished, there are some options for testing described in the documentation. The simplest is to use Node Red to subscribe to the topics you define and send the message to email.
NB: Apparently I can only attach 4 files, I will attach the optional .dat files in a separate post.
Here are the .dat files mentioned in my earlier post. These are used to specify the message content instead of using the data entry form. Since I simplified the message format, it is much shorter and therefore it is probably unnecessary to use the “file” option. However they are included as examples.
Update
I have been working on the Node Red application to process the MQTT messages that are sent in response to events in OSPI. These events are either users changing the configuration through data entry, or for stations starting and stopping.
To simplify things I added a new web call. If you enter in your browser http://ospi-host:port/gv?pw=password then you will get all of the variables in the gv module sent as a JSON encoded string. See gv.reference.txt in the OSPi directory for a description of these variables. I also added the ability to specify the QOS and Retain features of the MQTT messages to make testing easier.
This simplified the Node Red processing as it was no longer necessary to track where I was in requesting data from OSPi.
The resulting Node Red flow is shown in the attached jpeg. The section labelled “Initialisation and polling” requests the gv data from OSPi. This section builds the request, then processes it. The function fnProcessOSPIhttpresponse places all the gv variables into globals within the Node Red environment. These globals can be accessed from any Node Red function in a similar manner that modules in OSPi can access these variables by importing gv (Python talk there!). This section of the Node Red flow is triggered once an hour and this will form the basis of a system to inform me of failures in the OSPi system.
The remainder of the flows simply respond to MQTT messages. The “Respond to events” section processes messages such as login events, reboot events or failure of the OSPi system to respond to http requests.
The section labelled “something started or stopped” responds to changes in the status of stations (ie sprinklers start or stop events). The start events are sent to my TV and email, and the stop events are similar. The duration is calculated and the stop event plus run time is also sent to Emoncms to log.
The section labelled “Someone changed something” responds to someone changing the OSPi configuration by entering changes in the web interface. I have simply processed tthe “change station names event”. These changes are mirrored in the Node Red global variables, and sent to email and XBMC for display on my TV.
I am sure that I don’t need all these notifications but this was proving the concept. With this system, you can be notified of most significant events in OSPi, and process them as you wish, either logging them into systems such as Emoncms or Openhab (or others). All the important variables OSPi uses are available in Node Red.
I will continue testing and will soon make the Node Red flows, and the update MQTT.html, MQTT.py and MQTT.html modules available.
The attached file is a screenshot of the Node Red flows involved. I am surprised that this application is so straightforward. There is very little code involved in total.
I will continue testing but hope to have the updates to the OSPi modules I posted and the Node Red flows ready by the weekend together with updated documentation.
Regards
Ian
Attachments:
- You must be logged in to reply to this topic.