Forum Replies Created
-
Posts
-
Sorry I took so long to respond. I have been away for a couple of days.
I have attached my configuration file. I wasn’t totally sure what would be needed, so I monitored the OSPi service and the Mosquitto MQtt broker service. I am also monitoring a couple of other Pi’s so I monitor the network by letting them Ping each other. I will probably stop this as I doubt it is necessary. Their addresses were 192.168.0.150 and 192.168.0.58 if you are wondering what those lines are.
Ian
I have attached my version of weather_level_adj.py – if you place this in the plugins directory in place of the version that is already there, and make sure the new version is executable, then you will solve the problem.
Due to the restrictions on attaching files in this forum I have had to rename weather_level_adj.py to weather_level_adj.txt You will have to rename it back to weather_level_adj.py after you download it.
This version works but a better (more elegant and complete) solution has been proposed by ShawnHarte in this thread. Feel free to implement that instead if you wish.
The reason it calls wunderground so many times is that the error routine retries every 60 seconds. I have changed this in this version to every 1800 seconds to avoid the problem.
Regards
Ian
Dan
I have been thinking a bit further in how my plugin integrates with the rest of the system, and I think it may need some more work!
I have hijacked your notify_change routine which had stubs for code. However the manner in which I have implemented it means it simply sends the appropriate MQTT messages as I intended, but it does not allow other routines, systems or internal plugins to be notified of the changes.
If the Blinker system allows the signal to be received in multiple places, then each plugin could simply implement the same code as in notify_change.py that responds to the signal.
If not then the notify_change plugin will need to know who wants to know of these events, and then call a series of routines similar in purpose to my publish_mqtt function.
It could be that other external systems require to be notified of changes eg the mobile app. If MQTT is defined as the standard way this will happen, then the plugin I developed will meet this need. If however they need/want direct notification then some changes will be required.
I am happy for my plugin to be adapted to fit in with any overall plans for OSPi.
Regards
Ian
.
Shawn
To clarify. The plugin normally retrieves Wunderground data as part of a main loop which executes on an hourly basis. The main loop however is in a Try: Except: structure and it is in the Except: condition that the self_sleep(60) statement occurs. Perhaps the author assumed that any error would be in the call to retrieve the data, and that retrying every 60 seconds would soon fix the problem? However the Except: catches all errors, and this particular one (the float problem) would not resolve itself until it rained next!
It would be possible to improve the error checking, but I doubt there is a lot of point. Changing this particular self_sleep(60) to self_sleep(1800) ensures that you don’t exceed your Wunderground limits for retrievals in the event of as yet unanticipated errors. Your safe_float suggestion fixes the problem I encountered in an elegant manner.
Regards
Ian
Thanks Shawn. That’s a much better way to handle it!
I think it is still worthwhile changing the self_sleep(60) to self_sleep(1800).
Ian
Additional files. .dat.txt files for the /home/pi/OSPi/data directory after renaming to remove the .txt extension.
The Nodered.txt file is the sample Node Red application. Open this file, copy it to the clipboard, and in Node Red import it from the clipboard.
Update
I have been working on the Node Red application to process the MQTT messages that are sent in response to events in OSPI. These events are either users changing the configuration through data entry, or for stations starting and stopping.
To simplify things I added a new web call. If you enter in your browser http://ospi-host:port/gv?pw=password then you will get all of the variables in the gv module sent as a JSON encoded string. See gv.reference.txt in the OSPi directory for a description of these variables. I also added the ability to specify the QOS and Retain features of the MQTT messages to make testing easier.
This simplified the Node Red processing as it was no longer necessary to track where I was in requesting data from OSPi.
The resulting Node Red flow is shown in the attached jpeg. The section labelled “Initialisation and polling” requests the gv data from OSPi. This section builds the request, then processes it. The function fnProcessOSPIhttpresponse places all the gv variables into globals within the Node Red environment. These globals can be accessed from any Node Red function in a similar manner that modules in OSPi can access these variables by importing gv (Python talk there!). This section of the Node Red flow is triggered once an hour and this will form the basis of a system to inform me of failures in the OSPi system.
The remainder of the flows simply respond to MQTT messages. The “Respond to events” section processes messages such as login events, reboot events or failure of the OSPi system to respond to http requests.
The section labelled “something started or stopped” responds to changes in the status of stations (ie sprinklers start or stop events). The start events are sent to my TV and email, and the stop events are similar. The duration is calculated and the stop event plus run time is also sent to Emoncms to log.
The section labelled “Someone changed something” responds to someone changing the OSPi configuration by entering changes in the web interface. I have simply processed tthe “change station names event”. These changes are mirrored in the Node Red global variables, and sent to email and XBMC for display on my TV.
I am sure that I don’t need all these notifications but this was proving the concept. With this system, you can be notified of most significant events in OSPi, and process them as you wish, either logging them into systems such as Emoncms or Openhab (or others). All the important variables OSPi uses are available in Node Red.
I will continue testing and will soon make the Node Red flows, and the update MQTT.html, MQTT.py and MQTT.html modules available.
The attached file is a screenshot of the Node Red flows involved. I am surprised that this application is so straightforward. There is very little code involved in total.
I will continue testing but hope to have the updates to the OSPi modules I posted and the Node Red flows ready by the weekend together with updated documentation.
Regards
Ian
Attachments:
Here are the .dat files mentioned in my earlier post. These are used to specify the message content instead of using the data entry form. Since I simplified the message format, it is much shorter and therefore it is probably unnecessary to use the “file” option. However they are included as examples.
The MQTT plugin is ready for wider testing. You will need the Blinker Test version (see https://opensprinkler.com/forums/topic/using-mqtt-to-log-data/page/2/#post-34412) for instructions – at this stage I am not up to date enough with Git to upload it to the repository, so you will have to use the attached files.
To use it, you need the files mqtt.py and notify_change.py installed in your /home/pi/OSPi/plugins directory, and the mqtt.html file in your /home/pi/OSPi/templates directory. The mqtscfile.dat, mqtlgfile.dat and mqtzcfile.dat are optional and if used should be placed in your /home/pi/OSPi/data directory. The attached notify_change.py file replaces the one provided in the Blinker test branch. The mqtt.py module is new. If you don’t want to install the .dat files, just change the message specification for the zone change, station name change and logged in events so they don’t use the “file” option.
The mqtt.py and notify_change.py files need to be made executable to enable the plugin. These files are attached, but due to forum restrictions I have had to name them mqtt.py.txt and notify_change.py.txt. You will need to rename them before using them.
When using it for the first time use the plugins menu item to go to the MQTT configuration screen. Enter the host address and port of your MQTT broker here as a minimum. The attached word document should be self explanatory, and replaces the version I provided earlier.
I am working on a Node Red application that will process these MQTT messages. In the interim period until it is finished, there are some options for testing described in the documentation. The simplest is to use Node Red to subscribe to the topics you define and send the message to email.
NB: Apparently I can only attach 4 files, I will attach the optional .dat files in a separate post.
munnecke
I am not familiar with that part of the system which is the basic web templating part. However it looks like it is trying to load the home page and getting confused when listing the stations. I assume from your earlier posts you have more than 1 board ie more than 8 stations so it could be part of the multiboard logic, (which I dont use).
Although you have that error you can access other web pages. So do a sudo service ospi stop, then deactivate all plugins (mobile_app, monthly_app, relay, weather_adj, weather_level_adj. To do this go to the plugins directory and do a sudo chmod -x mobile_app.py etc.
Then restart ospi with sudo python ospi.py from the /home/pi/OSPi directory and try to load the home page. If it was one of the plugins causing it (I doubt it – but it is good to remove as many variables as possible) the problem should have disappeared.
If you still have the internal server error, you can still load other pages, so load http://ospi ip address:8080/vo where ospi ip address is whatever IP address your OSPI system has.
You should see the options page. Click on the Stations handling option and check your system is configured properly, especially that the number of extension boards is correct. You could also set this to 0 as this should work.
If this checks out then try http:// ospi ip address:8080/vs and see if you get the right number of stations displayed.
Let me know how you get on. I suspect that there is somehow an inconsistency in these options introduced when you installed the different versions.
Regards
Ian
Request for input.
I have most of the functionality of the MQTT plugin working. This plugin allows you to configure the topic and content of MQTT messages sent in response to events (logged in, reboot, options change, controller change, station names change, programs change, zones change).
To configure the message content, you define a set of parameters as comma separated variables. The parameters can be a text string, a variable name from module gv, or a defined keyword such as Time, CPUtemp, CPUusage, RAMtotal, RAMused, RAMfree, Getlogin, and Uptime. The gv variables are as defined in gv.reference.txt available in the /home/pi/OSPi directory on your system.
It is possible to define additional keywords that allow you to specify some other values not available via the gv module. These could be values derived from other variables, or obtained from other plugins. I would like to hear from anyone who has any suggestions along these lines (it would be helpful if you could explain how to calculate or otherwise obtain such values).
I still have to work on the Node Red example to process these messages, although the example will build on the earlier examples I posted.
Regards
Ian
Thanks Dan
If the variable for the Blinker data payload is added to gv.py, then I would not have to do anything. The user could configure the MQTT message payload to include this information. It would probably be a good idea though to include this variable in the default configurations supplied.
I have posted on another topic a problem I encountered with the weather_level_adj plugin. This plugin encountered a serious error which it could not resolve, and printed a message (which in normal operation I would not see), then repeated this constantly. It seems that this could be another use for MQTT – to enable a plugin to get an error (or success?) message out to where it can be seen?
munnecke
As I have now quickly skimmed the e-book on Git, I may be able to help. I am guessing you follwed Dan’s instructions to get the Blinker_test version of OSPi.
If you go to the OSPi directory (or whichever directory you have as your working directory), and type git branch it should show you two branches – Blinker_test and master. Blinker_test should have an asterisk next to it. If this is so, then if you now type git checkout master you should be back to your original configuration.
Before you do it, (and if you are running the Blinker_test branch) could you go to the OSPi directory and type
sudo service ospi stop
sudo python ospi.py
Then you should see the console messages. Fire up your web browser and try to get the Internal Server Error message again. You will probably see a python error message on the screen. That would be helpful information to see why your system got this error (I am running the Blinker_test branch currently with no issues!)
Ian
I have a version that is close to being ready for wider testing. This consists of two plugins – mqtt.py and notify_changes.py. mqtt.py is the data entry part where you define the events you are interested in, the topics, and the content of the messages. notify_changes.py does the actual work, and is an updated version of Dan-in-CA’s version that fills out the “stubs” he provided.
All topics and message content is user definable, either from the data entry screen, or by defining the message content in a file.
I have attached a Word document which documents what I have done, and provides a screen shot of the data entry form. Please feel free to make any suggestions (or complaints).
An update.
I have decided to allow the user to input the string that will be sent with each message, so it can be customised. I will set some defaults. That effectively resolves one of the most difficult issues I faced. So the input screen will change.
With this change each user can decide which events will trigger an MQTT message, what the topic will be, and what the payload of the message will be. I may as well allow them to set the QOS and Retain values of the MQTT messages as the optimum values for these may depend on how the messages are to be used.
Ian
I see that I can’t upload files with extensions of .py and .html. So I have renamed the files to be mqtt.py.txt, mqtt.html.txt, and notify_change.py.txt. Hopefully that will work. You will just have to rename them when you download them.
I have taken Dan’s version of OSPi with the notify_change.py plugin and modified it (Sorry – I am not familiar with Git, I tried, but eventually just downloaded a zip file and extracted the files. When I have some more time I will read more about Git!).
I have a working version (call it early alpha!) that uses the notifications Dan introduced to send MQTT messages whenever these events occur. There is a web page that allows you to select which notifications you want published to MQTT, and what the topic will be for each. You can also set the MQTT host and port. There are defaults for all these parameters.
I haven’t developed any Node-Red flows, but I have used an Android app called myMQTT to check the MQTT messages are being sent. They are set with the retain feature of MQTT, so that if you subscribe after the event, you will get the latest message in the topic. They appear almost instantly on my phone.
Please excuse the code (whats wrong with Fortran anyway!) and feel free to make any suggestions. I am very much a Python novice, and struggled with some aspects – getting numerous errors. You will note some routines duplicated in the modules mqtt.py and notify_change.py – it was easiest that way and I am sure someone can let me know how to avoid it! The module mqtt.py is mostly about getting the parameters input by the user, whereas notify_change.py uses the stubs Dan created to process events.
This needs a lot of thought as to what data is to be sent for each event. I am quite sure I have got it wrong in many cases, and will be spending more time looking at that aspect. The message that is sent can be a string (“Someone logged in”, or a python object eg {“snames”:[“s1”.”s2″,”s3″,”s4″,”s5″,”s6″,”s7″,”s8″]}. The topic should indicate which event the message is in response to.
mqtt.py and notify_change.py must go in the plugins directory (and be set executable). mqtt.html goes in the templates directory.
Node-red can be used to process these messages, if necessary reformatting them and sending them on to other applications such as email, storing in a database, and so on. The example I posted earlier provides examples that would be a good starting point.
Thanks Dan.
At the moment I am just getting a plugin working that has a web page to allow the user to say which of the notifications they want to receive, and what the MQTT topic should be for each notification. I am also extending your stubs to send a plain text message. When that works I will then look at the payload will be for each type of notification. So it is good we discuss it now.
Sending a python dictionary object containing the elements covered would make it easy to use that and send those elements in the payload. Maintaining the dictionary could be a problem, unless it could be automatically generated. I wonder if that is possible?
I wasn’t too concerned about the traffic as these events are relatively rare. eg Once OSPI is configured, the main events would be stations turning on and off, and these are a few per week, or in the worst case per day. If the python gv object was sent, anyone using it could simply replace all values. This is easy to write and maintain at both ends. However when I get to that stage I will do some tests.
Of course this will only improve performance if the apps making HTTP calls are modified to take advantage of it.
Thanks Dan. I agree this would be good to have as standard functionality. I will update my plugin to reflect this new capability. My plugin will now simply be a configuration plugin that enables different messages to MQTT and allowing the user to specify the topics for these messages. My understanding is that notify_change.py would be modified to send the MQTT message (instead of printing on the console or perhaps as well as? What do you think?). I think that it is possible to send the whole of gv in the message body together with a time stamp. I will also develop some Node-red flows that initialise global variables corresponding to items in gv, and update them when they change.
The topics I will have as default will be something like OSPI/login, OSPI/zones-changed, OSPI/controller-changed, OSPI/options-changed, OSPI/reboot, OSPI/station-names-changed, OSPI/program-changed, OSPI/programs-deleted, OSPI/programs-toggled. Users can changed them as they wish. The Node-red flows will need to incorporate some mechanism to notify when the global variables change, and this would then trigger any user defined flows to process the change.
Once we have tested all this we need to provide documentation for plugin developers so they can incorporate similar functionality. eg my plugin would need to have a mechanism for notifying changes to the Topic names, and provide global variables in Node-red that correspond to these variables. Other plugins would need to do the same.
From a performance perspective, changes would be very infrequent (maybe daily, or bi-weekly) – a very significant improvement over the polling approach. I am confident that there will be some easy way to get the updated gv values from Node-red to Android and IOS apps such as the mobile OSPi app.
munnecke: your wish to have a record of changes to your system could then simply be achieved by archiving the updates to gv to file every time one of these messages arrived. You would need to develop your own application to play back/ recover these changes from the archive.
Dan – I think MQTT is getting more and more acceptance in this whole home automation world. So by placing the core OSPi variables in MQTT, it open up OSPi. Some users may not want to use Node-red, and that is OK as they can get the data directly from the MQTT broker. If an alternative emerges in the future to MQTT it would be trivial to allow it as an option in my plugin.
I think that you would be better off using MQTT to send data from your arduino rather than Modbus. (I have 35 years experience in the SCADA field – this is much easier!)
Ian
Hi Dan
I am thinking it would be a good idea to have the MQTT module become a general change of state publisher via MQTT. If the blinker python module works and Dan is prepared to build it into the core OSPi code, then my MQTT plugin could subscribe to these events and send MQTT messages. I will develop a Node-red application to log the data.
One reason for thinking this way is that I noticed the Android Opensprinkler app polls the OSPi three times every three seconds for change of state via http://host/jc?pw=xxx, http://host/jo?pw=xxx and http://host/js?pw=xxx calls. It obviously needs to keep its knowledge of the OSPI status up to date. I had noticed that my OSPI seemed to be struggling at times when I was polling it in a similar way, and it became unresponsive at times. Clearly it is not possible to run more similar applications on the Android and IOS world, as I found out. It would be best if these type of apps did not have to poll OSPi for events.
So your investigation of the Blinker module is timely! Let me know how you get on, and when it is likely I could have some source code to test.
munnecke
Let me describe my setup. I started with Mythtv and have an old desktop running this. It has a large hard drive and records any programs we are likely to watch (and autoexpires them when the disk is nearly full. I have 2 TVs and each have a Pi running XBMC. One is networked via wifi, and the other by ethernet over powerline using D-Link powerline AV 500. The one using wifi is on a different circuit so I can’t use the D-links. Hence the wifi, but it works well. I also have XBMC on my Nexus 7 tablet and my wifes iPad. I use the Yatse application on the Nexus 7, and Nexus 4 phone to control XBMC. This is not a new house- we bought it in February and I have used wifi and powerline to avoid retro fitting Cat6e cable all over the house).
I have installed Emoncms on another Pi. This one has a small hard drive (even the small ones these days are massive overkill). I did this because the SD cards have a limited life (ie they have a limited number of write cycles and the Emoncms data logging application is expected to run into issues on SD cards. I use an Emontx at the main meter (fitted with 4 CTs) to monitor power consumption. This is essentially an arduino and it transmits by 433MHz radio using the RFM12b radio module. This system is partly to investigate whether a solar power system is worthwhile. I have also installed two EmonTH’s (arduinos)- monitoring temperature. These also transmit via 433MHz radio (which apparently has a range of several hundred metres). One thing that is non standard in my setup is that Node-red intercepts the incoming radio signals (via the serial port) and forwards the data to Emoncms. This is because Emoncms seems to have little or no query capability, although there is an export CSV function. The Emontx and EmonTH are well engineered and work flawlessly.
So as summer is approaching I needed to get my garden reticulation working. The old controller that came with the hosue needed replacing. Given the above infrastructure it OSPi seemed a good fit. The OSPi is connected to the network via the WiPi module. Node-red and MQTT have allowed me to integrate all these systems and potentially customise them without modifying their core code.
My immediate plan is to introduce some sort of logging capability for OSPi. With your and munnecke’s input the scope has grown! The next plan is to measure the water consumption using the small water meters I provided a link to. These provide a pulse output. I have a number of options to integrate these – I could connect them to the EmonTX, or to OSPi, or use a dedicated Arduino. The distance between the meters and the Emontx and OSPi is only a meter or so.
Ian
Hi Dan, munnecke
I have hacked the email_adj plugin (as Dan suggested), and have it sending MQTT messages as follows
– to topic OSPI/system when the system starts (presumably when the ospi service is restarted
– to topic OSPI/start when a station turns on
– to topic OSPI/run when a station turns off
– to topic OSPI/rain when the rain sensor detects rain
There is still some more work to tidy it up, but essentially the plugin has a loop running every second. It detects changes in state by looking at variables in gv and comparing them against the previous values. Almost any variable can be monitored providing it is accessible within my plugin (called surprisingly MQTT!). Doing the detection of events from within the one module by monitoring the content of the variables is going to be far easier than trying to modify the routines that make the changes to the variables. To do that you would need to scan code in all modules, and also rely on authors of as yet unwritten modules to add the necessary hooks in their plugins – very tricky.
There is a web page accessible from the plugins menu item that allows you to set parameters such as the hostname, and the port of the MQTT broker. There are check boxes to select which of these messages you want. I will probably allow the user to input the topic for each of these events, but that is not in place yet.
It would be easy to check for changes in any of the variables in gv. So detecting changes to station names, programs, watering percentage etc are all possible.
This method has much less load on the OSPi. However if the OSPi goes down (or the service stops and doesn’t restart), then you would get no messages. I will look at a mechanism for detecting this, as I am concerned that my sprinklers may not go on if the OSPi doesn’t recover properly say from a power failure.
I am interested in hearing any comments as to ideas for improvements/ things I have done wrong, suggestions for alternatives or anything relevant, etc etc.
One warning for people regarding the MQTT broker – don’t install it from the Raspbian repository -it will just hang when called from a Raspberry Pi. Go to the link I ooriginally provided in my initial post and install it from there.
Regards
Ian
I think that the MQTT Retain feature will automatically provide Node-red with the latest state, making life much simpler.
munnecke – I have put station run times into Emoncms (running on another Pi). I will also be collecting water metering data and placing this in the Emoncms database. From there it is simple to use the facilities of Emoncms to calculate running totals, convert units, apply calculations (including combining data from other feeds) etc. It is however only suitable for analog data (including pulse counter data) ie it is not suitable for status information eg enabled/disabled, and so on.
Once I have got the plugin to a more advanced stage I will make it available for testing and feedback.
In node-red I will take this data and direct it to different systems, such as Emoncms, XBMC, email etc. I will also develop flows that will effectively give node-red a set of globals that mirror the OSPI variables. That should allow people to come up with some inventive uses of the data.
Hi pierantonio
To get some more information about what went wrong, open a terminal session on the Opensprinkler Pi.
Then cd to the /home/pi/OSPi directory
Type sudo service ospi stop
sudo python ospi.py
This will allow you to see the console log messages. You will probably find some typo’s in function names or syntax errors.
Regards
Ian
