[tom]

+1 on the update process saving and restoring the configuration.

[tom]

For what it’s worth, I’ve been having huge problems with my router and port forwarding… I ended up buying an Amped AC1750 high power router, which works well for the LAN. But when I tried to open port forwarding to access my OS controller while on vacation, it was impossible. Also, it is impossible to shut off internet access to the router’s controller menu. Tech support just kept cycling me through the same diagnostics each time I called, then said to “send it in for service” – disconnecting my wifi network while they fixed it. I ended up buying a new one, but that had the same firmware problem. I won’t be buying any more Amped Wireless products.

[tom]

I installed an Arduino version for half my yard, and just installed the PI version in the other half of my yard. I also installed a PI version in my wife’s raised bed vegetable gardens (one zone per bed). My wife (more interested in gardening than geek stuff) really likes it. I also tested Dan’s Python version (Now called SID).

My reactions to all three:

If all I wanted was to plug in a controller and use it without modifications, I’d use the Arduino version. It is much more self-contained, doesn’t have to boot an operating system, boots faster, and is probably much more reliable than the PI, not being dependent on an SD card. But modifying the controller is much more difficult, if not impossible for certain tasks. There is no SSH shell available, you have to upload an entire new program to change anything.

If I wanted a reliable (and compatible) controller that I could run as a “production” controller but still run other programs, I would go with the PI or Beaglebone version of the unified firmware. This takes more setup time, but provides far greater expandability and customization. But it also requires more geek work to set up. I’m currently in a geeky mood, so this fits my current interests very well. I love being able to SSH into the controller and look at things, add a CPU temperature channel on ThingSpeak (see https://thingspeak.com/channels/41518 ) and monitor the status of my system with a simple Unix command (“watch curl http://osbedroom.local:8080/js” shows which zones, if any, are activated – replace osbedroom.local with your local IP address, turn off passwords on the site )

If I wanted to explore even more leading-edge PI technology, I would go back to Dan’s Python program. I really liked the openness and the overall general architecture and plug-in approach. But it wasn’t compatible with my previous Arduino system, and I wanted to have a reliable production controller that I could use as a foundation for my own techie adventures. It sure would be nice to have the best of both worlds… a solid core system that will run across Arduino and PI platforms compatibly, but also a plug-in (and REST API interface) capability to mod it as well.

All in all, I am delighted with the technology. It’s running well (fingers crossed), and I’m having fun analyzing flows and logs.

[tom]

thanks… I look forward to ET based adjustments.  Will you allow zone-by-zone adjustment (soil types, slope, vegetation, etc.)?  How will you incorporate rainfall?

[tom]

Rhillman:  I like where you are going with your flow logging.  Have you considered how to report flow that’s not associated with a specific valve? (i.e. a slow leak or gusher, or other sources drawing water through the meter?)

Have you considered publishing your flow rates via MQTT?  http://mqtt.org/  this is a very simple and lightweight publish/subscribe framework and would allow integration with other devices.  For example, you could publish your rows of data to topic “Rhillman/home/sprinklers” and then have your phone (or another OS controller) subscribe to the same topic to trigger an alert.

It would also be nice to track the actual cost of the water for logging.  This can get complicated with tiered water rates, but I’m happy to count all water usage at the highest rate for the billing cycle, and not worry about the cheaper water at the beginning of the cycle.

[tom]

Ray: yes, this is the sensor family I am using http://www.amazon.com/Dwyer%C2%AE-Multi-Jet-Pulsed-Output–C-03-1/dp/B00D2ME5MC  I am a little ticked at them that after talking with their tech support, they ended up recommending a meter with a 10 gal pulse rate.  This is way too slow of a rate (some times one pulse every 90 sec) but they won’t exchange or restock it for a 1 gal rate.  So 1 gal (or even .1 gal) is the way to go.  It’s hard to figure out when the flow has stopped with the longer pulses.  I’ve attached a DSO trace of the pulse wave form over 200 ft of cat 5 cable.  I seem to get some spurious interrupts that I don’t understand or can trace on my Bitscope.

My configuration is that my irrigation master line branches off from my house water line just after the main meter.  I am installing a Dywer flow meter on the irrigation line so that can measure all of my backyard water usage (including an auto pool fill valve, vegetable gardens, hose bibs, etc.  I am using two OS controllers for my main sprinklers.   It would prefer to use just one controller and slave the other circuits to an extension box, but that would be a 300ft run.  In the past, I’ve been sprinkling odd days on one controller, even on another.  But with the California water shortage, I’m supposed to water only on certain days.  So, now I have to figure out how to synchronize the two controllers.

So, I would like to see my flow data on both controllers, and also to see what is happening when the circuits are off (very important for leak detection.)

 

Attachments:

1. 

   good-trace-of-street-meter.png
   

[tom]

Hey Dan,  I appreciate all the effort you have put into this…  What’s the chance of some kind of happy reunion of technologies, so we can use the unified platform for our day-to-day use, but also a python-based framework for adding functionality for smarter additions?

[tom]

I discovered the same problem when I installed 2.1.3 ….. it took a while to find that the log option hadn’t been restored in the import.

[tom]

I have been playing around with this in Python with these meters…  http://www.dwyer-inst.com/Product/Flow/WaterMeters/SeriesWMT2  the meters seem very sound, the cap is a bit flimsy and the sensor wires probably need to be protected better.  They are delivering a very good square wave over 200 feet of cat-5 cable.  (I’ll dig out an oscilloscope trace if folks want to see it)  The prices seem to be pretty good, too. A 3/4″ meter for $87.50.

I would love to see the log contain the actual number of gallons delivered per zone activation.  The average flow rate is very interesting to see for tracking gushers, drip system problems, pressure problems, etc.  deviations in flow rate at a certain level should trigger some action… perhaps shutting down the master valve.

I also want to see the flow meter’s data even when the OS circuits are not running… the automatic pool filler (which I may want to switch on or off with its own sprinkler circuit), or other things.

I have a 1 1/2″ 100 GPM flow meter that pulses per 10 gallons… too much for my application.  I also have a 3/4″ model that pulses per .1 gal.  This gives a much nicer flow, and allows detection of the end of the session much better.

I don’t need the big meter, and would be happy to sell it at a discount, if anyone needs to work with the higher flow rates.  (the meter isn’t rated for pool water, I found out after purchasing )

 

[tom]

Here is some more info http://www.mercurynews.com/drought/ci_27729990/california-drought-state-passes-mandatory-new-water-conservation?source=infinite  it is a ban “from watering lawns and landscaping with potable water within 48 hours after measurable rainfall.”   Given the rarity of rainfall in the drought, I’m not sure how often this will be necessary.

(as an aside, California is next to the largest body of water in the world.  I’ll be drinking fresh desalinated water from it by the end of the year: http://carlsbaddesal.com/ )

[tom]

great.  I’m looking forward to it.  Do we have visibility into the next release?  a beta program?

p.s. it would be great if the zones would have their numbers and names displayed throughout (as in the /#os-stations display).  I like to use zone names while walking about and testing the system (“lawn near BBQ” makes it easier to find it), but also use the zone numbers for checking out the wiring, valves, etc.

[tom]

I understand, but it seems a shame to be limited by hardware constraints in this day and age 🙂

I’m torn between using the Arduino and the Pi versions.  The Arduino version seems to be more reliable and has a better scheduling system.  The Pi system is more open and available for programming, but doesn’t have the zone-level scheduling that I need.  I’m pulling down Evapotranspiration data from California’ CIMIS databank, and reading pulses from a Dywer flow meter, but am confused as to how best pull it all together, or what other efforts are under way… I’d love any advice folks might have.

 

[tom]

<div>
<div>Thanks, Samer… I appreciate all your work on the project.  You do great work.

</div>
I got things working OK by adding the additional parameters, but it sure is complicated .  It seems kind of dangerous that a badly formed GET command would shut down all access to controller.</div>
<div>
The way that it is set up,  a program that just wants to change the water level has to also manage the NTP, DHCP, ignore password, rain sensor, and type of rain sensor states.  Kind of goes against the whole spirit of a REST interface.

</div>
It seems to me that a better way would to make the state of each parameter independent of the others.  If a parameter is not mentioned in the GET command, it is not changed.  If it is set =0, it is off, and =1 it is set to on.  That way there is no confusion about unintended side effects of a state change, and a program does not have to be concerned with managing all the other states in the command.

Does this sound reasonable?

[tom]

Thanks for the info, Ray.  I wish I could have the best of both worlds – Python/plugin architecture as well as compatibility.  I really like what Dan has done, and thank him for it, it’s just that the flexibility isn’t there for the zones.

My impression is that you really had to cram things in to get it to work on the Arduino.  With the Linux-based versions, how can we make use of the expanded capabilities?  If not plugins, would there be a way to run parallel programs that could interact with the main sprinkler software?

Also, I’m wondering about the reliability of the PI version vs. the Arduino version.  I’ll be mounting things in boxes out doors, with full sun.  I’m concerned about relying on that tiny SD card to boot, as well as the temperature of the PI in the summer sun.  The Arduino boots so quickly and reliably.  any thoughts?

[tom]

The challenge isn’t just getting the sensor to work, but rather if the sensor is actually measuring anything useful.  Soil electrical conductivity varies with the salts in the soil more than the moisture.  Fertilizers, salt buildup, etc. all have a huge effect on the reading.  And the salinity also affects the ability of the plants’ roots to extract water.  So, a higher conductivity might mean that the plants are less able to use the water in the soil… exactly the opposite of what we are trying to measure.

And, assuming we had a valid sensor, it would be hard to extrapolate from that sensor to the soil around it.  Imagine a tree with a line of drip sprinklers spaced at 18″ intervals.  If the sensor happened to be directly under a drip outlet, it would show a drastically different reading than one 9 inches away.  Check out http://www.soilmoisture.com/ for more information.

An alternative might be to measure micro climate factors, such as shade and soil temperature.   Lawns might be shaded in the winter in yet full sun in summer, or trees might be shading the lawn in one season and not another, etc.  A simple soil temp probe or maybe solar radiation could be used to modify the zone’s ETo numbers.  Then the problem becomes how to change the sprinkler zone’s behavior as its micro climate changes.

[tom]

While we are looking at this, it would be nice to have the ability to do finer adjustments based on additional sensors.  For example, I notice that a section of lawn gets a lot of shade in the winter, but not in the summer.  I’m toying with the idea of setting up a security camera system that would be able to see the amount of sunlight on the lawn, and adjust the time on that zone by the amount of sunlight falling on the lawn.  Or looking at the color of the lawn to detect whether it needs more water or not.

Lots of possibilities for fine-tuning the zone; it would be nice to have the tools to plug in different ideas and see how they work.

And I sure would like to carry over the Arduino version’s “Program” logic to the OSPI.  I was very happy with the Arduino Program capability, and was hoping to move over to the OSPI version.  But I’m not sure I will be able to use the OSPI version for “production” use with the current OSPI program logic.  Maybe I don’t understand it, but it’s just too complicated to set up 22 zones and 22 start times individually on a grid of 7 days.

It would be nice to have compatibility between the OSPI and Arduino programs so that we could export configurations from one and import to another.  We could save these configurations so that we could look back and see what was used in the past, and what monthly adjustments were applied with what results.  So, the log data becomes important, as well.

If we split into a new approach for monthly adjust for each version, then we have to reinvent all the neat things we can do with smarter irrigation monitoring for each version.

 

[tom]

Hey Ian… I haven’t forgotten you… I just haven’t been able to get my system to work since I tried to do the upgrade.  I have done the Git Pull and the Blinker upgrade – and all is up to date… but whenever I try to run the software I get an “invalid server error”

here’s my error listing:

munnecke@ospilab /home/pi/OSPi $ sudo python ospi.py

plugins loaded:
mobile_app
monthly_adj
relay
weather_adj
weather_level_adj
Starting timing loop

http://0.0.0.0:8080/
Monthly Adjust: Setting water level to 100%
Traceback (most recent call last):
File “/home/pi/OSPi/web/application.py”, line 239, in process
return self.handle()
File “/home/pi/OSPi/web/application.py”, line 230, in handle
return self._delegate(fn, self.fvars, args)
File “/home/pi/OSPi/web/application.py”, line 420, in _delegate
return handle_class(cls)
File “/home/pi/OSPi/web/application.py”, line 396, in handle_class
return tocall(*args)
File “/home/pi/OSPi/webpages.py”, line 94, in GET
return template_render.home()
File “/home/pi/OSPi/web/template.py”, line 1020, in template
return self._base(t(*a, **kw))
File “/home/pi/OSPi/web/template.py”, line 881, in __call__
return BaseTemplate.__call__(self, *a, **kw)
File “/home/pi/OSPi/web/template.py”, line 808, in __call__
return self.t(*a, **kw)
File “templates/home.html”, line 311, in __template__
<td class=”scheduleTick” data=”2″></td>
IndexError: list index out of range

192.168.1.5:59442 – – [15/Nov/2014 19:50:23] “HTTP/1.1 GET /” – 500 Internal

[tom]

I tried to install the latest update, per the instructions for “upgrade.”  The GIT logic all worked OK, but it just gives me an “Internal Server Error” when I try to access it via HTTP.

I’m confused as to what version folks are working with, and how to get back to an operating version.

[tom]

Thanks, Ian…  I’ve been busy with other issues recently, and haven’t had time to keep up with all the stuff happening here. re: Fortran: yes, those were the good old days, punching cards, dropping decks, waiting for the computer operator to run your job, waiting 2 hours to find out that you missed a comma somewhere and blew the whole job.   But I also learned how to make cool paper airplanes by stapling together punched cards.  Just a quick calculation: my Raspberry Pi has 43 million times the “disk” storage and 80,000x the main memory, of the IBM 360/50 I learned on.

Dan: I did try to convert my controller over to OSPI from my working Arduino-based version, but couldn’t figure out how to program the individual zones to specific times… they were all fixed to the same time (15 min).  Am I missing something here?

P.S. my water district is now talking about doubling the water rates for the highest tier usage, which will really put a focus on more efficient watering…

[tom]

wow… things are happening fast… Great work, Ian and Dan.  Hopefully, I’ll get to this tomorrow to test out.  In the spirit of “eating my own dog food,” I plan to install two OSPI controllers to control my real-world sprinklers, so I expect to see the reality of what we are coding here.  I played with this:

I’m wondering whether there might be some interesting things in the zeroconfig mDNS stuff… (I really like the “zero config” name). It adds the ability to discover services dynamically (“domains” such as printers, servers, music libraries, etc.).  Maybe we should add a domain “sprinkler” that then has topics for all the interesting things sprinklers do.  The MQTT server(s) could become a domain(s) within the rDNS service, which would name the “last will and testament” fail-over server if the primary one failed.

here are some links:

http://www.ryukent.com/2013/09/a-local-url-instead-of-an-ip-address-for-your-raspberry-pi/

https://code.google.com/p/pybonjour/ – python-based framework (I got it to work a bit, even though it is very old code.)

 

 

[tom]

Have you considered a general state-change detector that looks at the whole Export JSON?  That could log everything that changes in the controller, kind of like the Recent Changes log in a wiki page.  At each interval it would compare the current JSON to the one from the past, and then note the differences.  This wouldn’t have to run so frequently, but it would be more general.  It would also allow a user to roll back to a previous setting, say to settings of the same month from the previous year.  I guess a simple version of this would be a simple cron job that would Export  to a file each night.  Get the data recorded, and let the future figure out how to detect the changes, if they want to 🙂  In any case, this could be valuable information for going back to try to figure out what was happening/when.

re: “If the OSPi goes down (or the service stops and doesn’t restart), then you would get no messages. I will look at a mechanism for detecting this, as I am concerned that my sprinklers may not go on if the OSPi doesn’t recover properly say from a power failure.”  Yes, this is a critical problem.   For starters, maybe each controller could publish a “heartbeat” message, saying “I’m alive” with various parameters, e.g. Time since reboot, CPU temperature, voltage, battery status, etc.  Maybe this could be every hour.  The first time after reboot it could publish another “wake up” message.

p.s. I’m planning to run at least two OSPI controllers, and may adapt one to control my pool, as well.  Maybe the publishing topics should be based on the assigned name of the controller, not just “OSPI”

 

I’m playing with some ideas about using “Promise/Futures” logic for this.  Basically, a controller has a set of promises that it has made and is working to fulfill. (“Deliver program

of watering by dawn,” “publish a <Heartbeat> message every hour,” “export the all of the controller’s variables at midnight”), etc. When it boots up, it publish a

For a more general solution, I’ve been looking at Promises/A http://wiki.commonjs.org/wiki/Promises/A  and Twisted Twisted: http://twistedmatrix.com/documents/current/core/howto/defer-intro.html.  A controller would publish (or commit to) a promise, which the network would then see as a future event.  (e.g. running a program by dawn).  The controller would then flag the promise as completed when done, or the network could detect that the promise was never completed (or the controller could admit failure).  I think, (but am not sure 🙂 that this could simplify the scheduling process when it comes to rain detection, exception days due to drought restrictions, microclimate sensors tweaking a zone, etc.  It also builds a foundation for doing alerts when things don’t happen as expected.  We can’t expect the failed task to announce that it didn’t work, so there has to be some external promise/completion process to drive this.

I’ve got to get my sprinklers installed and running.  I haven’t been able to get the SSH key pair generation to work right… I’ll probably set up two OSPI controllers, an XMBC (or two), and and MQTT/Emoncms machine, too.

Ian: what metering hardware/communication system are you using for your Emoncms setup?

 

[tom]

<div>
<div>
<div>Thanks, Dan.</div>
I’m looking at interfacing this water meter http://www.dwyer-inst.com/Product/Flow/WaterMeters/SeriesWMT2 which would require a process or microcontroller to count the pulses.  It could update a global variable in gv.py as well as publish it via MQTT… Don’t know anything about modbus, I have about a 150 foot run from the meter to the controller.  I was thinking about a solar powered zigbee transmission link.  I was also wondering if it might be possible to transmit the pulse information over one of the sprinkler zone wires.  The wires would still have to work as the 24v actuator for the zone’s valve, but would also carry the meter pulse info back to the controller, saving me from having to run a long cable.  Maybe wireless is the way to go.  I’m happy to not reinvent the wheel, if there is something out there already.

</div>
Having status APIs is nice, but I’m particularly interested in state <i>changes</i>.  I want know when someone has changed a program – say, a new daily schedule schedule. Or adjusted the zone time.  It would be good to see time/week per zone, which be a good overall metric.  And I want to go back and see what I was doing last year at this time… minutes/week (or gal/week) would probably be the best metric for this.</div>
<div></div>
<div>re: soil moisture sensing.  This is a very complicated thing to measure.  Simply measuring resistance is more a function of salt content than moisture.  It is also dependent on the type of soil, plant, temperature, etc.  Check out http://www.allianceforwaterefficiency.org/soil_moisture_sensor-intro.aspx</div&gt;
<div></div>
<div>I happened to have attended the University of California, Riverside, that you mentioned.  And as a further coincidence, I got my first job at the USDA Salinity laboratory, and my first taste of computer programming, doing soil moisture determination with the use of soil psychrometers.  (see fig 13).  This was a ceramic bulb (that simulated the root’s access to the soil) within which a thermocouple was placed.  I’m a little fuzzy as to how it worked (this was 1968 🙂  There might be some interesting opportunity to do some microcontroller control of the thermocouple, and 3d printing for the psychrometer body, as well.</div>
<div></div>
<div>gotta run now, but I’ll get back on the ideas for the MQTT publishing points.</div>

[tom]

So many intertwined topics… let me respond to some of them raised on this thread:

Dan: re: “My original goal when I started working with Python and OpenSprinkler was to develop a platform for experimenting, and something that can be easily customized.”  Wonderful idea, and thank you for acting on it.  Hopefully, I can build on your platform – and use my own home as a experimental site. I have a huge water bill and am in the middle of California’s drought (San Diego), so this is quite timely for me.

Ian: re: “I haven’t had much of a look at gv.py yet. I assume your modifications will create data items that can be input via the Opensprinkler web interface?” And thank you Ian for pointing out these neat new technologies.  I’ll let Dan comment on this, but my understanding is that gv.py is the home for establishing the global variables used by the other OSPI.py software and plugins.

I want the capability to track ALL state changes in my sprinkler system(s).  I want to know when a program changes, when the rain sensor comes on and goes off, when someone changes the minutes per station within a program, how long a valve has been actuated, how much water flowed through my irrigation meter during the actuation, if there were any unexpectedly fast flows (a gusher), or low flows (bad valve).  I would like to add micro-climate sensors to adjust my zones (the north side of my home is much cooler than the south side, for example, and varies with the season, leaves on or off the trees, etc.)  Obviously, I don’t intend to do all of this at the onset, but my design style is to design for the general and then narrow down to the specific.  I would like to design for the general framework, and then populate it with one instance.  It’s a bit like creating a wiki (which is really a very simple technology) and letting Wikipedia emerge rather than collecting standards committees and experts to write the “correct” encyclopedia.  So, what are the simple initial conditions we can create to allow other innovation to emerge?

It seems to me that we need to trigger the messages at the time of the state change in the main program.  I was thinking of the mpay global variable as a string that would control which messages to be published.  Maybe others have a better idea, but my first thought was to assign character codes to each type of messaging: “s” would represent sprinkler state, “p” would be program state, “r” would be rain state, for example.  mpay=”srp” would cause OSPI to publish all events, mpay=”s” would publish only sprinkler events, etc.

 

 

[tom]

Thanks, Ian.  I got the single node fnBuildurl.. I noticed that you used two different topics – one with plural “api-responses” on lines 48 and 38. I get an “Unexpected Token” error when I try to deploy the node…. not very helpful; it just gives the message and no other info.  Maybe it was garbled in the email.
I don’t understand the full data flow of how this all works…
It would seem simpler to me to put the publisher logic in the main OSPI routine, synchronized with the state change of the sprinkler.  Polling seems to be a lot more complicated, and subject to some missed events if the poll happens to happen during a state change in OSPI.  I haven’t done real time systems like this before, but it seems to me that the whole value of having a event/message architecture is to get around all the problems of polling.  Plus, polling doesn’t seem to be scalable; as we add more detail and events, it would seem that the polling process would get overwhelmed.
How do you display the messages in XBMC?
Maybe this is worth moving off to a Github repo?

This is a lot of fun.

[tom]

Hey Ian…

Thanks for pointing me to Node Red. It’s been a lot of fun exploring it, as well as thinking of how it might be extended.  I’m poking around OSPI now to figure out how to publish MQTT publishing from within OSPI.  I have it working from within Python now, and played around with a fork of Dan’s Github gv.py, adding:

    [“MQTT Host”, “string”, “mhost”,”Address of MQTT server for reporting events.”, “System”]
    [“MQTT Topic”, “string”, “mtopic” , “Topic to which this OSPI server will post”, “System”]
    [“MQTT Payload”,”string”, “mpay”, “What logging to perform”,”System”]
    [“MQTT pub?”, “boolean”, “mpub”,”Whether to publish events via MQTT”,”System”]
re: your comments about needing to develop a structure for topics… I agree 100%.  This is a very deep subject, and a problem that is rarely handled well (I spent 40 years in medical informatics, and this problem is everywhere).  One approach that might be applicable is Topic Maps http://en.wikipedia.org/wiki/Topic_Maps

This is also related to Semantic Web http://en.wikipedia.org/wiki/Semantic_Web and Linked Data  http://en.wikipedia.org/wiki/Linked_data

I think that this is one of the most powerful things the open source community can do: converging on an open, extensible definition of MQTT topics rather than Apple pushing “iTopics” or Google pushing “Gtopics” and having corresponding topic-wars.

I am also interested in “Promise” software technology: http://en.wikipedia.org/wiki/Futures_and_promises  This is kind of related to an MQTT’s “last will and testament” of a server, to handle fail-over.  (each OSPI could be a failover server for MQTT).  An OSPI client, for example, could promise to send a “heartbeat” message every 10 minutes, which would trigger activity only when the promise was broken.  Sprinkling programs could be communicated as “promises” which would run silently unless the promise was broken.

I’m just starting with OSPI and Github stuff.  I used to be able to leap tall buildings with a single line of code, but that was many years ago,  I’m afraid I’m a bit creaky.  I hope to get up to speed with Python, Node.js, Node JS, and other tools real soon now 🙂

 

[Author]

Posts