Python interval program update 8/01/13

[Dan in CA]

Hi all,
A new update to the Python interval program for OSPi is now available. It is now compatible with Firmware version 1.8.3 and includes the concurrent operation option.

This update includes changes to several key files. The recommendation is to do a complete install. Future updates should not be this drastic.

If you have been using the logging feature of the Python interval program and want to archive your data, now would be the time to do it. See “About Log Data” below for instructions.

If you have an existing instllation of the Python interval program, the easiest thing to do is probably just rename your OSPi directory to somethning like OSPi-182. You can then revert back to it if you want.

Then install the latest version using the new shorter URL.

1. From the Pi directory, use the following command to download the program:
wget http://rayshobby.net/sourcecode/ospi.tar.gz

2. Once the program has finished downloading, use this command to unpack it:
tar -xvf ospi.tar.gz

It should now be ready to run. These instructions work for first time installations also and only take a couple of minutes.

You can transfer your irrigation programs and station names to the new installation by copying the following files from the old OSPi/data directory (e.g. OSPi-182/data) to the newly installed OSPi/data directory:

programs.json
snames.txt

Do not copy the old sd.json file to the new directory. It Has been modified and the old version will cause errors.

About Log Data
The Python interval program’s logging feature stores data in a spread sheet friendly comma seperated values (csv) text file.

The data file is set up to be easily downloaded from the Raspberry Pi to another device, e.g. laptop,
You can download the log file by pointing your web browser to:

[URL of your pi]/static/log/water_log.csv
for example on my system the url is:
http://192.168.1.22/static/log/water_log.csv

Your browser should offer to save the file to your local device.

Once the file has been downloaded it can be imported in to your favorite spread sheet program for viewing, sorting, etc.

Dan

[craigmw]

Dan, thanks for the update and the detailed description about how to install it. This really helped!

[djagerif]

I like the fact that if a station is running, and you go to Manual Mode, it keeps running for the remainder of the program time.

Ingo

[Dan in CA]

@ djagerif
I guess you could call that an accidental feature.
I think its a good one. Thanks for pointing it out.

@craigmw
Now that Ray is hosting the download for the program on rayshobby.net, installation is much more straight forward. Thanks Ray.

I am currently putting the final touches on a wiki page for the Python interval program. It is intended to be a full set of documentation for the program.
It should be up in the next day or two. I will post an announcement here on the forum when it is ready.

Dan

[craigmw]

Dan:

A couple of odd things I noticed. As I described to Samer in another post, I had some weird corruption of the snames.txt file which was showing 24 stations in the web app (but not in the interval program). It seems that the webapp enumerates the stations based on the snames.txt listing, but the interval program does this using the number of extension boards field (?). In any event, I’m not sure what caused the snames.txt to become corrupted and list 24 stations when I only have a single extension board (and thus 16 stations).

A second issue is that the port field in the options page does not allow me to change from 80 to 8088. Despite this, it seems to be working fine (because I force the port to be 8088 via the python command in the rc.local file, as you so nicely detailed).

Thanks for all of your work on this!

[craigmw]

BTW, setting the number of extension boards to zero, saving and then back to 1 appears to have fixed the first issue. I’m not sure how this might have gotten corrupted, but it seems to be working fine for now.

[Dan in CA]

craigmw,

There a couple of option settings that the Python interval program does not use including the port number and rain sensor related settings.
I included them to maintain compatibility with the Arduino based micro controller version of the program. They could be implemented in the future.

I have been thinking about making the port number functional but it would probably require some operating system level code and that would add to the complexity of the set up.

Changing the expansion board number does cause the snames.txt file to be re-written as you discovered.

I will include that in a trouble shooting section of the documentation that I am working on for the wiki.
At this point the wiki page is located at:
http://rayshobby.net/mediawiki/index.php?title=User:Dan_in_CA
but it is not yet listed in the navigation pain for the wiki. I think Ray would need to add it there.

EDIT:
The documentation for the Python interval is now located at:
http://rayshobby.net/mediawiki/index.php?title=Python_Interval_Program_for_OSPi
and can be accessed through the wiki menu under OpenSprinkler Pi.

Dan

[craigmw]

Actually, I don’t see a need to make the port number field within the program functional, so long as it can be set in the command line. I was just concerned that having it set to the wrong value would impair communications. Obviously it works just fine.

Thanks for the link to the Wiki. I’m still confused about how to have different station times. I know that it is possible to set it so that I can have more than one program run for certain stations to add to the times run. But, I was under the impression that this could be done by running the programs with the same start time. This does not seem to work in my hands.

[Samer]

@craigmw I can confirm this feature works so there is probably something configured incorrectly. The best thing to do is to use the program preview for guidance. Make sure the program interval is greater than 0. Make sure if you are using multiple programs with the same start time that the selected stations don’t overlap between programs. Maybe give us some generic information on how your programs are setup and we can help further.

[craigmw]

@salbahra wrote:

@craigmw I can confirm this feature works so there is probably something configured incorrectly. The best thing to do is to use the program preview for guidance. Make sure the program interval is greater than 0. Make sure if you are using multiple programs with the same start time that the selected stations don’t overlap between programs. Maybe give us some generic information on how your programs are setup and we can help further.

Samer – Wow, you are quick! I think I’ve figured this out. Previously, I had three programs two of which that started at 3:30 (end time at 5:00). One of these programs was set to run for 4 minutes on all stations, and the second was to run 4 additional minutes on two stations. A third program was set at 14:00 to run two stations. The two programs set to run at 3:30 did not show up together. I could see the events at 3:30 all from one program (on the 9 stations), and the events at 14:00 (on 2 stations). Note that the three programs I had scheduled ended up in reverse order (due to a previous bug).

So, within your webapp, I deleted all three programs to redo this and it now works fine. I set up Program 1 to start at 3:30 AM (end time at 5:00 AM) to run all 9 stations with 4 minute duration and 1439 minute interval (to run only once daily). Program 2 was originally set for 3:30 to 5:00 AM to run only on two zones that need extra water, 4 minute duration and 1439 minute interval, and Program 3 from 14:00 to 14:30 with 4 minute durations and 1439 interval also to run on 3 stations that need more water. This time, I could see that Program 1 and Program 2 were interspersed, with the two stations in Program 2 showing up interspersed with Program 1 events. So, I adjusted the duration for Program 2 to 8 minutes, and now I see Program 1 and 2 events interspersed with 4 minute durations for 7 stations, and 9 minute durations for the two. Program 3 shows up with events at 14:00.

So, I think I get the logic now. If you want different times per station, you need to add programs with an incremented program number on top of a base program (e.g. Program 1). The additional program(s) need to have the intended duration set… the events in additional programs scheduled concurrently do not add time to the “base” program. So, it seems that this must be calculated on the fly (e.g. as the schedule is running, or in the Preview Programs page).

It would be amazingly cool (and far beyond what is currently out there from my understanding), if the calculated durations could include a multiplier (e.g. duration * multiplier) where the multiplier could be 0 to 2 (e.g. 0 to 200%). A daily call to an internet weather site for location specific evapotranspiration info could be used to set the multiplier based on precipitation, humidity, wind, temperature, etc. This would be far more refined than just having the programs turn off when the Rain Delay is flagged manually.

[craigmw]

Dan – One other thing…

On the Wiki page for your program on the OSPi, you might want to add a SLEEP 60 to the section on automatically starting up the interval program. I had trouble with this because when I switched to a Class 10 SD card. Apparently, there is not enough time for the RPi to gather the WiFi IP address before the script runs. Putting a SLEEP 60 just before your code in rc.local seems to fix this. I had been running my OSPi with a Class 4 SD card, but it became corrupted and I switched to a Class 10 card. After this, the interval program would not autoload.

My rc.local is:

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will “exit 0” on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

# Print the IP address
_IP=$(hostname -I) || true
if [ “$_IP” ]; then
printf “My IP address is %sn” “$_IP”
fi

sleep 60

host=$(hostname -I | sed ‘s/ *$//g’)
port=:8088
cd /home/pi/OSPi/
## If you aren’t sure that the path to python on your Pi is /usr/bin/, use whereis python to determine the location
/usr/bin/python ospi.py $host$port

exit 0

[Samer]

That actually exists and is referred to as the water level and can be adjusted from 0-200%.

Sorry I haven’t read your whole post been a busy afternoon and will hopefully digest it more later but it sounds like you have got everything working now!

[craigmw]

@salbahra wrote:

That actually exists and is referred to as the water level and can be adjusted from 0-200%.

Sorry I haven’t read your whole post been a busy afternoon and will hopefully digest it more later but it sounds like you have got everything working now!

Samer – Yeah, I figured this out just now looking at the Device Settings page. Given this, how difficult would it be to add a subroutine to check for evapotranspiration info? If that water level parameter is stored in a file, would it possible to set up a small cron job to alter the parameter in the settings file once a day? I would guess that a Python program would be capable of doing so, or it could be added to the interval program. Would changing the parameter in the file impact the interval program (i.e. would the interval program update its stored water level value by loading it from the file, and if so, how often does it do this)?

[Samer]

You can accomplish what you want without touching any files or modifying things directly. Both the OpenSprinkler and the interval program for the OSPi support an API for changing settings. For the watering percentage it can be modified using the following command:

http://192.168.1.10:8080/co?pw=opendoor&o23=250

The above code will change the watering level to 250%. The range is 0-250. This is a direct multiplier on the minute duration for the stations.

With this information it is absolutely possible to augment my already present weather based rain delay to get more control. However, you need more robust weather sources than Yahoo. For weather underground an API key is needed which is why I haven’t switched to this service. I also don’t have an algorithm to base my program.

[Dan in CA]

@craigmw

It think the Water level % in options is what you are describing as a multiplier.

Dan

Samer, you beat me to it.

[craigmw]

Thanks Samer and Dan. I guess I need to investigate how these work already! In any event, how difficult would it be to access the weather underground data and use this to tweak the Water Level parameter?

[Samer]

Weather Underground is very easy to poll since they have a well documented API. What you need is an algorithm you can trust.

Example:

1″ of rain – delay for 12 hours
2″ of rain – delay for 24 hours
Temp > 90 – water level 150%
etc.

This is less about programming and more about researched data. For a system like this to work, you have to be able to trust it will make the same decisions you would, or an expert would. It has to cover all possible scenarios otherwise it’s useless (if it misinterprets data and activates sprinklers while it’s raining, for example).

I think Weather Underground’s data can be trusted however I have not seen a solid algorithm, yet.

If you can show me or convince me something is worth while, I have no problem switching away from Yahoo and building it right into my web app. But, that is a lot of effort and unless I have tangible data showing the algorithm is legitimate I don’t think it would be worth it.

Update: A flow chart for example would be amazing. I could also be over thinking however trusting a program to modify your schedules seems scary (yard dies if it miscalculates while you’re on vacation). In my opinion, this is something you use until it messes up and then you never use it again. So, if we roll this out it needs to be perfect from the start.

[craigmw]

I agree, Samer. This would have to be reasonably rock solid. A commercial example of this is:

http://www.rainbird.com/landscape/products/controllers/ETmanager.htm

I will look around to see if I can identify some solid algorithms that would take into account weather data (e.g. wind speed, humidity, temperature, rainfall, etc) that spits out a reasonable evapotranspiration value that might be easily used for this.

As for killing someone’s yard while on vacation, there are several issues here. First, if the hardware goes on the fritz in an infinite loop, this could lead to uncontrolled open valves and flood something, which would be bad. I would think that coming home to some dead plants would be the better alternative than a flooded neighborhood or thousand dollar water bill. Even so, this is a hardware problem that could be addressed with other hardware, but beyond the scope of our discussion.

[Samer]

You are absolutely right about your scenario being worse than mine 😛 But as you said that is a hardware issue which is outside of my control quite frankly. When it comes down to it people buy devices like that Rain Bird not for the hardware but for the software preloaded. We can absolutely do the same thing and probably better just need some information.

It sounds like you have the right metrics in mind and a good end goal metric however I have zero time to research. Again, if anyone can bring me the information I would be glad to incorporate it!

[Dan in CA]

craigmw,

Being able to use ET data for irrigation management was actually my goal when I started this project.

Its pretty complicated. There are basically three levels of detail you can use for irrigation management.

1. Monthly adjustments based on historical weather data.

2. Daily adjustments using minimal weather data and the Hargreaves-Samani equation

3. Daily adjustment using additional weather data and the Penman-Monteith equation

All these methods take into account the type of plants and growth stage (crop factor), along with such things as soil type and efficiency of the irrigation system. You can read all about it at:
http://www.fao.org/docrep/x0490e/x0490e00.htm#Contents

Here in California we have a network of specialized weather stations specifically for the purpose of providing Evapotranspiration data. I already have Python code for accessing this data as well as code for both Hargreaves-Samani and Penman-Monteith calculations. This is as yet untested.

These techniques are mostly used in agriculture but there is development of landscape related applications. A very informative free publication can be found at:
http://www.water.ca.gov/pubs/conservation/a_guide_to_estimating_irrigation_water_needs_of_landscape_plantings_in_california__wucols/wucols00.pdf
This is focused on California landscapes but the principles can be applied anywhere.

Ideally it should be possible to develop a system that can be used anywhere in the world.

Dan

[craigmw]

Dan:

Yeah, as I’m also in California, I’m aware of the local ET info. In my case, and likely the case of most that have typical landscaping, my zones are a mix of different types of plants and vegetation. So, I really just need to come up with an averaged value for the entire property based on our location. Monthly historical values would make little sense. Here in SoCal, as you know, one day it can be raining and the next day, we have Santa Ana winds. Even on a day where there is some sprinkling, it might make sense to have some watering. So, it would be nice to simply come up with a simplified algorithm taking into account humidity, precipitation, wind, and temperature to alter the Water Level parameter.

[Dan in CA]

That second link in my previous post has instructions for just that.

Dan

[craigmw]

Maybe we are overthinking this. For most of us using this to control sprinklers in our yards, a simple Water Index value as such would be most appropriate:

http://www.bewaterwise.com/wat_index_02.html

Perhaps I can email them to find out what parameters they use to come up with their Water Index (WI) value.

Edit… Okay, the Watering Index is simply: (ETw / ETwm) * 100, where ETw is the total of the previous 7 day ETo values, ETm is the highest weekly ETw in the last year. This is pretty straightforward, but the real issue is how to obtain those ETo values. This is not particularly difficult in California due to the CIMIS system (http://wwwcimis.water.ca.gov). I’m not certain how one would automatically obtain those values for a specific location using the web. If that were doable, it would be great for those of us in California. But, it would be preferable to have a way to use web-based weather data to estimate ETo for a specific location, in the spirit of providing this functionality to everyone. After looking through the calculations, it doesn’t look all that simple. The simplest calculations only consider temperature. Also, the Penman–Monteith equation appears to have several variables that are unlikely to be readily available from weather sites. However, some of those variables may be less likely to change over a yearly basis, and thus could be removed from the calculation as the WI gives a ratio, not an absolute value. We really don’t need to know the predicted volume of water lost, just a daily fraction of water lost to evapotranspiration vs. the maximum daily water lost for a specific location.

[Dan in CA]

Sounds like you’re on the right track. It needs to be fairly easy.

Some of the data used in the calculations don’t change over time such as location (Lat. and elevation). So there needs to be a one time configuration plus the ET value from the weather source.

I understand Australia has a system like the one here in CA and also I think Idaho does but that is still just part of the world.

Ray once mentioned that many of his customers are outside the U.S.

For the U.S., the NOAA has freely accessible data available:
http://www.nws.noaa.gov/view/national.php

There may be similar resources for other parts of the world.

This is where thing really start to get interesting.

Dan

[daver]

Hi Dan,

As someone who followed your install instructions in the exact order presented, I just wiped out my program and station data (DOH!) because the backup step is AFTER the tar xvf step.

Can I make a suggestion?

Put the data directory OUTSIDE of the regular OSPi, making upgrades a snap and allowing people to keep their data in tact without the extra step. I imagine as this gets more popular, this will be a more persistent problem. Seems easy enough to fix with an ENV variable to point to a well-known data location and have the program copy the data there on the first run if nothing is around, otherwise use the data files from that location and never copy them again.

I would also suggest a full reboot as part of those steps, because I noticed that my history logs weren’t displaying the web app after I upgraded, nor was I able to edit station data until I did this. Just FYI on what I saw.

Thanks as always for everything you guys are doing! I love showing this off to my friends and family. They’re all amazed when I start the sprinklers from my iPhone.

[Author]

Posts