Interval – Plugin development

[scottsh]

I figure I’ll post here for things I’m finding during my plugin development for the Interval program. Overall, plugins are relatively easy to develop.

A few notes in case you want to write one also – I’ll try to keep this post updated as I run into things I think are helpful.

Plugins are python scripts that reside in the ./OSPi/plugins directory. The main ospi.py file will attempt to load any .py file it finds in that directory as long as the execute bit is set for the group attribute. While I am working on my script, I have attributes set to 777 (rwx for all).

[list=2:zsqneqkh]You will most likely want to import the global variables from ospi – do that by importing gv

import gv

[list=3:zsqneqkh]If you want to extend the UI via web pages, you should do that by extending the URL list.

from urls import urls

urls.extend()

[list=4:zsqneqkh]To debug your plugin, you’re going to want to stop the ospi service from running in the background and instead run ospi from the command line and watch for debug output.

sudo service ospi stop

sudo python ospi

[list=5:zsqneqkh]If your plugin uses web.py to display web pages, your template should go into ./OSPi/templates

[scottsh]

UPDATE: Either you fixed it or I had an older version of the code – in the latest pull this is fixed. Thanks!

Dan,I’m noticing that gv.snames isn’t being updated when the station names are updated through the UI. The ospi code uses data(‘snames’) in most places itself, so this isn’t apparent.

I’m thinking somewhere around line 704 gv.snames should be updated?

gv.snames=data('snames')

[Ray]

Cool. If possible, would you mind putting the instructions on the wiki page here:
http://rayshobby.net/mediawiki/index.php?title=Python_Interval_Program_for_OSPi
Thanks!

[scottsh]

I will – I didn’t know this wiki existed, thanks for pointing me to it.

Update: Done. I modified the previous add-on section with what I think is now accurate information.

[scottsh]

Dan (or anybody familiar with the Interval program) – what are your thoughts on the best way to determine the historical watering data? I suppose I could use the log if you think that’s best? If so, I’m going to want it to be able to hold 7 days worth of log data (rather than number of entries.)

Or, is there a need for a database of some kind that would hold historical watering data?

Scott

[Dan in CA]

scottsh,

I have tired to keep things a simple as possible. That’s why data is stored in json/text files rather that a db. If you only need 7 days of data, a db might be overkill.
That said, the number of records that will cover 7 days depends on the number of schedule programs, if zones run several intervals in a day and the number of active zones. Keeping things in json files makes it easy to load data into Python lists and dictionaries.

Of course it depends on the requirements of your plugin.

EDIT: One thing to keep in mind is how easy it will be for users to install and setup your plugin. That is one of the reasons I stayed away from using a DB.

Just as a matter of interest; I have some fruit trees that get watered once every 14 days. There may be other cases where schedules span more that 7 days.

I think the current log is probably not in the most efficient format. If you would find it useful to have the records hold different data or in a different order or something it is certainly possible to reconfigure it.

Dan

[scottsh]

Great points – I’m still thinking about the best way to do this. The information I need is the amount of watering done over the previous week (or whatever specified time interval – as you state you have some 2 week cycles.)

Here is what I consider to be the worst case scenario. I see in another post that somebody has 6 expansion boards. That’s 56 zones. If every zone was activated 3x per day every day, you’d end up with over 1000 entries per week. I did say ‘worst case’ ;). In my real-world case, I have 24 zones that are activated 3x per day 3x per week. That’s only 200 entries. I’m worried about consume too much memory, but maybe I’m over-thinking it. Currently my Pi (128MB version) has 23MB of free RAM, so there does appear to be plenty of memory even for 1000 entries.

[Dan in CA]

Sounds like you’re planning ahead. That’s always a good thing.

If memory won’t be a problem, keeping the data in a list or something would probably be good. Database programs tend to write to disk a lot and this can be hard on the SD card unless the data is stored somewhere else.

I have a couple more thoughts. Of course its your plugin so feel free to ignore my suggestions.

1. OpenSprinkler is used all over the world but here in the U.S. we are still using imperial measurements. Calculations are easier with SI (metric) units and most of the weather data sources I have seen offer both SI and imperial units. You might consider using SI units internally in your program and offer the user the option of using imperial units in the UI. I think you would only need to convert inches -> mm, Gallons -> liters and possibly sq ft -> sq m.

2. What ever source of weather data you decide to use it might be a good idea to keep the data acquisition code in a separate (importable?) module. That way you or someone else can add support for other data sources as needed.

Dan

[scottsh]

Thanks Dan, these are good thoughts. Right now I can switch from inches to mm with nothing but a UI change. It’s not complete yet, but the algorithms remain the same. So far, I’ve been able to avoid any code that required me to use inches as anything other than just a unit of measure. In other words, I don’t have to divide by 12 or calculate area or something like that with it.

I had the same thought that you did to allow for the weather history data to be a module people can change. I’ve designed the code and UI to allow this but haven’t implemented it yet because I’m so weak with Python. But it will happen.

One question – I’m at the point where I have to create a schedule program automatically. However, the code is very hard to decipher. Do you have any thoughts on the best way to have my plug-in create a program or set of station schedules?

My first thought was to create a new set of programs, but that’s looking like I might have to create a new program for every zone because it is likely that each zone will have its own unique precipitation rate (at least this is true in most of my zones) and potentially a unique evapotranspiration value too. This means each day, each zone will need watering on a unique duration. My next thought was to create a single program with each station having a unique duration value. Does that sound right? I was thinking of copying the code out of timing_loop, settings up the station values and calling schedule_station. Does that make sense?

[Dan in CA]

Scottsh,

Creating a program for each zone using the existing UI and then modifying the durations for each program on a daily basis would be a good approach

The program data is held in memory as the list of lists gv.pd (program data) and as the file programs.json in the data directory.
The format was ported from Ray’s micro controller code where every bit had to count because of limited memory.

Here is a breakdown of program data that I took straight from Ray’s C code with added notes:
pd[0]=[
v[0] = 1, //Program enable flag (byte/boolean)
v[1] = 1, //days byte (includes a flag for odd – even restriction) Mon. = 1, high bit: 0 = no restriction and no interval, 1 = odd/even parity or interval enabled.
//When used for interval: high bit = interval on, rest of byte used for days countdown set as “starting in” = days remaining in cycle.
v[2] = 0, //Interval days: if 2 -> 255 = interval enabled (most sig bit set), — if value is 1 (least sig bit set) = odd restriction, if value is 0 = even restriction. This field shows 3 by default in the UI when no interval is selected.
v[3] = 360, //Start time in minutes (from 0 at midnight)
v[4] = 1080, //End time in minutes (from 0 at midnight)
v[5] = 240, //Repeat (cycle) interval in minutes
v[6] = 900, //Duration in seconds for each active zone in this program
v[7] = 1]; //Zones bytes – indicates which zones are active (selected) in this program).

As you can see the duration (in seconds) for a program is stored in element 6 of the program data list. You should be able to modify the duration of any existing program from your plugin.

Here is an example of a program from Python’s gv.pd list:
1, 21, 0, 360, 1080, 240, 900, 21

And the corrisponding program creation page:
[attachment=0:3qj05420]Prog page.jpg[/attachment:3qj05420]

EDIT: Here is a web site I found extremely useful for visualizing bit patterns and figuring out what the values in the program data represent
http://calc.50x.eu/

Their Timestamp page is also very helpful
http://unixtimestamp.50x.eu/

This whole project has been a huge learning experience for me. I think you will gain some valuable knowledge from developing your plugin

Dan

[scottsh]

Thanks Dan, this helps a bunch – I appreciate the pointers. I get why the stored attributes (nbrd, bsy, etc.) had short names to work with the Arduino code.

Is there any real limit to the number of programs? I was thinking about making all the automated programs have pid beyond the current max (32) to avoid conflicts. Again, thinking about the maximum number I could see the need for 56 programs to handle all the various zones that might be in use.

Is all of gv documented somewhere? In particular I’m looking to understand what gv.rs and gv.ps are.

Thanks, Scott

[Samer]

The gv.rs is rain sense bit which should reflect if the rain sensor is on/off. The gv.ps is station status in regard to programs. It is an array of all stations and each element is 2 numbers (first is program id and second is seconds left).

The program ID indicates a user program ID (1,2,3,etc), manual program (99 or 255) or a run-once program (98 or 254). A program ID of 0 means no programmed is scheduled to run on that station.

You can compare this information with the current status of each valve. This would allow to the discern between an open valve versus a scheduled valve opening.

[Dan in CA]

All of gv is not yet documented but a good part if it, the gv.sd (settings dictionary), is described in the file sd_refertnce.txt in the OSPi directory.

Also, if you look in ospi.py starting at line ~378. The Global vars section is where the all the gv (global vars) attributes are defined. There are some comments describing what they are.

I am planning to add the rest of the gv info to the sd_reference.txt file and probably rename it to gv_reference.txt. I’ll post here when it is ready.

EDIT: Forgot to answer your question about the 32 program limit. That was the case on the micro-controller version of OpenSprinkler. I left the JavaScript files that produce the UI pretty much as they were. So, short answer: disregard the 32 program limit.

EDIT: Here are the rest of the gv attributes:
gv.ver firmware rev number (int, based on firmware version this program was ported from)
gv.rev release number of this program (int)
gv.revdate date of release (string)
gv.now current time, updated once per second at top of timing loop
gv.srvals shift register values, used to turn zones on or off (list of one byte per station, 1 = turn on, 0 = turn off)
gv.rovals run once values – list of duration times in seconds for a run once program (list, length = number of statons)
gv.pd program data – loaded from file at startup (list of lists)
gv.ps program schedule used for UI display (list of 2 element lists i.e. [program number, duration])
gv.pon program on (Holds program number of currently running program)
gv.sbits station bits, used to display stations that are on in UI (list of bytes, one byte per board)
gv.rs run schedule (list [scheduled start time, scheduled stop time, duration, program index])
gv.lrun last run, used to display log line on home page (list [station index, program number, duration, end time])
gv.scount station count (not used — dead code ????)
gv.snames station names, loaded from ‘snames.txt’ file (string)

Dan

[scottsh]

Thanks Dan – that helps – I somehow missed that section of ospi defining gv.

salbahra, I don’t think gv.rs is the rain sense bit. I think its the run schedule.

gv.rs = [] #run schedule
for i in range(gv.sd):
gv.rs.append([0,0,0,0]) #scheduled start time, scheduled stop time, duration, program index

I was thinking I might be able to create a program that had a unique duration for each zone by setting gv.rs[station_id] with start, stop, and duration times, set program index to maybe 98 (to indicate an auto-program) then set bsy=1 and let the controller go.

This way, I avoid messing with the user-created programs (which might be confusing to users).

From what I can see, gv.ps is the ‘program schedule’ and isn’t used in the OSPi? It looks like it was just for the display of the Arduino version?
UPDATE: Whoops, I was wrong – gv.ps is used by the web UI – so it is needed.

scott

[Lamby]

@Dan in CA wrote:

All of gv is not yet documented but a good part if it, the gv.sd (settings dictionary), is described in the file sd_refertnce.txt in the OSPi directory.

Also, if you look in ospi.py starting at line ~378. The Global vars section is where the all the gv (global vars) attributes are defined. There are some comments describing what they are.

Dan

Ah! Thanks. I have been looking for those!

I am trying to do a sunset light plugin. If we want to set some options (like latitude and longitude) should we create a new options page specifically for the plugin or should we amend the main options page?

[Dan in CA]

Lamby,

If your plugin is something you would like to share with the community, I would suggest keeping it as self contained as possible and not modifying the existing UI. However there is already an unused location field on the option page that could hold lat-lon data (gv.sd).

It sounds like you are planning to calculate sunset time from geographic location rather require a light sensor. That would be really cool.

Dan

[scottsh]

I think I have an alpha version that works, I think. If anybody is interested they can clone it from https://github.com/scottshaffer/OSPi.git

To use the new plugin(s), browse to http://ospi:8080/auto. It does require that you setup the Zone information and Weather settings before working.

I’ve added the ability to simulate operation. It does everything as normal except it prevents OSPi.py from writing to the GPIO pins. This will prevent your OSPi from working at all, not just the automation.

UPDATE: Drip irrigation support is not working yet – it will dramatically over-water those zones if used.

Scott

[Dan in CA]

The home page loads fine but when I try */auto there is a server error in the browser and on the command line:

File “templates/auto_program.html”, line 125, in __template__
KeyError: ‘simulate’

Can’t wait to see what you’ve done.

Dan

[scottsh]

Oops, bug in initializing an empty settings file. I fixed it and pushed the changes.

Also, if you notice I had to make small modifications to ospi.py and gpio_pins.py. The ospi.py change is twofold – first I modified added the station # to the log file. This allowed me to scan the log for watering history without having to string-match the station names. Second – and it involves the change to gpio_pins.py as well – is the addition of the simulation capability. I added a gv.simulate flag (initialized in ospi.py) that prevents the writing to the registers. The idea was to let the software run completely and exercise it without actually triggering solenoids.

These 2 changes are obviously not aligned to a plug-in strategy. I’m hoping to convince you of the need for the log change :). The simulation capability is probably only useful for developers, so that is really just for me.

Scott

[Dan in CA]

Hmm..

Pulled the update, still getting the same error.

Dan

[scottsh]

You might have to rm the wx_settings.json file in data and make it create a good one.

I just did a fresh clone and it loaded OK.

[Dan in CA]

OK. I deleted the OSPi directory and re-cloned. Now it works fine.

Looking really good, especially for such a short development time using what had been an undocumented plugin system.

I like the fact you included drip systems. My entire place is on drip, no lawn, no “sprinklers”.

Your change to the log output can stay. I will add it to my dev. version.

Dan

[Lamby]

@Dan in CA wrote:

Lamby,

If your plugin is something you would like to share with the community, I would suggest keeping it as self contained as possible and not modifying the existing UI. However there is already an unused location field on the option page that could hold lat-lon data (gv.sd).

It sounds like you are planning to calculate sunset time from geographic location rather require a light sensor. That would be really cool.

Dan

Doesn’t the mobile app use gv.sd for the current weather?

[Samer]

Yes it does 🙂

[Dan in CA]

Yup. I forgot about that.

I guess the answer is that it is best to make a separate options page for your plugin. That will also help avoid conflicts as more plugins are developed.

Dan

[Author]

Posts