How to read current station status?

[snewman]

Hello,

In going through the existing demos, I couldn’t find (or just didn’t see?) a way to identify which stations are running. I’m struggling to understand the concept of shift registers, so this might be trivial and I don’t know it – feel free to chide me…

I’ve created a basic interface for scheduling timed jobs:
https://github.com/greencoder/opensprinklerpi-neptune

On line 90 of opensprinkler.py, I’d like to be able to just set the current station to zero and not all stations. (This presents a problem when jobs overlap)

Does anyone have a strategy for polling the hardware to see what stations are active?

[andrew]

I’ll chime in with some blind speculation. Ray (or possibly someone else) will need to provide a more authoritative answer.

If I had to guess, I would think that the communication between the RPi and the OS controller board is one-directional. In other words, the RPi sends commands to the board, but does not receive anything back in the way of status updates. All of the demo programs that I’ve looked at simply remember the last state that it sent to controller. Therefore, if you wanted to obtain the state of your controller (i.e., which stations are open/closed) you would need to query the program that was responsible for issuing the commands to the controller (and if it’s your own program that’s sending the commands, you’ll need to remember the current state in memory).

More generally, I think that what you may be misunderstanding is that every time you muck about with the shift registers, sending commands to controller board, you are effectively setting the *entire* state of the controller, not just changing the state of one or two specific stations. Via the shift registers, you’re basically sending a sequence of 1s and 0s to the controller with each bit representing the desired state of one of the specific stations. If, then, you send the binary 0010100 to the controller, you’re telling it to open stations 3 and 5 and keep all of the others closed. (Let me reiterate, this is speculation. I may have 1s and 0s reversed, and there may be some other details I’m missing. Don’t take this as gospel).

If you want to do some funky stuff with your own program where you are allowing for overlapping jobs, you’ll need to work out the logic on your own and send the complete state to the controller each time you want it to change. For example, let’s say you have the following desired schedule:

At time t0, start station S1.
At time t1, start station S5.
At time t2, stop station S1.
at time t3, stop station S5.

In the above, you have two jobs. One for station S1 which starts at t0 and stops at t2 and another for station S5 which starts at t1 and stops at t3. The commands you’ll need to send out via the shift registers are as follows:

At t0: 1000000
At t1: 1000100
At t2: 0000100
At t3: 0000000

At time t2, for example, even though we really only wanted to turn off station 1, we still had to send details to the controller that station t3 should still be active.

I could give you some additional guidance as to how this might be achieved in your own custom program, but my post is already rather wordy. Let me know if you’re curious and I’ll write it up.

Edit: you may want to consider, by the way, that allowing for overlapping jobs could have undesirable real-world effects. Many users, I suspect will not want multiple stations to be open at the same time because there may not be adequate water pressure to handle more than one station at a time.

[andrew]

By the way. You mentioned that you are struggling to understand shift registers. This article offers a good explanation: http://bildr.org/2011/02/74hc595/. This will also explain why the the complete state of the controller has to been reset each time a station is opened or closed. The shift register is cleared out and 8 new bits are sent to the register (or more if you are using an expander board; the linked article also explains how shift registers are daisy chained, which is exactly how the expansion boards work — I think).

[doczaius]

Check the interval program… requesting http://xxx.xxx.xxx.xxx/sn1 (sn2, sn3 etc) will return 0 or 1. I assume its doing some polling…

[andrew]

@doczaius: I don’t believe that it’s polling anything, and this is the point I was making above. The interval program keeps a variable in memory that represents the current state of the stations. When a station is “turned on” the bit for the station in that variable is set to a 1 (or a 0, not sure which) and then the program calls the set_output function, which then takes the values for all of the stations and sends it out through the GPIO to the shift registers on the controller board. There is no polling of or feedback from the controller board that I’m aware of. The program just assumes that whatever *it* last sent out to the shift registers is what the current status of the controller is. If you’re running the interval program, and then in a separate process send some new values to the shift registers, thereby changing the state of the controller, the interval program will be completely unaware of the change, and still think that the controller’s state is whatever it had previously set it to.

That being said, if some is running the interval program exclusively, they can indeed use http://xxx.xxx.xxx.xxx/snX to obtain the state of the stations (as far as the interval program is aware). I believe, however, that snewman was intending on writing/running his own program and would not, therefore, be making direct use of the interval program.

[virtus]

@andrew wrote:

By the way. You mentioned that you are struggling to understand shift registers. This article offers a good explanation: http://bildr.org/2011/02/74hc595/. This will also explain why the the complete state of the controller has to been reset each time a station is opened or closed. The shift register is cleared out and 8 new bits are sent to the register (or more if you are using an expander board; the linked article also explains how shift registers are daisy chained, which is exactly how the expansion boards work — I think).

So if the interval program is in the middle of a program (let’s say valve #1 is open), and you manually tell it to open valve #3 using the http command, does it automatically close all the valves first, then open valve #3?

[snewman]

Wow, thanks for taking the time to thoroughly answer my question!

Andrew – This helps quite a bit. It looks like my understanding was the same as what you sent.

Virtus – Correct; every time a command is received it first closes all valves then opens the desired one. This prevents multiple valves from being opened at a time but presents a challenge when the original valve is ready to turn itself off. I’ll have to come up with some better logic to manage state. (but I’m trying to keep this simple and not run task queues and databases)

Doczaius – I have looked at the interval program. It was written by someone who obviously knew what they were doing, but the code is a bit obtuse with short variable names and a lack of comments. Does anyone know who the original author was? Perhaps he/she can answer some questions so I can go through it line-by-line.

[andrew]

@snewman – The interval program was initially written by Ray, and ported to python for use with a Raspberry Pi by Dan Kimberling.

@Virtus – This question is probably a bit out of scope for this thread, and is best answered by Ray or Dan. If your question is aimed at understanding at a higher level how shift registers work and are used in the interval program, then @Virtus’s response is probably good enough. When any program (including the interval program) desires to change the state of a station, it must specify the entire state for all stations, and will overwrite the existing state completely. In the interface between the RPi and the controller board, there is no notion of “just change the state of station X”. You are always indicating the entire state of all stations and the existing state is always overwritten in its entirety. If, on the other hand, you really want to know precisely what happens in the circumstances you described, the answer may be a bit more complicated. Dan is probably best suited to answer, but from what I can see in the code (and I’ve only taken a cursory glance), you may actually get some very strange results. It appears that if you set manual mode while a program is already running, the manual mode thread will actually be competing with the main loop that is executing the normal program, and it looks like they’ll just go on interfering with each other until either the program ends, or the manual mode is turned off. Each of the competing threads will, during the course of their respective executions, be setting various states for all of the stations Unless I’m missing something (and in fact I probably am), this would be a bug, but not a particularly important one.

[snewman]

Andrew – your continued help is very appreciated!

Looks like a good solution is to write some kind of long-running scheduler/controller that is the only process allowed to manipulate the shift registers. It can also act as a state manager to keep track of the status of stations. I really don’t want to require people to set up task queues or other network processes, so I’ll probably do it as a simple HTTP-based system that uses threads.

[doczaius]

@andrew wrote:

@doczaius: I don’t believe that it’s polling anything, and this is the point I was making above. The interval program keeps a variable in memory that represents the current state of the stations. When a station is “turned on” the bit for the station in that variable is set to a 1 (or a 0, not sure which) and then the program calls the set_output function, which then takes the values for all of the stations and sends it out through the GPIO to the shift registers on the controller board. There is no polling of or feedback from the controller board that I’m aware of. The program just assumes that whatever *it* last sent out to the shift registers is what the current status of the controller is. If you’re running the interval program, and then in a separate process send some new values to the shift registers, thereby changing the state of the controller, the interval program will be completely unaware of the change, and still think that the controller’s state is whatever it had previously set it to.

That being said, if some is running the interval program exclusively, they can indeed use http://xxx.xxx.xxx.xxx/snX to obtain the state of the stations (as far as the interval program is aware). I believe, however, that snewman was intending on writing/running his own program and would not, therefore, be making direct use of the interval program.

I was afraid of that — the reason I mentioned was because the previous version of the interval program did not accurately reflect whether zones were on or off if a command was sent via get. I assumed the GUI was referencing a variable where as the GET commands were not (unless of course there are two sets). I have the interval program installed but I don’t use the built in scheduler because I need multiple zones to open/overlap. So I have crontab set up to execute manual commands with cURL. If I wanted to know the true status, I would have to manually goto /snX to view. The new version of the interval program seems to have fixed this issue however.

[doczaius]

@virtus wrote:

@andrew wrote:

By the way. You mentioned that you are struggling to understand shift registers. This article offers a good explanation: http://bildr.org/2011/02/74hc595/. This will also explain why the the complete state of the controller has to been reset each time a station is opened or closed. The shift register is cleared out and 8 new bits are sent to the register (or more if you are using an expander board; the linked article also explains how shift registers are daisy chained, which is exactly how the expansion boards work — I think).

So if the interval program is in the middle of a program (let’s say valve #1 is open), and you manually tell it to open valve #3 using the http command, does it automatically close all the valves first, then open valve #3?

I use the manual commands specifically to open multiple valves at the same time — so I can’t speak for the scheduler.

[andrew]

@snewman wrote:

Looks like a good solution is to write some kind of long-running scheduler/controller that is the only process allowed to manipulate the shift registers. It can also act as a state manager to keep track of the status of stations. I really don’t want to require people to set up task queues or other network processes, so I’ll probably do it as a simple HTTP-based system that uses threads.

Precisely. You might even want to go a step further and decouple the scheduler from the controller. You’d have one process (the controller, or perhaps more aptly named, the driver) which is the only process that actually interfaces with the shift registers (it would be a single, running process). It should support low-level commands like set_state and get_state which sets/gets the entire state of the shift registers (again, getting the state would have to be implemented by simply recalling the last state that was sent). It might also support some slightly higher-level commands like turning on/off specific stations. Then the scheduler (or better yet, the controller, if you’re calling the other thing the driver) would operate at the level of individual scheduled programs, and it would handle when a program activates specific stations during the execution of a program, when programs are executed, and (most relevant to this thread) what happens when multiple programs are trying to run at the same time (e.g., whether programs are smartly combined, or one overwrites the other, or one is delayed until the other is finished, etc.) This process would also be a single, long-running thread (and technically could be in the same process as the driver; the decoupling is more for organization). You may want to look at some existing scheduling libraries to help with the timing stuff (e.g., http://pythonhosted.org/APScheduler/).

I’m thrilled that you’re working on a separate implementation. I’ve been happy with the interval program in conjunction with the mobile web app, but there’s never anything wrong with healthy competition, especially in the open source community.

[virtus]

@snewman wrote:

Virtus – Correct; every time a command is received it first closes all valves then opens the desired one. This prevents multiple valves from being opened at a time but presents a challenge when the original valve is ready to turn itself off. I’ll have to come up with some better logic to manage state. (but I’m trying to keep this simple and not run task queues and databases)

That’s good to know – just got my OS RPi today and will begin tinkering with it soon. I had originally planned on using one of the OS valve controllers to activate a relay and simulate a button-push for my garage door openers. It sounds like opening / closing the garage would interrupt any watering programs that might already be running. Now I might have to rethink that and move the relay to one of the Rpi’s GPIO pins and use custom python code to toggle the relay. Anyway, that’s another topic for a another thread. Maybe snewman’s program will handle it better.

[Ray]

Sorry guys, just had time to read the thread here, and I can help answer some of the questions:

– The control of shift register is a one-way communication: the microccontroller or RPi sends command and the shift register changes state. Your program usually has a variable that stores the current state of the shift register (for example, in OpenSprinkler firmware, it’s an array called station_bits). Since the microccontroller is single-threaded, the firmware variable and the actual state of the shift register should always match. The only exception is when there are signal interference, which may cause a bit flip on the shift register. This is why the OpenSprinkler firmware refreshes shift register every second to ensure it matches the firmware variable. Now, on RPi, since you can potentially run multiple processes / threads that simultaneously write to shift register, you need to be careful to avoid concurrent writing.

– Contrary to what was mentioned in this thread earlier: when updating the 74HC595 shift register, the stations will NOT all reset, that’s why you don’t see all valves close first, and then one of them re-opens. This is because the 74HC595 shift register has a storage buffer and an output buffer. When a new byte comes in, it gets stored in the storage buffer first, and then latched to the output buffer. The ‘latching’ step is controlled by a low-to-high rise on the latching pin. Before the latching pin is pulled from low to high, you can update the storage buffer as many times as you want, and these will not be reflected in the output buffer. The shift register will ‘dump’ the storage buffer to output buffer on the low-to-high rise of the latching pin. Hope I’ve explained this clearly.

Not all shift registers have the latching pin. For example, 74HC164 does not. So whatever you are writing to the shift register gets reflected on the output immediately.

– I wrote the microcontroller-based OpenSprinkler firmware, but if you want to know more about the RPi-based version, Dan is the best person to ask. On the microcontroller, everything is serialized, and when the controller is switched from program mode to manual control mode, all stations will reset first. So there is no question of a manual control ‘interrupting’ a program — all stations would have been closed before the manual control mode is turned on.

[snewman]

A big thanks to everyone who has chimed in. I think I’ve found a good software solution that will be ready in another day or so:

I’ve written a simple Python socket server that runs forever and acts as the controller/proxy for modifying valve status. It runs the station program in a thread and keeps a state chart of what valves are running.

If a request for a second operation comes in while a station is operating, the software sends a Queue message to the thread (basically saying, “wrap it up, new work is coming in”) and it stops the current run before starting the next operation. This solves that problem I had about the first job waking up and killing the second job.

I wrestled with doing this as a simple TCP server or as a web server, but decided on TCP. This gave greater flexibility for clients: You are free to write clients in any language that supports TCP sockets (any modern language should) and you aren’t forced to do anything else in Python if you choose not to.

This modular approach also keeps things simple and clean when writing advanced clients, schedulers, etc. If you want to write a pretty advanced web application, the low-level details of the hardware operation are abstracted away.

The other nice thing is that only this TCP server needs to run with superuser privileges.

Here are a couple example snippets that send a command to run station 1 for 2 minutes: (they immediately close after the command is sent, the client doesn’t stay connected during the run)

PHP

$fp = fsockopen("127.0.0.1", 9999, $errno, $errdesc);

fputs($fp, "1,2");

fclose($fp);

?> 

Ruby

require 'socket'



s = TCPSocket.open('localhost', 9999)

s.puts('1,2')

s.close

Python

import socket



sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

sock.connect(('localhost', 9999))

sock.sendall("1,2")

sock.close()

[Author]

Posts