- This topic has 45 replies, 5 voices, and was last updated 3 years, 6 months ago by
.
-
Posts
-
Due to the discussion in this thread:
https://opensprinkler.com/forums/topic/controller-lockups-crashes/page/2/#post-67243
we have started testing OS 3.2 with W5500 Ethernet module, to address the firmware lockup issue that some users encountered. Here are the instructions:1) If you have OS 3.2 with wired Ethernet module (officially it’s branded as OS 3.0 but only revision 3.2 supports wired Ethernet module), and if you have experienced issues with the controller locking up after some time of use, you may want to give this a try. The symptoms are that the controller may lose response after hours or days of use; a reboot temporarily fixes the issue, but it may reoccur after some time of use. The post above explains our current understanding of the issue, possible work-around (e.g. using VLAN if your router supports it, or use a secondary router to isolate OS from other devices), and what’s the difference between the two Ethernet modules. This is only relevant if you are using wired Ethernet — it does NOT apply if you use WiFi mode. It also does NOT apply if you have OS 2.3 because it does not have a replaceable Ethernet module.
2) To start, send a support ticket at:
https://support.opensprinkler.com
Please include your order number, your shipping address, and say that you want to request a W5500 pin adapter.3) The pin adapter does NOT include W5500 module itself (because we don’t currently have stock of them). You can purchase it on Amazon, or eBay or other stores of your choice. Here is a sample link:
https://www.amazon.com/ARCELI-Ethernet-Network-Hardware-Microcontroller/dp/B07JLFN3T1
it has 2×5 pin headers. The purpose of the pin adapter is to convert W5500 module’s pin ordering to be compatible with ENC28J60, so that you can plug it into OS 3.2.*** IMPORTANT: in the steps below, whenever you remove or insert Ethernet modules, always power off the controller first. Do NOT remove or insert Ethernet module while the controller is alive (i.e. NO HOT PLUG please) ***
4) You will need to flash a new firmware to your controller. This is a custom firmware compiled specifically to support W5500:
http://raysfiles.com/os_compiled_firmware/v3.0/experimental/
Follow the firmware update guide here:
https://openthings.freshdesk.com/support/solutions/articles/5000832310
Note that firmware update must be done through WiFi (i.e. you will need to power off the controller first, remove Ethernet module, then power it back on so it boots into WiFi mode). But it can be done either in WiFi Access Point (AP) mode, or station mode — in AP mode, you do NOT need another WiFi router, the controller itself can launch an access point, which you can connect your computer / laptop to. In station mode, the controller is connected to an existing WiFi router. You can upload firmware in either mode.The experimental firmware does NOT support ENC28J60. If at any point you want to switch back to your original (ENC28J60) Ethernet module, you can just re-upload the official release of the firmware:
http://raysfiles.com/os_compiled_firmware/v3.0/5) When you are ready, power off the controller first, then plug in the pin adapter to W5500 through the female headers. Next, plug the OS 3.2 Ethernet connector cable to the male headers — the orientation is such that the red line on the connector cable should be on the side of the pin adapter that says ‘W5500’. Below is a picture that shows the connection:
https://opensprinkler.com/wp-content/uploads/2020/07/IMG_20200709_092951.jpg
Power the controller back on. If the controller successfully makes connection, then you are all set.6) I’ve designed a new 3D printed enclosure for W5500 with the pin adapter. The 3D file is attached below. Here is the Tinkercad share link in case you need to make modifications. The sample enclosure was printed on Prusa MK3S with PLA material. You can print it vertically with the T-shaped end at the bottom, and no support structure is needed that way. You may need to adjust the dimensions slightly if you use a different printer or material.
https://www.tinkercad.com/things/aNnhAbKHMhG-copy-of-w5500/edit?sharecode=GpubfowzR2M6eI4Hm6t1G9My4Oq-Z2yqmii7PbL6KBM7) Since we have just started testing W5500, we don’t yet know what issues to expect. If you encounter any technical issue, please be patient and report the problems here.
The source code for the customer firmware is currently available in this github branch:
https://github.com/OpenSprinkler/OpenSprinkler-Firmware/tree/os32/219_4_w5500
The changes are pretty minor, mainly changing UIPEthernet library to Ethernet library.Hey Ray,
I installed the firmware for w5500, got the controller turned on, network sees it. The Script to run stations you provided me finds the controller and can run stations. However, I can not get into the app. This is on a pc as well as my the iphone. The app shows connect in green on the Manage site page. I can load the page on my pc, input the password and it will briefly say “loading” and then pop back to enter device password page, can’t get passed that. I can still load the app and operate the controller while in WiFi mode.
I use port forwarding, I changed the static IP for the controller to see if the MAC address needed to be changed or something, but my thinking is that it wouldn’t of found the page in the first place if that’s the case. Any thoughts?
Open a browser on your PC, and open the console — for example, in Chrome, go to Menu -> More Tools -> Developer tool, then click to open the Console tab. Then type in your controller’s IP address (and :port if you use a port other than 80). The console should show error messages.
Okay did the console as you said, below is the error report it comes up with after entering password and then submitting.
DevTools failed to load SourceMap: Could not load content for chrome-extension://hdokiejnpimakedhajhdlcegeplioahd/sourcemaps/onloadwff.js.map: HTTP error: status code 404, net::ERR_UNKNOWN_URL_SCHEME
VM17 onloadwff.js:71Uncaught TypeError: Cannot read property ‘type’ of undefined
at setFieldValue (VM17 onloadwff.js:71)
at HTMLFormElement.formKeydownListener (VM17 onloadwff.js:71)
setFieldValue @ VM17 onloadwff.js:71
formKeydownListener @ VM17 onloadwff.js:71Navigated to http://192.168.1.1:82/ <<<<<<< (NOT THE REAL IP ADDRESS…I just modified it here)
Some googling says the Uncaught TypeError is related to LassPass Extension on chrome. I disabled, used a different browser, went into private mode and with all three methods the error goes away, but the results the same, can not get passed log in screen. Also, have same issue in OS iphone app so I assume the LassPass error is unrelated.
It sounds like the controller is alive but there is a corruption in the returned json data. To verify, you can try:
http://x.x.x.x/ja?pw=yyyyy
where x.x.x.x is your OpenSprinkler’s IP address (with :port if you use a port different than 80), yyyyy is the md5 hash of your device password. You can use https://www.md5hashgenerator.com/ to generate the hash from your password.The return should be a long json string. You can paste it to:
http://json.parser.online.fr/
to verify if there is anything invalid.If something is corrupted you will most likely have to do a factory reset (though, if you know how to use HTTP API, you can issue an HTTP API command to just reset whichever option is corrupted)
Hi Ray,
Okay just spent some time trying to work through this. Here is what I found.
I reset the controller to factory default. From this point I can get into the controller App on PC/iphone just fine with Ethernet module plugged in. I uploaded my configuration file on to the controller and from then on I can only access the controller while in WiFi mode. In Ethernet mode, I can not get in, same issue as I stated above.
So I went back to factory default and decided to rebuild my configuration manually instead of uploading it. I did this while using the Ethernet mode. Got all my stations named (56 of them) and about 12 programs re entered when the controller locked me out again. At this point on I could not get into the App any more from PC/iphone while using Ethernet Module. I went back into WiFi mode and I was able to get into the App as I should be. Tried going into Ethernet mode a few times, no luck.
TO SOLVE: While in WiFi mode, I deleted about 4 programs that I had just entered, to shrink my configuration file size down. This worked. Went back into Ethernet mode and it lets me log in fine again. This reminds me of the buffer overflow issue you addressed with me a few months back, where you had to increase the buffer size to correct. So while in Ethernet mode, the configuration file is too large???? Hopefully this is enough info for you to find a the solution to this as I believe I am stuck now…
No this can’t be the buffer overflow issue, because the buffer in this firmware is large enough to fit the maximum number of programs and maximum number of zones. I will check later this evening. Since the firmware was prepared in a hurry, I didn’t check what happens when the configuration is very large. But it should be easy to debug now that I know at least the symptom.
Okay thanks, let me know if you want me to try anything. I sent you my config file awhile back, it hasn’t changed much, so if its useful for testing you likely have it. If you’d like I can send it to you.
Sorry for the delay getting back to you with the results of my W5500 upgrade. I just got back in town and am now working with the new ethernet module. I updated the OS controller firmware and got the new module plugged in, and it booted up fine. Apparently you are using a software defined MAC address (because the MAC address is the same as it was for the older ethernet module)?
For the first couple of minutes after rebooting I wasn’t able to connect to OS through either the app or the browser, but then it started connecting and everything seems to be working now. I did not have to restore my configuration from the backup… everything seemed to survive the firmware update. I did notice that it wasn’t logging my 1 minute manually started test runs (and all of my previous log entries were gone), so I cleared the log (via the GUI option to do so) and after that it seems to be logging my test runs.
At this point I need to start running a bunch of manually started zones and see if it crashes like it did when using the original ethernet module. I will provide feedback as my tests progress. Fingers crossed!
1) Mac address: neither the ENC28J60 nor W5500 chip has hardware MAC, so yes it needs to be software defined. The Mac is derived from ESP8266’s Mac, with only the last byte differing from each other:
https://github.com/OpenSprinkler/OpenSprinkler-Firmware/blob/master/OpenSprinkler.cpp#L442
so it’s not totally made up from nowhere, but it’s based on the true Mac of ESP8266.2) Configurations: the firmware only triggers a factory reset if upgrading across major revisions (like from 2.1.9 to 2.2.0). Within the same major revision, the configurations are preserved, because flash structure and all file formats are the same within the same major revision so there is no need to reset flash.
3) John reported issues when the configuration is large (i.e. large number of programs and/or zones). I haven’t had time to look at it yet but will import some large configuration file to check. Meantime, if you can see the homepage that means it’s fine. I would suggest not adding more programs or zones until I figure out the issue.
For the mac address, I’ve always used a static IP in the Unifi software. When I switched over to the W5500, the controller connected to the network right away, no issue. I only thought to try changing because I was having some odd behavior. I thought the mac address was coming from the ethernet module, and so would require going into Unifi to re assign an address, but I guess not? Regardless, this didn’t prove to be an issue, just a curiosity for me.
I think I just have an overly large irrigation system and so I seem to be running into issues most others wont see. Why I am locked out when I am using the ethernet module, I don’t have the technical background to know, but I figured out what triggered it for me (posted above) and so hopefully Ray will have an easy time spotting the problem. Unfortunately, I’m pretty sure this is unrelated to the main issue I’m dealing with (freezing controller). Since you are up and running, hopefully you will see improvements soon. I think I would have to delete half of my configurations (programs) to get back in, in order to start manually testing. Will wait to see what Ray finds.
I have have been testing the new W5500 module and firmware since Monday and unfortunately the OpenSprinkler has stop responding twice.
When the controller stops responding, I cannot access it via the Android App or the Web interface. However, it still responds to pings and the buttons on the controller still respond. It also continues to run watering programs correctly. This is consistent with previous reports.
After the first lockup, I switched from DHCP to a static IP address and after a day it stopped responding again.
In the other thread regarding lockups, Ray suspects Dropbox may be one cause of the issue.
I also have a computer running Dropbox, so I have turned off the “Enable LAN sync” setting this morning and will watch for another lock up.
If it happens again with this setting off, I will deactivate Dropbox all together and report back.Finally, I have a handful of IOT devices on my network, Ring doorbell, Hue lightbulbs, Amazon echo, Sonos, etc. If Dropbox can cause this issue, I wouldn’t be surprised if another one of these devices could also cause a similar issue. If there is an easy way to check the network register values (ESTAT.BUFFER and EIR.RXERIF), I’d be willing to try to narrow it down.
I have managed to run in excess of 30 manual zones since installing the W5500 and it hasn’t crashed yet. With the original ethernet module it would have had a pretty high likelihood of crashing at least once by now, but until I’ve run for a few more days without lockups I’m not quite ready to say that the new module has fixed the problem, because once in a while the original module would make it a few days without locking up.
Most of the prior lockups resulted in the on-screen clock freezing and the buttons being unresponsive. It would usually still respond to pings, but any zone I had manually started would still be running if the lockup occurred while the zone was on. I think this may be similar to what John was seeing too.
What Bena describes above actually (mostly) reflects what I was seeing while I was using the WiFi interface for the last week or so. The majority of the times I tried to connect to the controller (using either the iOS app or the web interface) it would time out. It would often take upwards of 15 – 20 minutes of trying before I could get connected. What was different from what Bena reported is that pings would also time out. The controller’s on-screen display reported a strong WiFi signal, and it apparently had no trouble syncing NTP. Several times when I was unable to connect I would run a continuous ping (once every 2 seconds) from my iMac, and after some number (usually 150 – 300) of timed out pings it would start responding quite reliably to the pings and at that point I was able to connect from the app or browser right away. I never had the controller crash while on WiFi, but it was devilishly difficult to connect to it most of the times I tried. Several times I tried cycling power when I was unable to connect, and it would reboot without errors and sync the time, but that didn’t seem to help me get connected.
After a few more days of testing the new W5500 module and OS firmware, I continue to get lock ups. As mentioned above, I disabled “LAN Sync” in Dropbox and then tried disabling Dropbox all together. Both times the OpenSprinkler would stop responding after about a day.
I suspect some other device on my network is interfering. At this point I’ll revert back to WiFi.
@bena: it would help if you can provide some details about the symptoms of the lockup, for example:
– does the controller respond to button clicks?
– is the time displayed on the lcd correct? if not, how far is it from the current time?
– does the controller respond to ping?@John K: I figured out why the Json data is incomplete when you have a large number of programs — apparently by default the Arduino Ethernet library (which supports W5500) limits each ‘send’ to a maximum of 2048 characters — even though OpenSprinkler firmware’s send buffer is 8192, the library caps the number of characters to send each time to 2048. As a result, when you have a large number of programs, it’s being truncated so the Json data is incomplete (therefore the homepage can’t render correctly). UIPEthernet (which supports ENC28J60) does not have this limit. Gotta love how different libraries make different assumptions. Anyways, it’s an easy fix and I will update the firmware shortly to increase this limit and that should address the issue.
When the controller stops responding, I cannot access it via the Android App or the Web interface.
It does continue to run programs and water correctly.– does the controller respond to button clicks?
Yes– is the time displayed on the lcd correct? if not, how far is it from the current time?
Yes, the time is correct.– does the controller respond to ping?
Yes@John K: I’ve updated the firmware — it’s at the same folder, and I added today’s time stamp so we know this is the most recent version:
http://raysfiles.com/os_compiled_firmware/v3.0/experimental/
The Ethernet library’swritefunction is technically incomplete: it’s supposed to be able to send a buffer of any size, but apparently it only sends the first 2048 bytes of it, resulting in incomplete Json data. I’ve raised this issue in their Github library:
https://github.com/arduino-libraries/Ethernet/issues/141
and I’ve fixed it by adding a loop until the entire buffer has been sent out. I’ve tested the new firmware with 20 programs and it works fine so far.
@bena: given your description, the controller is at least not freezing, it just stops responding to web request. There are several possibilities, one is if you have a relatively large number of zones and/or programs, maybe the issue you encountered is the same as what John K reported. An easy way to tell is to export your configurations to a file and open it to check how many characters it has. If it’s larger than 2048, it’s very likely the same issue, and just flash the Jul18 version of the w5500 firmware. That should fix it.Yes, my configuration export is 2264 characters, so that could be it.
I’ve upgraded to the Jul 18 version of the w5500 firmware and I’ll continue to monitor.Just a quick update. My OpenSprinkler stopped responding again this morning even after the Jul 18th w5500 firmware.
Looking back, I don’t think my symptoms are similar to John K as I don’t get a password prompt or any response when accessing it through a browser. Instead, it just times out. No errors other than the time out show up in the browser developer console.
My symptoms seem to be closer to the original issue where the ENC28J60 module is impacted by some other device on my network.
Again, if there is an easy way to check the network register values (ESTAT.BUFFER and EIR.RXERIF), I’d be happy to try to pinpoint what device on my network is interfering.
@bena: the network register values (ESTAT.BUFFER and EIR.RXERIF) are only for ENC28J60. These don’t exist for W5500, which uses hardware TCP/IP stack. I don’t know what’s the root cause of the issue with W5500 since so far I haven’t heard of hanging issue with W5500 from the other users.
One thing you can try is when the hanging happens, open a browser and type in:
http://x.x.x.x/ja
or
http://x.x.x.x/su
where x.x.x.x is your OpenSprinklers IP. If you use a port that’s not 80, you also need to explicitly specify the port number, such as x.x.x.x:port_numberSee if you get a response. If so, that means the Ethernet controller is still working, the issue is somewhere else.
Ah, ok. That makes sense.
I just ran a test while the OpenSprinkler was not responding and both the ja and su command time out in my browser.
However, it continues to respond to pings.I’ll just revert back to wifi and wait to see if anyone else has this issue with the W5500.
I have now been running for several days without any lockups / crashes, and there’s no way I would have made it this long with the original Ethernet module, so the W5500 / new firmware version seems to have resolved my frequent crashing problem! I will continue to monitor for problems and update this forum if anything changes, but right now I’m feeling pretty positive about the W5500.
The other thing I’ve noticed is that I no longer get periodic timeouts when I run continuous pings. This doesn’t surprise me, because the hardware TCP/IP handling should be pretty close to bulletproof when it comes to seeing and responding to pings, but it’s yet another sign that the new module has improved Ethernet communication reliability.
- You must be logged in to reply to this topic.