Data logging via MQTT

[pb]

My OpenSprinkler DC controller is mounted in a shed, powered via a solar panel and battery setup. Beside it is an arduino based controller to operate motion sensors/switches etc and includes an wiznet ethernet interface, this controller communicates with my home network via MQTT over ethernet. My plan is to integrate my OS with my MQTT infrastructure for logging purposes, control into OS would still be via the existing API interfaces. I know the status info could be logged by polling the API, but that just seems like a poor solution.

I have had a look around the internet to see how difficult it would be to modify the firmware to integrate MQTT directly, but the general conclusion is MQTT isnt very reliable when using the enc28j60 ethernet controller as used by OpenSprinkler. I remember in the past Ray has mentioned he has modified the enc driver somewhat to improve reliability, perhaps these tweaks would also improve the MQTT implementation?

One option would be to form a serial connection of some sort between the two so the shed controller can act as a MQTT gateway. Looking at the schematic, the primary serial port of the ATMega is connected to the serial-USB chip and out to the USB connector. This could be hijacked and routed out of the enclosure to the shed controller. Of course, when the OS firmware is updated this would need to be disconnected.

Another option would be use the second serial port on the Atmega, but unfortunately the TX pin is used by the flow sensor. Since the flow sensor will soon be connected to a flow meter, this would need to be moved to an alternative interrupt IO pin.

Hardware options:
a. Modify the firmware to directly send MQTT messages
b. Connect the main serial port to the shed controller
c. Connect the second serial port to the shed controller and rewire the rain/flow sensor to another interrupt pin

Software wise, it seems like I could simply add additional code to the current logging mechanism so every time a log entry is written to the SD card a similar message is sent out over MQTT directly or via the the serial port bridge. I would also like to log the weather poll and zimmerman results; these look easy to aadd using the existing logging methods.

Anyone have any comments about these options, or perhaps suggestions about other options? I think I am leaning towards option b, seems like the least amount of work going forward to support future firmware updates.

P.S.:
– I don’t want to use a RPi based OS. I think using a full linux distro is overkill for things like this and just becomes a support problem in waiting as the operating system needs updates to retain security etc. Also, I need the DC version to run easily off my solar/battery system.
– Minimising changes to the OS firmware are desired to make merging changes with future firmware updates easier. This counts against option a. since it woudl require much more wholesale additions to the FW.

[Ray]

I think ENC28J60 generally works fine, as long as you don’t have too many packets to send. The more packets, the higher likelihood of one or two packets getting lost on the way. The main problem with the open-source ENC library is that it’s not a complete implementation of TCP/IP — it doesn’t guarantee delivery of packets. So when necessary you have to manually send duplicate packets, which is a crude way to fix the problem.

I didn’t modify the driver of ENC28J60 to make it more reliable. Instead, I assumed it’s not totally reliable and added higher-level features to handle issues when they happen. For example, when it detects it can’t connect to the router over several tries, it re-initializes the Ethernet controller.

A more modern library for ENC is UIPEthernet, which I think is much more reliable. I’ve planned to replace the current library with UIPEthernet, but that’s likely to exceed the RAM limit on the older OS hardware 2.0 to 2.2 — only the current OS 2.3 (including both AC and DC) will be able to work with this new library. For backward compatibility reasons, I haven’t made the switch, but it will happen at some point.

My suggestion is to directly modify the firmware to support MQTT. However, if this turns out to be too hard, using serial is a good alternative, and probably involves less changes to the firmware code.

[pb]

I agree, an in-built MQTT approach seems best, but difficult with the current firmware. It seems the current implementations of MQTT (e.g npanel.project.free.fr/wp/?p=607) for the enc28j60 use the UIPEthernet library. So I think for now use a serial bridge will be the path of least resistance, even if it is a bit more of a hack.

[Author]

Posts