Plugin installation – opinions please

[Dan in CA]

Hi All,

Until now there is not a standard procedure for installing plugins for the Python Interval Program. I have been giving it some thought and want to get feedback from the community on a possible method.

It needs to be as easy as possible for both plugin authors and users who just want to add a plugin to their installation.

Python has some features for creating program installers but they are more complex than what we need.

In the simplest case, a Python file just needs to be copied into the plugins directory under the program directory and the group permissions need to be set to executable. With more complex plugins there are also other files that need to be put in different locations.

I am considering the use of Makeself:
http://megastep.org/makeself/

It creates a self extracting archive that puts the extracted files into a temporary directory and can include an installer script that automatically copies the files into the necessary locations.

Plugin authors would just need to install Makeself on their system and the creation of downloadable archives looks to be very easy.

Users would just need to download the archive for the desired plugin into the OSPi directory and type the name of the file to install the plugin.

It should also be easy for a plugin author to include an “uninstall” script with their plugin.

Comments and suggestions are welcome.

Dan

[rederikus]

Dan, I am very far from being a programmer. I used to use Dartmouth BASIC (line numbers) a very long time ago. I am more a hardware guy (even that I learned on tubes…). So from a user standpoint it would be nice to have a verifiable installation that is as easy as possible to accomplish. Software installs, upgrades and removals are never easy and they often affect other stuff that the user never thought about.

Firstly is the question of upgrading OSPi itself. Could that be made easier? Could you put a button in the options screen that said. “Check for Updates?” Even better would be one that said “Update available, do you want to install it?” See what I mean?

With regard to Plugins, there were some installed when I opened up 2.04. One was a thing called “Monthly Irrigatin Adjustments” (Copy/Paste from the screen 😀 ). when I ran it, it said “Note: In order for this plugin to actually work, The apscheduler Python module must be installed.
Visit the Wiki for instructions.”. I followed the Wiki and it did the things it said and I got no errors. I also got no changes displayed on-screen when in the Spinkler webpage. Did something install correctly or, did it fail? How would I know as a user?

I believe that it would be nice if somehow a user could see just what has been installed and, if it is going to work. I have no idea if this is possible but, I sure would like to know that my system is working as I intend it.

Please understand that I am not complaining. I think that OSPi is about the best piece of hardware and software I have bought in a long time. I have had lots of fun building it making it work. It has amazed my neighbors and gotten me at least one free beer so, I am a very happy bunny indeed. I guess it really depends on where you see this product going.

Thanks for a great product and your support.

[KanyonKris]

My first thoughts were along the lines rederikus mentioned. Since there is a web interface, seems the user could access a list of plugins, click on one and have it installed (and also uninstalled). Having some programming experience I believe this would take a fair amount of work to implement, but I think this would be a good goal to shoot for.

In the short-term I like makeself. While it takes some manual steps I doubt many users will be going nuts installing many plugins so the occasional manual process isn’t a huge barrier – folks had to do a bunch of steps to get the RasPi setup and interval running.

[rederikus]

Well, maybe not offer a list of available plugins but at least give a way to show that a particular plugin has actually err, plugged in when a neophyte user has installed it. My original was perhaps a little obscure in that the plugin was already installed and it told me that I needed to install another module before it would work. I installed the module (I think). The OSPi software gave me no indication as to whether:
– the needed module had already been installed before I started. This would have been sensible since the plugin needs the module to work.
– it was working (or not) when I installed it according to the published Linux instructions in the linked Wiki.
– I broke something by installing a later version of the module that the plugin could not use.

You get my point?

[Samer]

I am not a fan of APscheduler and another user has contributed a solution that doesn’t depend on it. Hopefully it gets merged in and we can avoid the dependency issue in the future.

Problem with APScheduler is 3 different versions and the method used by these plugins requires version 2 (not interoperable). Very poor documentation for the plugin as well in my opinion.

[scottsh]

First, great points from all – it would be good if we could make it easier for folks to install from the web rather than having to manually run an installer from the shell.

It would be interesting if the Interval program could offer the ability to pull the makeself install package from a specified URL, then run it.

As a plug-in developer, I don’t know anything about makeself but a quick read makes it looks pretty straight forward. I can imagine making the complete package and placing it up on my web server for people to download pretty easily.

Scott

[rederikus]

For me the important part is that the system report what is currently installed. That way, months later I can still see what is on my system even if (highly likely) that I have forgotten.

Oh btw, the Monthly Adjust plugin is working on my system even though it still says I need to install “Note: In order for this plugin to actually work, The apscheduler Python module must be installed.”

I did this and waited for 1st August (+10%) to come around and, yay, it did, BUT it still says that I need to install it to make it work.

[KanyonKris]

Thinking about this some more, and discussing it with a software engineer at work, a different approach:

All plugins hosted in Github repository with interval program and downloaded during install and updated with git pulls. Instead of the interval program loading all Python files found in the plugins directory, it checks a configuration file and only loads the plugins listed in that file. Only a few critical/needed/basic plugins loaded upon new install.

Plugins web page has link to a list of all plugins where user can select new plugins to use or to disable currently active plugins.

Changes to plugins would be committed and approved by Dan, or Dan can add collaborators for plugins.

I’ll admit it seems weird to have all the plugins downloaded, but I doubt there will be that many plugins and the file sizes are small. Since the interval program is already hosted on Github, makes sense to me to use that tool for the plugins too.

[Dan in CA]

Thanks for the great feedback so far.

Based on your comments here are my thoughts at this point.

Using GitHub as a repository for the plugins was my first thought, but due to the varied nature of possible plugins and some restrictions of GitHub, e.g. not being able to upload individual files:
https://github.com/blog/1302-goodbye-uploads

That does not appear to be a viable option.

Including plugins in the program download has a couple of issues:

1. Avoiding “bloat”.
It would be good to keep the program download and updates as small and fast as possible. Not everyone who might want to use the program is guaranteed to have a fast or reliable internet connection.

2. Security:
I for one would prefer to have more control over what gets downloaded and installed on my system. Allowing users to pick and choose the plugins that get downloaded to their Pi would be preferable.

I agree that managing plugins should be part of the program’s UI and not require users to ssh into the OS to make changes.

A Plugins page that would list all the installed plugins on the Pi is doable. It could include a check box next to each plugin name so the individual plugins could be enabled / disabled or even removed from the system. Currently plugins must be enabled by setting file permission and the program could change permissions without requiring a config file.

I think it might be good to have something like a wiki page + plugin repository. Not part of the program but a separate space on the web that could include descriptions and documentation for plugins so users could browse for plugins of interest. A list of plugin names in the UI would not provide all the information a user would want about a plugin. Having a separate wiki would allow plugin developers to keep their information updated without requiring users to constantly pull updates to their local systems. The plugins themselves could be hosted at various locations and links provided on the wiki.

The plugins page in the UI could include links to descriptions and documentation for installed plugins. There could also be a link to the plugins wiki and a form field for downloading and installing new plugins.

The program is evolving and your input will guide further development.

Thanks.

Dan

[Samer]

I wouldn’t rule out version control (Github) yet. I think using a Github repository with a folder for each plugin is ideal for development. Also, Github has an API and has methods to grab raw files. I think you can place an XML document in the root of each plugin folder that has info on the plugin and the OSPi will just connect to repo and get folder list (aka plugin list) thats available. The user sees this picks the ones they want and OSPi just copies that portion of the repo into the plugins dir and does chmod a+x to the directory and reboot OSPi. This allows easy plugin install and easy development and management. No one has to worry about making packages, uploading them etc.

Ultimately, you will merge each request to the repo. Me, you and Ray can use an organization repo to manage the plugins (making the pull request process easier). Each developer would then submit their plugin via a pull request to a new directory in the main repo.

[KanyonKris]

Dan, your points are valid. There’s a lot to consider when designing this plugin system. Perhaps a hybrid approach that uses Git for some things, wiki for others and functions added to the interval program to install, configure and delete plugins?

salbahra, good details about how Git can be used for plugins (I’m a Git noob).

One thing that makes these plugins tricky is the HTML pages, they have to go in the templates directory so you can’t just have one subdirectory under the plugins directory (ie plugins/autowater2000). The makeself tool can handle this with an auto-executed script, as Dan pointed out. But it’s trickier for version control systems, hence salbahra’s suggested workarounds.

Regarding this point, I suggest a naming convention for plugins so it is clear what files belong to what plugin. For example, if the plugin is SuperWater, all of its HTML files in the templates directory should use that name, ie SuperWater-config.html, SuperWater-status.html). In fact, perhaps they should be: “plugin-SuperWater-config.html” so it’s easy to know which HTML files are for the core interval program and which are for plugins. A standard naming convention would also make it easier to delete a plugin via the web interface and core interval code functions.

[Samer]

KanyonKris,

I think the filename makes 100% sense! In regard to the Github approach, the XML file will define which files go where (and they can have generic layout such as subdir in main plugin folder: templates, plugin, etc) and the installer can move them when installed (from the menu).

[Dan in CA]

There is also the possibility that a plugin would require a non-standard Python module e.g. for machine learning. A script in a makeself archive could handle installing the required module. I don’t see how that could be done with GitHub.

There are some plugins under development that are designed to work with additional hardware such as an LCD display. They needs extra documentation. How would that be handled?

Don’t get me wrong, I would like to use GitHub but just don’t see how.

Dan

[KanyonKris]

Dan, more good points.

So is it better to have a plugin system that can handle everything but is more complicated, or a simpler system that makes it easy to install simple plugins (Python and HTML pages)?

I believe most plugins will be just Python code and HTML pages so I favor a simple plugin manager. And there’s no reason we couldn’t also have a wiki or other system for the more involved plugins. Just throwing out ideas.

[Samer]

The idea behind the XML/JSON is you can add things like modules. You basically build a small installer in OSPi that reads the info and does what’s needed. For example:

OSPi-Plugins <--- Main repo

weather_adj/   <---top dir, represents plugin

templates/   <--- subdirectory containing template files (the installer can auto prefix them to prevent collision)

modules/   <--- subdirectory containing modules to be installed (the installer can check if already installed and skip)

source/   <--- subdirectory containing python code (installer will move this to the plugin directory with the plugin name)

weather_adj.py  <--- main plugin code

package.json   <---- holds info represented below

That would be the basic layout and the package.json would have the info for OSPi to use like this:

{

"name": "Weather Adjust",

"version": "1.0.0",

"description": "Allows weather based rain delay activation.",

"author": "Samer Albahra"

"modules": "apscheduler"

}

The modules can be just a name and the installer would grab them or you can add a folder and manually install them as suggested above. I think in the long run this system will be easier to find and use plugins.

Update: I already see ways to clean that up so of course this isn’t a final implementation by anyways but an example to represent what I had in mind.

[martinpihrt]

Hi all,
intercede for updates from git (with git can not handle yet), but I somehow wrote a plugin for sms, pressure, email, and lcd display. Currently doing the update so that i will send sms with command update (sms system receives and updates from the git dan, that I always current system – fixed at a distance). I would also like to see the web interface of a button to check for updates. When you click on the key can be previewed window changes or corrections, and then I clicked on the update.

For example, for my sms plugin need another system tools (gammu python). For lcd display (LCD library for I2C pylcd.py) …

I would be happy if they were somewhere plugins (dan git?) centrally.

They would not bother me as a user to download all the plugins always with updates.

I would welcome the plugins menu a button that would plugin permitted or prohibited, without having to manually change the permissions. Thank you all for a great job on OSPi. Sorry for my English

[Author]

Posts