Updating project to use luarocks

[travishathaway]

Please see my original comment here for a little more context:

• Versioning #20 (comment)

Description

To make this easier to package and distribute, I'm proposing that we update this repository to use a luarocks spec file. This pull request still needs some refine, and I am more than happy to do that. I just want to know whether this would even be considered at all.

Notable changes:

• Adds a new osm2pgsql-themepark-scm-1.rockspec
• Adds a new THEMEPARK_PATH environment variable for finding user defined themes.
• Reorganizes the folder structure to move themes/ into the the lua/ folder so these default themes can appear in the package.

I've left my implementation notes in the .openspec/ folder intact so you can have even more context around how everything works. We can of course remove this if we decide to merge this.

Like I said in the comment above, I have a complete working example of all this in my CLI tool osmprj, so I'm feeling quite confident that this could be a worthwhile change.

More than happy to discuss further, and thank you so much for taking the time to read and consider this change 🙏.

[joto]

This is quite a big change and I haven't digested it all yet. Lets try to split this up into steps.

1. Can you please remove all the docs from the PR and add them to this PR as comments? They are easier to read that way, because Github renders the Markdown and they are not mixed up with what actually changes. If this is merged, we'll need to put some of it on https://osm2pgsql.org/themepark/ , but we can deal with that later.
2. If I understand you correctly we'd end up with one luarocks package for Themepark itself plus extra packages for themes. They are a bit unusual as luarocks packages, because they will not work with a normal Lua binary, they need osm2pgsql. Is there some precedence for that in the Lua community? Is it okay to have Luarocks packages that need more than a normal Lua?
3. I see a lof of changes are just wrapping the Lua files in return function(themepark, theme, cfg)...end. Can you say something about why that's needed/useful?

[travishathaway]

Hi @joto,

Thanks for your feedback. I'm going to hopefully have some time this weekend to go through these items. I do agree that this is large, and I'm going to figure out how we can split this up and verify everything is be done in accordance to Lua best practices.

Stay tuned for more updates 📺 📡 .

[travishathaway]

Here's the requested proposal for the changes that was originally in the pull request changes:

Proposal: LuaRocks Packaging

Summary

Reorganize osm2pgsql-themepark so it can be installed via LuaRocks (luarocks install osm2pgsql-themepark) and used without manually cloning the repository or managing LUA_PATH.

Problem

Currently users must:

1. Clone or download the repository
2. Manually set LUA_PATH=lua/?.lua;; before running osm2pgsql
3. Know to look for themes in the themes/ sibling directory
4. Packaging is also difficult because we cannot use lua specific packaging tool (LuaRocks)

There is no published LuaRocks package, no rockspec, and the theme discovery mechanism relies on a file-system path hack (themepark.dir:gsub('/lua/$', '/themes')) that breaks completely when the library is installed anywhere other than the exact cloned layout.

Solution

1. Move the built-in themes into the Lua module tree (lua/themepark/themes/) so LuaRocks installs them alongside the core library.
2. Switch theme and topic loading from io.open/load() to standard require(), making themes proper Lua modules.
3. Keep a file-system fallback for custom user themes (via add_theme_dir() / THEMEPARK_PATH) for backward compatibility.
4. Add lazy-loading with helpful error messages to the three plugins that depend on optional external libraries.
5. Create an osm2pgsql-themepark-scm-1.rockspec.

Goals

• luarocks install osm2pgsql-themepark installs the full framework and all built-in themes
• require('themepark') works from any install path without manual LUA_PATH setup
• themepark:add_topic('shortbread_v1/streets') API is unchanged
• Custom themes via add_theme_dir() and THEMEPARK_PATH continue to work
• Plugins fail with a clear "install X via luarocks" message when optional deps are missing

Non-Goals

• Packaging themes/explore/ and themes/external/ (contain non-Lua binary/shell files; deferred to separate packages)
• Changing the user-facing config file API
• Publishing to the LuaRocks server (that is a follow-on step)

Success Criteria

• luarocks make osm2pgsql-themepark-scm-1.rockspec succeeds
• require('themepark') works after a luarocks install
• All existing tests pass
• CI passes

[travishathaway]

@joto,

Here are my arguments for points 2 and 3:

2. Why add a .luarocks file?

You bring up a great point about this LuaRock's external dependency on osm2pgsql that we cannot properly define. That's why I wouldn't propose adding this to the LuaRocks registry. I instead see this being distributed in package managers like "brew", "choloatey", "apt" or "conda" that can account for a wide variety of dependencies and not just Lua ones.

Keeping the .luarocks file in this repository and organizing it this way will make it easy for these package managers who wish to package it. Additionally, it also provides a central place for storing all the metadata associated with a project. This also makes packager's lives much easier.

My final argument for adding this file is that it makes it easier for developers or people who want to clone this repository too. Instead of manually fiddling with LUA_PATH, users will simply be able to run luarocks make in the root of the repository to begin using it.

3. What's up with these return statements?

In order to properly answer this, I had to read up on the differences between the varargs style of loading (what's current there) and this proposal's functional style. In varargs loading files implicitly wraps everything in a function, and you then have the ability to simply add arguments that get unpacked with the local arg1, arg2 = ... statement. While this works fine in an embedded context like osm2pgsql, it doesn't work when using the require statement found in luarocks packages because it's not possible to add additional arguments.

You may be asking be asking yourself, "why does this matter when we're highly unlikely to use it outside of an embedded context?" Great question! My answer is that the new functional approach is still compatible within an embedded context and it has a few other benefits:

1. Easier to reason about for non-lua programmers; explicitly defining theme topics as functions is clear to new users that this module is expected to return a callable.
2. Compatible with as a LuaRock in case we ever do decide to distribute them as LuaRocks on that package registry
3. Gotchas around caching; using these functions is actually required to get around the fact that require caches what it returns

There's a couple of other benefits too such as being able to first inpsect and verify the lua file was actually able to be parsed which could help us later with printing friendlier error messages.

I feel like that alone presents a pretty compelling argument in favor of this approach, but I am of course happy to hear your thoughts too. Thanks again for taking time out of your day to review this and hear my thoughts.

[joto]

Now I am confused. I though the point of this exercise was to get the packages into LuaRocks? Now you are talking about Homebrew, apt, conda, etc. Those are all specific to certain operating systems or environments which would lead to a lot of confusion if some packages are here and others are there. And we'd have to convince them all that they should take those packages. That sound a lot more complicated than having them in LuaRocks!?

[travishathaway]

I can understand your confusion. In this case, it's better to think about luarocks as a "build backend". You use it just like you would a pyproject.toml in Python. Just becase a project has a pyproject.toml file associated with it, doesn't mean that it's available or published on pypi.org.

I think the best way to understand all of this is by looking at it from the packager's point of view. Below is a link to how this would be packaged in conda:

• https://github.com/travishathaway/gis-forge/blob/main/osm2pgsql-themepark/recipe.yaml#L21C1-L21C105

Because we have a luarocks file defined, we can use it as a build backend that exactly what you as the code author want any package manger to include. The above is similar to how you would package this for any packaging ecosystem.

But this does depend on the user experience you utlimately want for osm2pgsql-themepark. Right now, I'm making the assumption that this is going to be bundled separately from osm2pgsql because it's seen as a addon. Is that true? Or would you ideally like this to bundled together everywhere with osm2pgsql?

I'm currently the package maintainer for osm2pgsql on conda-forge, so we could experiment with bundling it there.

If I was you though, I would keep this separate for the time being because it's still in beta.

Hope this makes a little more sense now. Sorry for the confusion.

[joto]

At some point Themepark might become a more integral part of osm2pgsql and can be shipped in the same package. But for the time being it is an extra thing.

What we need is a way to publish/install different Thempark themes easily and have them work together without having to configure special Lua paths or whatever. That's what could be interesting about the LuaRocks thing. If we can piggyback onto an existing infrastructure for distributing packages. Ideally you can publish a theme somehow and somebody else can include that theme into their config with a single command. Putting these into Debian or Conda or whatever packages isn't something I would expect will happen, though.

Maybe I am missing something, but the problem I am seeing with going the Luarocks route is that there is no way to publish a Lua package without going through the Luarocks repository. I can't just say: Install that theme from this URL, for instance a Github URL. Not everybody wants to go through the hassle of registering with Luarocks and keeping their theme package up to date. That seems to me to be a drawback of the LuaRocks approach.
But maybe there is a way to do this that I haven't understood yet.

An advantage of using LuaRocks would be that dependencies on other Lua packages are automatically taken into account. So if you have a Lua theme file that needs, say a yaml package for something, it could just say so in the rockspec file and all should magically work.