Discussion:
Contrib 2.0
(too old to reply)
thron7
2013-06-03 15:09:14 UTC
Permalink
concerns: Developers using a qooxdoo Git master checkout and
contributions with their apps

Hi all,

I've just pushed the new version of the tool chain that works with the
new qooxdoo contrib 2.0 catalog repo.

I've also added appropriate entries for existing contribs from
Sourceforge into the catalog. Ideally, you shouldn't realize any
difference after pulling the changes from master, all your apps should
compile as before. If this is not the case, please get back.

For the entries imported into the new catalog, this only concerns
contribs that already *had* versions with existing Manifest.json files
in them. Contribs and versions where this was missing were skipped.

If you are a contrib maintainer you might want to check the entries for
your contribution in the new catalog repo [1]. If you already maintain
your contribution *outside* of Sourceforge's SVN, you might want to add
corresponding Manifest.json files. I've added a few for testing so you
can peek at e.g. the QxGC/master [2] or Dialog/master [3] contribs for
reference.

You can send in catalog changes, or use what will become the standard
way of modification, fork the catalog repo on Github and send pull requests.

Cheers,
Thomas

[1] https://github.com/qooxdoo/contrib-catalog
[2]
https://github.com/qooxdoo/contrib-catalog/blob/master/QxGC/master/Manifest.json
[3]
https://github.com/qooxdoo/contrib-catalog/blob/master/Dialog/master/Manifest.json
thron7
2013-06-06 10:36:45 UTC
Permalink
I've just pushed (#9b475cd) changes to the tool chain to support http(s)
URLs as arguments to library/manifest keys. This means you can now write

"libraries" : {
"library" : [
{ "manifest" :
"http://server.tld/path/to/my/contrib/vers/Manifest.json" }, // {1}
{ "manifest" :
"http://server.tld/path/to/my/othercontrib/vers/archive.zip" }, // {2}
...

Mind, though, that the URLs you enter are not just arbitrary. These two
examples compries the entire scope of possible entries, you have

{1} a URL ending in a Json file (.json extension)
{2} a URL ending in an archive file (.zip, .tar, .tar.gz, .tar.bz2
extensions)

In case {1} the Json file must be a catalog manifest file that has the
usual contents and semantics, particularly an info/download and
info/checksum key. The URL path itself needs to reflect the contrib's
name and version in its last two directory elements ("contrib" and
"vers" in the example).

In case {2} the archive is downloaded and extracted directly, again
expecting the contents as if linked from a catalog entry (single
top-level directory, which contains the library's Manifest.json etc.).
Again, contribution name and version are taken from the URL path, so
make sure there is something sensible. Also, as there is currently no
freshness check for these direct archive downloads, the download happens
*every time* you build your app. So going through a catalog entry has
advantages.

Particularly, this also means you cannot just point to the source tree
of a contribution even if web-accessible, as there just isn't a reliable
way of downloading a directory tree recursively over HTTP. (The
well-known exception is spidering from svn.code.sf.net, to support the
existing contributions there. So there is actually a {3}'d type of
supported URL, which I don't want to popularize at this point so left it
out).

T.
Post by thron7
concerns: Developers using a qooxdoo Git master checkout and
contributions with their apps
Hi all,
I've just pushed the new version of the tool chain that works with the
new qooxdoo contrib 2.0 catalog repo.
I've also added appropriate entries for existing contribs from
Sourceforge into the catalog. Ideally, you shouldn't realize any
difference after pulling the changes from master, all your apps should
compile as before. If this is not the case, please get back.
For the entries imported into the new catalog, this only concerns
contribs that already *had* versions with existing Manifest.json files
in them. Contribs and versions where this was missing were skipped.
If you are a contrib maintainer you might want to check the entries for
your contribution in the new catalog repo [1]. If you already maintain
your contribution *outside* of Sourceforge's SVN, you might want to add
corresponding Manifest.json files. I've added a few for testing so you
can peek at e.g. the QxGC/master [2] or Dialog/master [3] contribs for
reference.
You can send in catalog changes, or use what will become the standard
way of modification, fork the catalog repo on Github and send pull requests.
Cheers,
Thomas
[1] https://github.com/qooxdoo/contrib-catalog
[2]
https://github.com/qooxdoo/contrib-catalog/blob/master/QxGC/master/Manifest.json
[3]
https://github.com/qooxdoo/contrib-catalog/blob/master/Dialog/master/Manifest.json
------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite
It's a free troubleshooting tool designed for production
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://p.sf.net/sfu/appdyn_d2d_ap2
_______________________________________________
qooxdoo-devel mailing list
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
John Spackman
2013-06-07 07:45:46 UTC
Permalink
Hi Thomas

I've been taking a look at getting my contribs on the new format, just a
couple of questions:

(1) When the contrib-catalog's Manifest/info/download key points to a
github archive .zip (eg Qx-GC), it's really cool that github does the work
of generating the .zip file automatically, but does that mean that the
generator will not accept the .zip until I've updated the contrib-catalog
Manifest/info/checksum to match?

(2) I know it's a bit late to be asking for features because this has been
up for discussion for so long, but could you add a redirect feature to the
Manifest that tells the generator to download the "real" manifest from
some other URL? EG that would let me maintain the one Manifest.json
inside each of my contribs and mean that I don't have to issue pull
requests each time the Qooxdoo version number bumps. A lot of the info
seems duplicated between the contrib-catalog Manifest.json and my
contrib.'s Manifest.json (description, qoodoo-versions, etc)

John
Post by thron7
I've just pushed (#9b475cd) changes to the tool chain to support http(s)
URLs as arguments to library/manifest keys. This means you can now write
"libraries" : {
"library" : [
"http://server.tld/path/to/my/contrib/vers/Manifest.json" }, // {1}
"http://server.tld/path/to/my/othercontrib/vers/archive.zip" }, // {2}
...
Mind, though, that the URLs you enter are not just arbitrary. These two
examples compries the entire scope of possible entries, you have
{1} a URL ending in a Json file (.json extension)
{2} a URL ending in an archive file (.zip, .tar, .tar.gz, .tar.bz2
extensions)
In case {1} the Json file must be a catalog manifest file that has the
usual contents and semantics, particularly an info/download and
info/checksum key. The URL path itself needs to reflect the contrib's
name and version in its last two directory elements ("contrib" and
"vers" in the example).
In case {2} the archive is downloaded and extracted directly, again
expecting the contents as if linked from a catalog entry (single
top-level directory, which contains the library's Manifest.json etc.).
Again, contribution name and version are taken from the URL path, so
make sure there is something sensible. Also, as there is currently no
freshness check for these direct archive downloads, the download happens
*every time* you build your app. So going through a catalog entry has
advantages.
Particularly, this also means you cannot just point to the source tree
of a contribution even if web-accessible, as there just isn't a reliable
way of downloading a directory tree recursively over HTTP. (The
well-known exception is spidering from svn.code.sf.net, to support the
existing contributions there. So there is actually a {3}'d type of
supported URL, which I don't want to popularize at this point so left it
out).
T.
Post by thron7
concerns: Developers using a qooxdoo Git master checkout and
contributions with their apps
Hi all,
I've just pushed the new version of the tool chain that works with the
new qooxdoo contrib 2.0 catalog repo.
I've also added appropriate entries for existing contribs from
Sourceforge into the catalog. Ideally, you shouldn't realize any
difference after pulling the changes from master, all your apps should
compile as before. If this is not the case, please get back.
For the entries imported into the new catalog, this only concerns
contribs that already *had* versions with existing Manifest.json files
in them. Contribs and versions where this was missing were skipped.
If you are a contrib maintainer you might want to check the entries for
your contribution in the new catalog repo [1]. If you already maintain
your contribution *outside* of Sourceforge's SVN, you might want to add
corresponding Manifest.json files. I've added a few for testing so you
can peek at e.g. the QxGC/master [2] or Dialog/master [3] contribs for
reference.
You can send in catalog changes, or use what will become the standard
way of modification, fork the catalog repo on Github and send pull
requests.
Cheers,
Thomas
[1] https://github.com/qooxdoo/contrib-catalog
[2]
https://github.com/qooxdoo/contrib-catalog/blob/master/QxGC/master/Manife
st.json
[3]
https://github.com/qooxdoo/contrib-catalog/blob/master/Dialog/master/Mani
fest.json
-------------------------------------------------------------------------
-----
Get 100% visibility into Java/.NET code with AppDynamics Lite
It's a free troubleshooting tool designed for production
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://p.sf.net/sfu/appdyn_d2d_ap2
_______________________________________________
qooxdoo-devel mailing list
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
--------------------------------------------------------------------------
----
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
qooxdoo-devel mailing list
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
thron7
2013-06-07 08:52:50 UTC
Permalink
Post by John Spackman
Hi Thomas
I've been taking a look at getting my contribs on the new format, just a
(1) When the contrib-catalog's Manifest/info/download key points to a
github archive .zip (eg Qx-GC), it's really cool that github does the work
of generating the .zip file automatically, but does that mean that the
generator will not accept the .zip until I've updated the contrib-catalog
Manifest/info/checksum to match?
If the Manifest/info/checksum is non-empty and has been downloaded
before, the Generator will not realize that the archive has actually
changed and will re-use the cached version. The user will stick with the
previous version until he runs a generate.py distclean.
Post by John Spackman
(2) I know it's a bit late to be asking for features because this has been
up for discussion for so long, but could you add a redirect feature to the
Manifest that tells the generator to download the "real" manifest from
some other URL?
Another layer of indirection, eh ;-).
Post by John Spackman
EG that would let me maintain the one Manifest.json
inside each of my contribs and mean that I don't have to issue pull
requests each time the Qooxdoo version number bumps.
Which wouldn't be too often, would it?! But I get your point.

But for one thing, you cannot possibly maintain the checksum of the
archive in a file *that is part of* the archive. With every update of
the local Manifest file, even just changing a single character, the
checksum of the archive changes.

Another reason would be that it makes it easier for *us* to eyeball the
catalog entries, to ensure some sanity.
Post by John Spackman
A lot of the info
seems duplicated between the contrib-catalog Manifest.json and my
contrib.'s Manifest.json (description, qoodoo-versions, etc)
That's totally by intention, to make it easy for authors to just copy
over their contrib's Manifest, with as little modification as possible.

T.
John Spackman
2013-06-07 10:28:53 UTC
Permalink
Post by thron7
Post by John Spackman
(1) When the contrib-catalog's Manifest/info/download key points to a
github archive .zip (eg Qx-GC), it's really cool that github does the
work
of generating the .zip file automatically, but does that mean that the
generator will not accept the .zip until I've updated the
contrib-catalog
Manifest/info/checksum to match?
If the Manifest/info/checksum is non-empty and has been downloaded
before, the Generator will not realize that the archive has actually
changed and will re-use the cached version. The user will stick with the
previous version until he runs a generate.py distclean.
OK, so if the checksum's blank then it only downloads after a distclean?
Post by thron7
Post by John Spackman
(2) I know it's a bit late to be asking for features because this has
been
up for discussion for so long, but could you add a redirect feature to
the
Manifest that tells the generator to download the "real" manifest from
some other URL?
Another layer of indirection, eh ;-).
Post by John Spackman
EG that would let me maintain the one Manifest.json
inside each of my contribs and mean that I don't have to issue pull
requests each time the Qooxdoo version number bumps.
Which wouldn't be too often, would it?! But I get your point.
But for one thing, you cannot possibly maintain the checksum of the
archive in a file *that is part of* the archive. With every update of
the local Manifest file, even just changing a single character, the
checksum of the archive changes.
Ah - yes of course, not the correct solution then!

I'd really like to get away from having to issue pull requests and having
a separate build process for releasing contribs - AIUI, if I make a minor
release I need to commit my changes, push it, download the archive .zip,
calculate the sha1, copy my contrib.'s Manifest.json to conrib-catalog,
edit it to add my sha1, commit it, push it, then submit a pull request to
you so that everyone else can see it.

That sounds like a lot of work, seven or eight steps just in order to make
a minor release and could be easily screwed up in a non-obvious way. Some
of it can be scripted but I'm not sure about scripting github pull
requests.

Also, looking at the commit history for my qx-serverobjects, I've made 6
commits in the past month - wouldn't that mean six pull requests for just
one contrib?
Post by thron7
Another reason would be that it makes it easier for *us* to eyeball the
catalog entries, to ensure some sanity.
Post by John Spackman
A lot of the info
seems duplicated between the contrib-catalog Manifest.json and my
contrib.'s Manifest.json (description, qoodoo-versions, etc)
That's totally by intention, to make it easy for authors to just copy
over their contrib's Manifest, with as little modification as possible.
OK, ISWYM although I see that as a bit of a disadvantage - having to keep
99% as a duplicate sounds like an opportunity to get things out of sync
and not notice!

John
thron7
2013-06-07 13:46:03 UTC
Permalink
Post by John Spackman
Post by thron7
Post by John Spackman
(1) When the contrib-catalog's Manifest/info/download key points to a
github archive .zip (eg Qx-GC), it's really cool that github does the
work
of generating the .zip file automatically, but does that mean that the
generator will not accept the .zip until I've updated the
contrib-catalog
Manifest/info/checksum to match?
If the Manifest/info/checksum is non-empty and has been downloaded
before, the Generator will not realize that the archive has actually
changed and will re-use the cached version. The user will stick with the
previous version until he runs a generate.py distclean.
OK, so if the checksum's blank then it only downloads after a distclean?
No, what I wrote concerns a non-empty checksum. If the checksum is
empty, the download will happen with every build run of the Generator,
as the checksum cannot be used to check freshness.
Post by John Spackman
Post by thron7
Post by John Spackman
(2) I know it's a bit late to be asking for features because this has
been
up for discussion for so long, but could you add a redirect feature to
the
Manifest that tells the generator to download the "real" manifest from
some other URL?
Another layer of indirection, eh ;-).
Post by John Spackman
EG that would let me maintain the one Manifest.json
inside each of my contribs and mean that I don't have to issue pull
requests each time the Qooxdoo version number bumps.
Which wouldn't be too often, would it?! But I get your point.
But for one thing, you cannot possibly maintain the checksum of the
archive in a file *that is part of* the archive. With every update of
the local Manifest file, even just changing a single character, the
checksum of the archive changes.
Ah - yes of course, not the correct solution then!
I'd really like to get away from having to issue pull requests and having
a separate build process for releasing contribs
So which alternative do you see to maintaining the catalog? Having a
wiki page and editing entries manually, with no connection to the
technical processes (like application build runs)?!
Post by John Spackman
- AIUI, if I make a minor
release I need to commit my changes, push it, download the archive .zip,
calculate the sha1, copy my contrib.'s Manifest.json to conrib-catalog,
edit it to add my sha1, commit it, push it, then submit a pull request to
you so that everyone else can see it.
For one thing, I do many pushes to Github per day, in more than one
repository. Pushing to Github is so slight it has become a no-brainer.
The Github approach was chosen for its ease, and also popularity. If you
happen to never work with Github *except* for the contrib catalog, I
understand it feels alien to you, especially as it would then occur only
infrequently.

We also had ideas about a web-editing interface to the catalog, but that
didn't make it forward so far.
Post by John Spackman
That sounds like a lot of work, seven or eight steps just in order to make
a minor release and could be easily screwed up in a non-obvious way. Some
of it can be scripted but I'm not sure about scripting github pull
requests.
I guess it would require maybe a couple of minutes if done routinely. I
wouldn't consider this "a lot of work", at least not in terms of
duration. But maybe you're more concerned about complexity, or recalling
details after some time.

* The initial commit/push actions in your list you do anyway, as they
are part of maintaining your source code, so this sort of doesn't
count into maintaining the catalog.
* Downloading the .zip archive is like creating the .zip archive for
people that don't use a hoster that provides this feature. So at
least there is no special burden here.
* Calculating the sha1 is tedious, I agree. We could offer a Generator
job to do that, so you don't have to concern yourself with extra
command line tools (which might differ across OSs). This would also
ensure everybody uses the same hash algorithm. You might want to
open an issue for that.
* Entering a string in a file, commit, push, pull - these are so
trivial and straight forward, esp. when you have the manual open to
recap the steps. If we had the web-editor interface, you would need
to fire up the editor web page in a browser, edit download location
and checksum, hit the Save button - these are also about three steps
to perform. To be honest I wouldn't feel very comfortable putting
effort in optimizing the first sequence.
Post by John Spackman
Also, looking at the commit history for my qx-serverobjects, I've made 6
commits in the past month - wouldn't that mean six pull requests for just
one contrib?
If each of the commits was a self-contained improvement of your contrib
you wished to publish - then yes. But what's the issue?! Pull requests
are fire-and-forget, it's like sending an email.
Post by John Spackman
Post by thron7
Another reason would be that it makes it easier for *us* to eyeball the
catalog entries, to ensure some sanity.
Post by John Spackman
A lot of the info
seems duplicated between the contrib-catalog Manifest.json and my
contrib.'s Manifest.json (description, qoodoo-versions, etc)
That's totally by intention, to make it easy for authors to just copy
over their contrib's Manifest, with as little modification as possible.
OK, ISWYM although I see that as a bit of a disadvantage - having to keep
99% as a duplicate sounds like an opportunity to get things out of sync
and not notice!
Mh, yes, but I'm sure there are ways around that. The Git log of your
catalog repo will always tell you if and when you have updated the
catalog's Manifest.json.

HTH,
T.
John Spackman
2013-06-07 13:08:10 UTC
Permalink
How about this - build the.zip manually and add it to Github as an
ordinary repo file. After creating the .zip, update the
Manifest.json/info/checksum with the SHA of the .zip, and then commit and
push the Manifest and .zip in one go.

Importantly this is easily scriptable - [1] is my first pass hack at a
script to automate the process, the .zip to be referred to by
contrib-catalog is at [2]

If the contrib-catalog refers to the Manifest in the individual contrib,
then contrib releases are not held up until the core team approve the new
release.

John

[1] https://github.com/johnspackman/qx-gc/blob/master/build.sh
[2]
https://github.com/johnspackman/qx-gc/blob/master/dist/dist.zip?raw=true
Post by John Spackman
Post by thron7
Post by John Spackman
(1) When the contrib-catalog's Manifest/info/download key points to a
github archive .zip (eg Qx-GC), it's really cool that github does the
work
of generating the .zip file automatically, but does that mean that the
generator will not accept the .zip until I've updated the
contrib-catalog
Manifest/info/checksum to match?
If the Manifest/info/checksum is non-empty and has been downloaded
before, the Generator will not realize that the archive has actually
changed and will re-use the cached version. The user will stick with the
previous version until he runs a generate.py distclean.
OK, so if the checksum's blank then it only downloads after a distclean?
Post by thron7
Post by John Spackman
(2) I know it's a bit late to be asking for features because this has
been
up for discussion for so long, but could you add a redirect feature to
the
Manifest that tells the generator to download the "real" manifest from
some other URL?
Another layer of indirection, eh ;-).
Post by John Spackman
EG that would let me maintain the one Manifest.json
inside each of my contribs and mean that I don't have to issue pull
requests each time the Qooxdoo version number bumps.
Which wouldn't be too often, would it?! But I get your point.
But for one thing, you cannot possibly maintain the checksum of the
archive in a file *that is part of* the archive. With every update of
the local Manifest file, even just changing a single character, the
checksum of the archive changes.
Ah - yes of course, not the correct solution then!
I'd really like to get away from having to issue pull requests and having
a separate build process for releasing contribs - AIUI, if I make a minor
release I need to commit my changes, push it, download the archive .zip,
calculate the sha1, copy my contrib.'s Manifest.json to conrib-catalog,
edit it to add my sha1, commit it, push it, then submit a pull request to
you so that everyone else can see it.
That sounds like a lot of work, seven or eight steps just in order to make
a minor release and could be easily screwed up in a non-obvious way. Some
of it can be scripted but I'm not sure about scripting github pull
requests.
Also, looking at the commit history for my qx-serverobjects, I've made 6
commits in the past month - wouldn't that mean six pull requests for just
one contrib?
Post by thron7
Another reason would be that it makes it easier for *us* to eyeball the
catalog entries, to ensure some sanity.
Post by John Spackman
A lot of the info
seems duplicated between the contrib-catalog Manifest.json and my
contrib.'s Manifest.json (description, qoodoo-versions, etc)
That's totally by intention, to make it easy for authors to just copy
over their contrib's Manifest, with as little modification as possible.
OK, ISWYM although I see that as a bit of a disadvantage - having to keep
99% as a duplicate sounds like an opportunity to get things out of sync
and not notice!
John
--------------------------------------------------------------------------
----
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
qooxdoo-devel mailing list
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
thron7
2013-06-07 14:31:39 UTC
Permalink
Post by John Spackman
How about this - build the.zip manually and add it to Github as an
ordinary repo file. After creating the .zip, update the
Manifest.json/info/checksum with the SHA of the .zip, and then commit and
push the Manifest and .zip in one go.
Importantly this is easily scriptable - [1] is my first pass hack at a
script to automate the process, the .zip to be referred to by
contrib-catalog is at [2]
Sure, if that is the way you prefer it. You would then only need to do
an exact copy of the Manifest.json to the contribs' catalog entry, and
you're almost done.

But also mind that you only have a master branch in this contrib, and
your approach is no ideal solution for people maintaining multiple
versions of the same contribution, and want to use Git branches to do
so. Your recipe would require them to create an archive *in every branch
of their repo*, adding redundant size to it, and forgoing the
opportunities Github is offering here.
Post by John Spackman
If the contrib-catalog refers to the Manifest in the individual contrib,
then contrib releases are not held up until the core team approve the new
release.
I'm not sure we are holding up much, and there is no question of
approving if we know the contributor. I'm sure you can tweak your little
script to also copy the Manifest.json over to your local catalog repo,
and do a git commit/push there. Then all there is left is the pull
request, and this is a matter of seconds if you have your Github page
open in the browser.

(If you have users of your contribution that are desperate to deploy a
new version within minutes or hours, they can always work around a
pending catalog update by using a direct 'http' URL in their config.)

You understandably take the author's side, but my primary concern are
unsuspecting users of contributions. If they are using "contrib://" URLs
I want to give them a minimum of confidence that what they get is solid.

T.
John Spackman
2013-06-07 17:15:05 UTC
Permalink
Hi Thomas,

Can you quickly clarify whether the generator will pull the catalog from
the master at github or will it take the version of the catalog as it
stood when the qooxdoo release was made? I'm assuming the former!
Post by thron7
Post by thron7
For one thing, I do many pushes to Github per day, in more than one
repository. Pushing to Github is so slight it has become a no-brainer.
The Github approach was chosen for its ease, and also popularity. If you
happen to never work with Github *except* for the contrib catalog, I
understand it feels alien to you, especially as it would then occur only
infrequently.
That's the point isn't? That commits to git are frequent. Contrib 2.0
requires me to get you to approve my commits before they're publicly
available, and there might be a lot of them. Git's not alien to me, I
also run a private Gitlab repo with half a dozen projects on it.
Post by thron7
Post by thron7
We also had ideas about a web-editing interface to the catalog, but that
didn't make it forward so far.
I don't want a web editor interface either ­ that would be no better
Post by thron7
Post by thron7
Post by John Spackman
That sounds like a lot of work, seven or eight steps just in order to
make a minor release and could be easily screwed up in a non-obvious
way.
I guess it would require maybe a couple of minutes if done routinely. I
wouldn't consider this "a lot of work", at least not in terms of
duration. But maybe you're more concerned about complexity, or recalling
details after some time.
Well, for the past 5 years releasing has been a matter of hitting "commit"
and job's done ­ that was perfectly acceptable before but now there's a
whole series of steps (including getting you to do something) and I don't
see the benefit. It's just added complication for no value. The only
value I can see is that spidering html-decorated file lists is no longer
necessary.

The more complicated contrib:// packaging would suggest that it's a good
idea to test the packaging - but I can't even do that without waiting for
you to commit my pull request. You guys might be on holiday (you have
different bank holidays). It might be the weekend. And when you've done
it, I'll probably be busy working on something else and testing will go on
the "TODO" list.

If the fix is in response to a user I "could" explain that they can
manually checkout, edit their config, etc but I have to ask myself whether
it's worth risking them coming back asking how to use git, or having some
other problem not related to my contrib.
Post by thron7
Post by thron7
If each of the commits was a self-contained improvement of your contrib
you wished to publish - then yes. But what's the issue?! Pull requests
are fire-and-forget, it's like sending an email.
Everyone makes mistakes - in the past I've submitted a pull request that
was mistaken as a duplicate for someone else fix and my pull request was
closed. Later, I had to find and debug the same problem and create a new
pull request.
Post by thron7
Post by thron7
Mh, yes, but I'm sure there are ways around that. The Git log of your
catalog repo will always tell you if and when you have updated the
catalog's Manifest.json.
Just because it's possible to track what happened doesn't help prevent
bugs from happening in the first place (and doesn't give you back the time
that you spend figuring it out). A mistake in the SHA will be difficult
to spot and will prevent the generator from using the new version of the
contrib, without the author or the user from being aware that they are
using an out of date version.
Post by thron7
Post by thron7
How about this - build the.zip manually and add it to Github as an
ordinary repo file. After creating the .zip, update the
Manifest.json/info/checksum with the SHA of the .zip, and then commit
and
push the Manifest and .zip in one go.
Importantly this is easily scriptable - [1] is my first pass hack at a
script to automate the process, the .zip to be referred to by
contrib-catalog is at [2]
Sure, if that is the way you prefer it. You would then only need to do
an exact copy of the Manifest.json to the contribs' catalog entry, and
you're almost done.
But also mind that you only have a master branch in this contrib, and
your approach is no ideal solution for people maintaining multiple
versions of the same contribution, and want to use Git branches to do
so. Your recipe would require them to create an archive *in every branch
of their repo*, adding redundant size to it, and forgoing the
opportunities Github is offering here.
Yes, I agree it's far from ideal. IMHO the best solution would be for the
generator to checkout from a github url and not a .zip at all but you
wanted to make it .zip archives. Alternatively, don't require that the
SHA of a .zip to be put in the catalog in order for it to work.

I'm not sure that what you're suggesting helps people maintaining multiple
versions either because contrib users are always going to be pulling the
catalog from master, which would be pulling from the master of the contrib
which might depend on the current master of Qooxdoo - even though the user
is using a non-master version that's potentially incompatible.

Maintaining multiple versions can be complicated, that's life - but IMHO
the difference is that with the catalog, it's obfuscated. For example, if
there is a problem, the question will never be as simple as "what contrib
url are you using" it's what contrib url, what's the sha, does that sha
relate to the archive we think it does, etc
Post by thron7
Post by thron7
(If you have users of your contribution that are desperate to deploy a
new version within minutes or hours, they can always work around a
pending catalog update by using a direct 'http' URL in their config.)
You understandably take the author's side, but my primary concern are
unsuspecting users of contributions. If they are using "contrib://" URLs
I want to give them a minimum of confidence that what they get is solid.
I don't think you're protecting the contrib users at all - AIUI you're not
going to test what I submit, and in any case they're still going to use
whatever I put in the .zip ... it's only the listing that you're policing.

Being focused on the users is automatic (or much more likely) if you're
focused on the authors. We make contribs in addition to our jobs and
businesses - not because of them. You only have to look at the Contrib
demo browser to see how easy it is to fall behind on getting updates out,
so I think it's a positive thing to take the author's sides in this case.

John
thron7
2013-06-07 20:58:00 UTC
Permalink
John,
Post by John Spackman
Can you quickly clarify whether the generator will pull the catalog from
the master at github or will it take the version of the catalog as it
stood when the qooxdoo release was made? I'm assuming the former!

The Generator does not "pull" in the Git-sense, it's just downloading
Manifest files over HTTP. It will always read the most current incarnation
of such a file. That's all unrelated to qooxdoo releases.
Post by John Spackman
Well, for the past 5 years releasing has been a matter of hitting "commit"
and job's done ­ that was perfectly acceptable before but now there's a
whole series of steps (including getting you to do something) and I
don't
Post by John Spackman
see the benefit. It's just added complication for no value. The only
value I can see is that spidering html-decorated file lists is no longer
necessary.

You're overlooking an essential point. So far, the source code of
contributions was openly accessible, all in one place. It was easy to
inspect it, for everybody, core team, users, other contributers. The
entire community was the QA.

Now the interface is to binary archives only. Besides the catalog there is
only a download site. The source code does not even have to be online,
authors could maintain it on their local disk. I don't expect many authors
to follow that model, but it's perfectly workable. So they only way to
scrutinize the contribution is to download the archive, unpack and inspect
it. This is far less easy and convenient than before.

I want to let sleeping dogs lie, but we have to be concerned about the
security of the contrib users, and don't want to see anything like the
jQuery plugin desaster. I don't want villains to discover qooxdoo
contribs as a convenient attack vector.

That being said, nothing is cut into stone. Let's gather some experiences
with the system for a few months, and if then most people agree it's too
much of a hassle we'll reconsider.
Post by John Spackman
The more complicated contrib:// packaging would suggest that it's a good
idea to test the packaging - but I can't even do that without waiting
for
Post by John Spackman
you to commit my pull request.
I'm not so sure. A URL like "contrib://UploadWidget/0.4/Manifest.json" is
just expanded into the less user-friendly
"https://github.com/qooxdoo/contrib-catalog/raw/master/contributions/UploadWidget/0.4/Manifest.json".
But you can use the https URL just as well. So you can test your manifest
by just putting it under a web server you have access to, and use the
corresponding URL. If your contribution works with the test URL, you can
consider the contrib URL covered.
Post by John Spackman
If the fix is in response to a user I "could" explain that they can
manually checkout, edit their config, etc but I have to ask myself
whether
Post by John Spackman
it's worth risking them coming back asking how to use git, or having some
other problem not related to my contrib.
Following your own recipe, you can just pass the public (raw) URL to your
current Manifest.json and ask them to use this until the catalog has been
updated.
Post by John Spackman
Yes, I agree it's far from ideal. IMHO the best solution would be for the
generator to checkout from a github url and not a .zip at all but you
wanted to make it .zip archives.

One constraint for the system is that the Generator doesn't employ VCS
operations *at all*, as this would basically require the users of
contributions to have git installed, or mercurial, or darcs, or svn, or
... whatever contrib authors are using on their side.
Post by John Spackman
Alternatively, don't require that the
SHA of a .zip to be put in the catalog in order for it to work.
You can leave the checksum field empty, but that results in repetitive
downloads with every build run. How would you check freshness without it?!
(Especially when people use the on-demand archive facility of Github,
where probably even an If-Modified-Since check would not work).
Post by John Spackman
I'm not sure that what you're suggesting helps people maintaining multiple
versions either because contrib users are always going to be pulling the
catalog from master,

I was talking about multiple versions of the *contrib*. Have a look at the
above linked UploadWidget catalog entries, to see what I mean. The Git
master branch of the catalog can hold multiple versions of each contrib
(they are just directories, and have nothing to do with Git branches).
Post by John Spackman
which would be pulling from the master of the contrib
Not at all. E.g. you can download any branch of a contribution as an
archive from Github.
Post by John Spackman
which might depend on the current master of Qooxdoo - even though the user
is using a non-master version that's potentially incompatible.
Well, the user must of course choose a version of the contribution that is
compatible with the qooxdoo version he or she is using.
Post by John Spackman
Maintaining multiple versions can be complicated, that's life - but IMHO
the difference is that with the catalog, it's obfuscated.

This I don't get?! The catalog makes contrib versions absolutely explicit.
Post by John Spackman
I don't think you're protecting the contrib users at all - AIUI you're not
going to test what I submit, and in any case they're still going to use
whatever I put in the .zip ... it's only the listing that you're
policing.

Indeed, it's a weak protection, and that really concerns me. If some of
the well-known authors like you is issuing a pull request, we will in most
cases just wave it through. But if some unknown guy shows up with a new
contrib, no homepage, but a download URL on some obscure Asian server,
then we might ask back.

We could go an entirely different way, e.g. like Golang: Abandon
during-build downloads of contribs, making that a separate Generator job
("generate.py contrib-install http://.....zip") and leave it to the users'
sole discretion when to download which version of which contribution, to
make his builds work. Add an incomplete and partly outdated link list of
"known" contributions on the web, and you have what Golang has. The user
has to find the download URLs, has to decide whether he can trust the
source, trigger the download manually, and keep up with updates. - That's
a possible model, and we would be off the hook :-).
Post by John Spackman
Being focused on the users is automatic (or much more likely) if you're
focused on the authors. We make contribs in addition to our jobs and
businesses - not because of them. You only have to look at the Contrib
demo browser to see how easy it is to fall behind on getting updates
out,
Post by John Spackman
so I think it's a positive thing to take the author's sides in this case.
I hear you, but would want to hear more voices from the community,
especially from contrib *users*, to get a better picture. So far, very few
have voiced their opinion, or replied to my comments when they did. You
might want to revise the discussion in the comments of the "Blueprint"
post [1].

T.

[1] http://news.qooxdoo.org/qooxdoo-contrib-2-0-the-blueprint
John Spackman
2013-06-11 09:55:32 UTC
Permalink
Hi Thomas
Post by John Spackman
Post by John Spackman
Can you quickly clarify whether the generator will pull the catalog from
the master at github or will it take the version of the catalog as it
stood when the qooxdoo release was made? I'm assuming the former!
The Generator does not "pull" in the Git-sense, it's just downloading
Manifest files over HTTP. It will always read the most current incarnation
of such a file. That's all unrelated to qooxdoo releases.
Sure, I understood - "pull" was a poor choice of words on my part
Post by John Spackman
Post by John Spackman
Well, for the past 5 years releasing has been a matter of hitting
"commit"
Post by John Spackman
and job's done ­ that was perfectly acceptable before but now there's a
whole series of steps (including getting you to do something) and I
don't
Post by John Spackman
see the benefit. It's just added complication for no value. The only
value I can see is that spidering html-decorated file lists is no longer
necessary.
You're overlooking an essential point. So far, the source code of
contributions was openly accessible, all in one place. It was easy to
inspect it, for everybody, core team, users, other contributers. The
entire community was the QA.
Now the interface is to binary archives only. Besides the catalog there is
only a download site. The source code does not even have to be online,
authors could maintain it on their local disk. I don't expect many authors
to follow that model, but it's perfectly workable. So they only way to
scrutinize the contribution is to download the archive, unpack and inspect
it. This is far less easy and convenient than before.
I want to let sleeping dogs lie, but we have to be concerned about the
security of the contrib users, and don't want to see anything like the
jQuery plugin desaster. I don't want villains to discover qooxdoo
contribs as a convenient attack vector.
You make it sound as though there is going to be some kind of auditing for
new contributors - obviously I see the point of avoiding jQuery-like
problems but I haven't heard much about this process so I'm a little
skeptical. Apologies if you've got this arranged already but IMHO that
raises questions like:

(o) Who's going to inspect new contributions and contrib updates from
contributors not automatically "green-light"-ed?
(o) At what point do contributors get considered "green light"-able?
(o) Who decides that? Will we be voting or is it a 1&1 decision?
(o) Are you sure you've got the time to add that to your schedule?
(o) Is this process what the community at large would call "open" as in
"open source"?
Post by John Spackman
Post by John Spackman
The more complicated contrib:// packaging would suggest that it's a good
idea to test the packaging - but I can't even do that without waiting
for
Post by John Spackman
you to commit my pull request.
I'm not so sure. A URL like "contrib://UploadWidget/0.4/Manifest.json" is
just expanded into the less user-friendly
"https://github.com/qooxdoo/contrib-catalog/raw/master/contributions/Uploa
dWidget/0.4/Manifest.json".
But you can use the https URL just as well. So you can test your manifest
by just putting it under a web server you have access to, and use the
corresponding URL. If your contribution works with the test URL, you can
consider the contrib URL covered.
AIUI that won't work because the SHA in the master repo won't be right so
the generator will ignore it?
Post by John Spackman
Post by John Spackman
If the fix is in response to a user I "could" explain that they can
manually checkout, edit their config, etc but I have to ask myself
whether
Post by John Spackman
it's worth risking them coming back asking how to use git, or having
some
Post by John Spackman
other problem not related to my contrib.
Following your own recipe, you can just pass the public (raw) URL to your
current Manifest.json and ask them to use this until the catalog has been
updated.
But unless we also put a .zip under VCS (which we both think is a pretty
poor solution for lots of reasons) this won't work because the SHA will be
incorrect, and if I edit my Manifest.json to have the SHA of the .zip then
the .zip will change and the SHA will be incorrect again.
Post by John Spackman
Post by John Spackman
Yes, I agree it's far from ideal. IMHO the best solution would be for
the
Post by John Spackman
generator to checkout from a github url and not a .zip at all but you
wanted to make it .zip archives.
One constraint for the system is that the Generator doesn't employ VCS
operations *at all*, as this would basically require the users of
contributions to have git installed, or mercurial, or darcs, or svn, or
... whatever contrib authors are using on their side.
Hm, yes ISWYM - that is a really good idea - requiring people to manually
install other packages is off putting to say the least; I think it was a
great leap forward when cygwin was no longer required for Qooxdoo for
instance.
Post by John Spackman
Post by John Spackman
Alternatively, don't require that the
SHA of a .zip to be put in the catalog in order for it to work.
You can leave the checksum field empty, but that results in repetitive
downloads with every build run. How would you check freshness without it?!
(Especially when people use the on-demand archive facility of Github,
where probably even an If-Modified-Since check would not work).
I've checked and unfortunately Github doesn't support If-Modified-Since
for the auto-generated archive .zip, though it does support it for
individual files, e.g. Manifest.json.

What would happen if a user has a version of my contrib, then I issue a
pull request, you merge it, and then I push another change to Github - and
only then the user runs a new build? Will the user's generator see that
the SHA in Manifest.json has changed, try to download the archive.zip, see
that the SHA does not match the current archive.zip and then abandon it?
Next time the generator is run would it try and download again?

Doesn't this means that even minor documentation changes will require you
to merge a pull request from me?
Post by John Spackman
Post by John Spackman
Maintaining multiple versions can be complicated, that's life - but IMHO
the difference is that with the catalog, it's obfuscated.
This I don't get?! The catalog makes contrib versions absolutely explicit.
I think that's obfuscated because it's all about the name of the correct
branch in my contrib being in your master with the correct SHA for the
current version of my branch. If you, me, or the user get any of that
wrong (or you're not around to merge a pull request) then the generator
will ignore my contrib.

EG to use Aristo on Qooxdoo 2 you can get it from aristo/master, and to
use Aristo with the Qooxdoo master branch (now that diet is merged with
master) contrib:// urls must refer to
qooxdoo/master/contrib-catalog/aristo/diet; once you release Qooxdoo 3,
I'll merge aristo/diet into aristo/master and (after you merge my pull
request) contrib users only get a compatible version if they use the new
qooxdoo/master/contrib-catalog/aristo/master contrib:// url.

If Qooxdoo 2.x was already using this mechanism, I cannot merge
Aristo/diet into Aristo/master to match Qooxdoo because that would break
current users, even though they're not using the (not yet released)
qooxdoo master.

After Qooxdoo 3.0 is released, users on Qx2.x (including brand new users
who decide to take the safe option and not try out the only-just-released
3.0) will need to know to use some other contrib:// url e.g.
qooxdoo/master/contrib-catalog/aristo/release_2_x. Other contribs might
call their Qx2.0-compatible branch something else, e.g. "0.5"

(the paths in my examples are not correct, for illustration only).

So users have to explicitly choose the VCS version they need from the
master version of qooxdoo, regardless of whether the branch name of their
contrib is the same. Or because there are no rules on branch names the
branch name could be the same but be the wrong one!

Also, if I maintain an old version against Qooxdoo v2.1 and I checkout
release_2_1, all of the contribs are taken from master even if they're not
compatible and the release_2_1 branch refers is correct.

It sounds like you're versioning the versioning, and I wonder if a big
underlying cause is because GitHub doesn't support If-Modified-Since in
it's auto archive, forcing you to use a SHA elsewhere in the VCS.


AIUI if anything goes wrong or I accidentally push a commit (e.g.
documentation typo) without running my script or hand editing a series of
steps, the generator will just ignore my updates.

I think that this will make experienced users just go straight for an
explicit checkout. As a contrib author, it may also be one of my first
recommendations if someone has a problem: ditch the contrib:// url and
checkout from git, because doing so will eliminate a load of potential
problems.

John
thron7
2013-06-11 13:10:20 UTC
Permalink
Post by John Spackman
Post by thron7
I want to let sleeping dogs lie, but we have to be concerned about the
security of the contrib users, and don't want to see anything like the
jQuery plugin desaster. I don't want villains to discover qooxdoo
contribs as a convenient attack vector.
You make it sound as though there is going to be some kind of auditing for
new contributors - obviously I see the point of avoiding jQuery-like
problems but I haven't heard much about this process so I'm a little
skeptical.
Well, "auditing" is a big word. I believe it will be substantially
different than what we did so far when granting people write access to
the qooxdoo-contrib SVN repository.
Post by John Spackman
Apologies if you've got this arranged already but IMHO that
(o) Who's going to inspect new contributions and contrib updates from
contributors not automatically "green-light"-ed?
I think it will be on the level of contributors, not individual
contributions.
Post by John Spackman
(o) At what point do contributors get considered "green light"-able?
I not sure what the procedure was so far, but it could be that Andreas
got in touch with each person individually before granting "contributor
status".
Post by John Spackman
(o) Who decides that? Will we be voting or is it a 1&1 decision?
Good question. I think as long as we will maintain the contrib catalog
it will be the qooxdoo commiters (Mind that this group is not
exclusively 1&1 employees, although we're the majority). But there is a
train of thought in the core team to release qooxdoo-contrib even more
into the hands of the community. E.g. I could see a point where the
authoritative catalog repo is no longer in the "qooxdoo" Github account,
but in some other, 1&1-unrelated, like the "qooxdoo-contrib" account
Derrell created a while ago. And then community members would decide who
gets added as maintainers of this repo, and those would in turn oversee
adding new contributions.
Post by John Spackman
(o) Are you sure you've got the time to add that to your schedule?
As I said, we've managed so far. Which doesn't mean we aren't thinking
of alternative ways.
Post by John Spackman
(o) Is this process what the community at large would call "open" as in
"open source"?
Nice p-question ;-). - Yes, I do think it complies with open source
principles, as "open source" means, well, the source is open, not
everybody is allowed to change things arbitrarily. You have this in
other OSS projects as well.
Post by John Spackman
Post by thron7
I'm not so sure. A URL like "contrib://UploadWidget/0.4/Manifest.json" is
just expanded into the less user-friendly
"https://github.com/qooxdoo/contrib-catalog/raw/master/contributions/Uploa
dWidget/0.4/Manifest.json".
But you can use the https URL just as well. So you can test your manifest
by just putting it under a web server you have access to, and use the
corresponding URL. If your contribution works with the test URL, you can
consider the contrib URL covered.
AIUI that won't work because the SHA in the master repo won't be right so
the generator will ignore it?
Here I was talking about the catalog Manifest (not the library
Manifest). All I was saying was, once you have this you can test it,
independent of its presence in the catalog.
Post by John Spackman
Post by thron7
Following your own recipe, you can just pass the public (raw) URL to your
current Manifest.json and ask them to use this until the catalog has been
updated.
But unless we also put a .zip under VCS (which we both think is a pretty
poor solution for lots of reasons) this won't work because the SHA will be
incorrect, and if I edit my Manifest.json to have the SHA of the .zip then
the .zip will change and the SHA will be incorrect again.
I was trying to pick up on your proposal. Leaving that behind, you can
still get around a pending catalog update (and I thought that was your
original concern). You either pass a URL to the archive directly, or you
pass the URL to the catalog-style Manifest (ie. the Manifest that will
be in the catalog) that you make available *somewhere*.
Post by John Spackman
Post by thron7
Post by John Spackman
Alternatively, don't require that the
SHA of a .zip to be put in the catalog in order for it to work.
You can leave the checksum field empty, but that results in repetitive
downloads with every build run. How would you check freshness without it?!
(Especially when people use the on-demand archive facility of Github,
where probably even an If-Modified-Since check would not work).
I've checked and unfortunately Github doesn't support If-Modified-Since
for the auto-generated archive .zip, though it does support it for
individual files, e.g. Manifest.json.
Yep, unsurprisingly. Thanks for checking.
Post by John Spackman
What would happen if a user has a version of my contrib, then I issue a
pull request, you merge it, and then I push another change to Github - and
only then the user runs a new build?
The checksum and the archive wouldn't match, so the archive would look
compromised.
Post by John Spackman
Will the user's generator see that
the SHA in Manifest.json has changed, try to download the archive.zip, see
that the SHA does not match the current archive.zip and then abandon it?
Yep.
Post by John Spackman
Next time the generator is run would it try and download again?
Good question, probably yes. But that could be avoided.
Post by John Spackman
Doesn't this means that even minor documentation changes will require you
to merge a pull request from me?
Look, the basic idea behind it is that also contributions have sort of
"releases". No big bells or whistles, but the author works for some
time, makes changes, fixes bugs, tests, and once he is confident
"publishes" a new state of a version of his contribution (Think of
qooxdoo publishing a 2.1.2 in the 2.1.x "version"). With "publishing"
meaning updating the catalog entry for this version.

That's sort of nice and clear. If, OTOH, you want something like
continuous deployment where every little change is directly propagated
to the users, it gets tedious.
Post by John Spackman
Post by thron7
Post by John Spackman
Maintaining multiple versions can be complicated, that's life - but IMHO
the difference is that with the catalog, it's obfuscated.
This I don't get?! The catalog makes contrib versions absolutely explicit.
I think that's obfuscated because it's all about the name of the correct
branch in my contrib being in your master with the correct SHA for the
current version of my branch. If you, me, or the user get any of that
wrong (or you're not around to merge a pull request) then the generator
will ignore my contrib.
For one thing, the user has very little to get wrong. He just enters
something like "contrib://UploadWidget/0.4" and be done with it. Not
much opportunity to e.g. get the "0.4" wrong, right?!

The other thing is the author who needs to put a Manifest.json into the
catalog, under UploadWidget/0.4, with the right URL to the archive, and
the right checksum over it. Plenty of opportunity to get things wrong.
You can copy the right manifest into the wrong catalog path. The
manifest can point to the wrong archive. The checksum was made over a
different archive than the URL points to. Phew!

On the other hand it's very flexible. You don't *have* to use Github's
auto-zip feature. Maybe you're on Bitbucket, and using their "Files"
feature. You don't have to map contrib versions to Git branches
(although I know of authors that will do exactly that). You can have
plain old subdirectories within a single repository, within a single
branch, create the archives by hand (read "script"), and upload them to
your favorite file hoster. So no danger that checksums will not comply
with on-the-fly archives. - So there is also plenty of opportunity to
get things on a level of robustness that you feel comfortable with.
Post by John Spackman
EG to use Aristo on Qooxdoo 2 you can get it from aristo/master, and to
use Aristo with the Qooxdoo master branch (now that diet is merged with
master) contrib:// urls must refer to
qooxdoo/master/contrib-catalog/aristo/diet; once you release Qooxdoo 3,
I'll merge aristo/diet into aristo/master and (after you merge my pull
request) contrib users only get a compatible version if they use the new
qooxdoo/master/contrib-catalog/aristo/master contrib:// url.
For one thing there is no contribution "aristo/master", but only
"Aristo/trunk" (mind capitalisation and version name). But let's just
pretend it was named "Aristo/master". What you should do is create
versions of the Aristo contribution, e.g. "Aristo/0.5", which would be
the one compatible with qooxdoo 2.x, and another version, e.g.
"Aristo/0.6", which would be compatible with qooxdoo 3.x. That should be
easy enough to convey to users.

The problem with "master" (or "trunk" or "tip" or the like) versions of
a contribution is that users need to be aware that while using the same
contrib URL all the time, requirements to the underlying qooxdoo version
(or other dependencies) might change. Only users that are prepared to
handle these changes should work with a "master" contrib version.
Post by John Spackman
If Qooxdoo 2.x was already using this mechanism, I cannot merge
Aristo/diet into Aristo/master to match Qooxdoo because that would break
current users, even though they're not using the (not yet released)
qooxdoo master.
Yes, as I tried to explain above it is not very recommendable to only
have a single master version of a contribution which only works with a
(moving) master version of qooxdoo. It's just about setting up sensible
versions of the contribution.
Post by John Spackman
After Qooxdoo 3.0 is released, users on Qx2.x (including brand new users
who decide to take the safe option and not try out the only-just-released
3.0) will need to know to use some other contrib:// url e.g.
qooxdoo/master/contrib-catalog/aristo/release_2_x.
As far as referencing is concerned there is no difference between users
of qx 3.x and qx 2.x. Both use e.g. "contrib://Aristo/trunk". But users
of qx 2.x will only use whatever is available on the Sourceforge SVN,
which is completely independent of what's in your Github account, which
is what qx 3.x users will use (provided you change the catalog entry to
point to that).
Post by John Spackman
Other contribs might
call their Qx2.0-compatible branch something else, e.g. "0.5"
Yes, and I don't see a problem with that.
Post by John Spackman
(the paths in my examples are not correct, for illustration only).
So users have to explicitly choose the VCS version they need from the
master version of qooxdoo, regardless of whether the branch name of their
contrib is the same. Or because there are no rules on branch names the
branch name could be the same but be the wrong one!
Mh, this is getting out of hand. The qooxdoo master version is not
involved at all, I'm only considering users of released qooxdoo
versions. 3.0 SDK users will use the catalog, older versions will go the
Sourceforge SVN directly, both will use whatever is available through
their method of resolution.

Version names for contribution versions are chooseable by the contrib
authors, and should reflect milestones of the contribution itself, not
of qooxdoo. Compatibility with qooxdoo versions is documented in each
version's Manifest.json, this is what contrib users wil want to look at
when picking a particular contribution version. (And this will become
more obvious when the planned web interface to the catalog goes online).

T.
thron7
2013-06-11 13:32:28 UTC
Permalink
Post by thron7
Post by John Spackman
Post by thron7
I want to let sleeping dogs lie, but we have to be concerned about the
security of the contrib users, and don't want to see anything like the
jQuery plugin desaster. I don't want villains to discover qooxdoo
contribs as a convenient attack vector.
You make it sound as though there is going to be some kind of auditing for
new contributors - obviously I see the point of avoiding jQuery-like
problems but I haven't heard much about this process so I'm a little
skeptical.
Well, "auditing" is a big word. I believe it will be substantially
Should read "not be substantially different" - sorry.
Post by thron7
different than what we did so far when granting people write access to
the qooxdoo-contrib SVN repository.
John Spackman
2013-06-11 17:48:04 UTC
Permalink
Post by John Spackman
You make it sound as though there is going to be some kind of auditing
for
new contributors
Well, "auditing" is a big word. I believe it will [not] be substantially
different than what we did so far
So no policing of content then == no protection of users from making us
add SHAs as pull requests. SHAs exist purely so that the generator can
try and avoid downloading.
Post by John Spackman
What would happen if a user has a version of my contrib, then I issue a
pull request, you merge it, and then I push another change to Github -
and
only then the user runs a new build?
The checksum and the archive wouldn't match, so the archive would look
compromised.
So the generator would...(a) ignore the contrib, (b) refuse to compile.
Doesn't matter really, anyone using the contrib is screwed until I fix it
and then you merge it. All because of (EG) a docs change. Not a waste of
time at all.
Post by John Spackman
Next time the generator is run would it try and download again?
Good question, probably yes. But that could be avoided.
Post by John Spackman
Doesn't this means that even minor documentation changes will require you
to merge a pull request from me?
Look, the basic idea behind it is that also contributions have sort of
"releases". No big bells or whistles, but the author works for some
time, makes changes, fixes bugs, tests, and once he is confident
"publishes" a new state of a version of his contribution (Think of
qooxdoo publishing a 2.1.2 in the 2.1.x "version"). With "publishing"
meaning updating the catalog entry for this version.
Sorry, didn't you say originally that the .zip would come from github
archive? That's not a release that's just a commit & push. You commit
and push 100's of times between releases. Your original examples referred
to master.
That's sort of nice and clear. If, OTOH, you want something like
continuous deployment where every little change is directly propagated
to the users, it gets tedious.
I find (with contribs) that after major releases everything settles down
quite quickly and there tend to be occasional bugs that I or users find.
They get fixed, tested, and released ASAP. IMHO it's pretty essential
that releases *can* be frequent. Its even a common theme in Git ("commit
early, commit often").

I wasn't aware that the contrib authors are causing loads of problems by
shipping out half-finished premature code and need to be persuaded to make
more concrete releases.
Post by John Spackman
Post by John Spackman
Post by John Spackman
Maintaining multiple versions can be complicated, that's life - but
IMHO
the difference is that with the catalog, it's obfuscated.
This I don't get?! The catalog makes contrib versions absolutely
explicit.
I think that's obfuscated because it's all about the name of the correct
branch in my contrib being in your master with the correct SHA for the
current version of my branch. If you, me, or the user get any of that
wrong (or you're not around to merge a pull request) then the generator
will ignore my contrib.
For one thing, the user has very little to get wrong. He just enters
something like "contrib://UploadWidget/0.4" and be done with it. Not
much opportunity to e.g. get the "0.4" wrong, right?!
Weren't we talking about contrib://YourContribNameHere/master (or trunk or
whatever)? (your examples were QxGC/master or Dialog/master) So your
point is that contrib authors must name their releases by branch and
contrib users must explicitly select their release?
The other thing is the author who needs to put a Manifest.json into the
catalog, under UploadWidget/0.4, with the right URL to the archive, and
the right checksum over it. Plenty of opportunity to get things wrong.
You can copy the right manifest into the wrong catalog path. The
manifest can point to the wrong archive. The checksum was made over a
different archive than the URL points to. Phew!
Nice sarcasm - but if you think that it makes sense to have a past release
of Qooxdoo download a Manifest from the master of Qooxdoo for a branch in
another repo then I guess that's cool. But hey - you could just write a
couple more pages in the manual then people have no excuse to not see your
point of view.
For one thing there is no contribution "aristo/master", but only
"Aristo/trunk" (mind capitalisation and version name).
Yes there is - https://github.com/johnspackman/aristo, and I did say that
path names were for illustration only. Update of the qooxdoo.org/cobntrib
page is pending release of qx3.0
But let's just
pretend it was named "Aristo/master". What you should do is create
versions of the Aristo contribution, e.g. "Aristo/0.5", which would be
the one compatible with qooxdoo 2.x, and another version, e.g.
"Aristo/0.6", which would be compatible with qooxdoo 3.x. That should be
easy enough to convey to users.
The problem with "master" (or "trunk" or "tip" or the like) versions of
a contribution is that users need to be aware
What's not clear then is that you want contrib authors need to implement a
release process and contrib users need to explicitly upgrade from one
version of a contrib to another.

I can see how that definitely applies to major releases, although given
that contribs tend to have a much smaller release cycle (e.g. than
qooxdoo) that seems a little overkill. Whatever.

Based on what you've said, it would presumably make sense to *not* list
trunk or master in the catalog? Just releases.
Post by John Spackman
Mh, this is getting out of hand.
Yes, frustrating
Post by John Spackman
The qooxdoo master version is not
involved at all,
What??
I'm only considering users of released qooxdoo
versions. 3.0 SDK users will use the catalog, older versions will go the
Sourceforge SVN directly, both will use whatever is available through
their method of resolution.
But there will one day hopefully be a qooxdoo version 4.0, yes? And a
v5.0? And so the principal will apply in the future i.e. When current and
previous versions pull from the qooxdoo master catalog?
thron7
2013-06-11 19:11:42 UTC
Permalink
Post by John Spackman
Nice sarcasm - but if you think that it makes sense to have a past release
of Qooxdoo download a Manifest from the master of Qooxdoo for a branch in
another repo then I guess that's cool.
Post by thron7
Post by thron7
The qooxdoo master version is not
involved at all,
What??
But there will one day hopefully be a qooxdoo version 4.0, yes? And a
v5.0? And so the principal will apply in the future i.e. When current and
previous versions pull from the qooxdoo master catalog?
There is this recurring theme of yours, about the "master of qooxdoo",
which I honestly don't get. So we are not talking about the master branch
of the main qooxdoo repo, qooxdoo/qooxdoo, are we?! We are talking about
the qooxdoo/contrib-catalog repo, which as it is a Git repo has a default
branch named "master", right?!

This is just an implementation detail. The git branch of the
contrib-catalog repo is of no relevance. We could have implemented the
catalog with the same semantics using a plain file and hide it behind a
web form. It's just a convenient way to store some hierarchical
information, with the catalog .json files as the leafs.

So the catalog lookup of the Generator is against the base URL of the
catalog repo, and yes, this is against its master branch. I don't expect
this to change anywhere in the future, it will always be that branch, and
the Generator will always fetch information from the latest commit in that
branch. - Does that answer your question?!

(BTW: This aspect is exactly the same as with the old Sourceforge repo,
where all the contribs with their individual versions were maintained in
the SVN "trunk" branch [1]).

T.

[1]
http://sourceforge.net/p/qooxdoo-contrib/code/HEAD/tree/trunk/qooxdoo-contrib/
John Spackman
2013-06-12 08:34:28 UTC
Permalink
You're right, I'd forgotten that it was a separate qooxdoo-catalog repo
but that doesn't change things at all; you've still made it so that

(1) either (a) every commit/pull breaks the contrib until you are
available to commit a pull request with a new SHA, or (b) every
commit/pull is a new release (requiring a manual upgrade by users) no
matter how trivial the change.

(2) We have to submit pull requests to you just to release a change to our
code, for which we have to maintain a clone of your catalog.

You've said that this is because (i) contrib users need to be protected
from being a jquery-plugin-style "attack vector", (ii) it's good practice
to make contribs have releases to avoid issues with master/trunk, and
(iii) that the generator needs a SHA to be able to avoid expensive repeat
downloads.

With (i), I'm not aware that we've had even a hint of the problem, there's
already a simple & effective foil (i.e. Andreas, and you've said you're
not proposing anything else), and IMHO we're hardly a target given the
small number of users we have. With (ii) that's not an unreasonable idea,
but forcing a release every commit is not so reasonable.

The only real reason I can see here is (iii), the generator, and in the
process you've enforced a release process on the contrib authors the
penalty for which is that their contrib will break. I can't see any other
reason and therefore it sounds like a solution in search of a problem to
me, and a PITA that I don't want.

----------

All I'm asking for is that you find a way to remove the dependency on
having a SHA in the catalog Manifest.json, and remove the duplication of
data between the catalog's Manifest.json and the contrib.'s own
Manifest.json (ideally, just have a Contrib.json or similar to specify the
URL to find the real Manifest.json).

I have a suggestion for how you might do this without costly downloads -
if it's a normal http:// file repo, use If-Modified-Since; if it's github
archive .zip URL, you can quickly check if there have been updates from an
RSS feed, e.g. see [1] for [2]. You could either auto-detect the RSS feed
or allow the catalog Manifest to specify it as an alternative to
If-Modified-Since.

PS the github RSS feed itself supports If-Modified-Since.

[1] https://github.com/johnspackman/aristo/commits/diet.atom
[2] https://github.com/johnspackman/aristo/archive/diet.zip

John
thron7
2013-06-12 09:44:38 UTC
Permalink
Post by John Spackman
You're right, I'd forgotten that it was a separate qooxdoo-catalog repo
but that doesn't change things at all; you've still made it so that
(1) either (a) every commit/pull breaks the contrib until you are
available to commit a pull request with a new SHA, or (b) every
commit/pull is a new release (requiring a manual upgrade by users) no
matter how trivial the change.
I think there is something you still don't get. Say the user is using
"contrib://Dialog/0.5" and the maintainer of this contrib is fixing bugs
in the 0.5 version. Then he can provide a new "release" of this version
in form of a catalog update in the Dialog/0.5/Manifest.json entry. Once
this is online the user would automatically get the new version, without
any change to his contrib://Dialog/0.5 config entry. ("version" of a
contribution is much more like a branch in a git repo, but I stuck with
the term as it was already established.)
Post by John Spackman
(2) We have to submit pull requests to you just to release a change to our
code, for which we have to maintain a clone of your catalog.
You've said that this is because (i) contrib users need to be protected
from being a jquery-plugin-style "attack vector", (ii) it's good practice
to make contribs have releases to avoid issues with master/trunk, and
(iii) that the generator needs a SHA to be able to avoid expensive repeat
downloads.
With (i), I'm not aware that we've had even a hint of the problem, there's
already a simple & effective foil (i.e. Andreas, and you've said you're
not proposing anything else), and IMHO we're hardly a target given the
small number of users we have.
Maybe I'm overly concerned, but with the new system we truely lowered
the barrier for abuse. I'm not sure we should wait until something
happens before we take any measure.
Post by John Spackman
With (ii) that's not an unreasonable idea,
but forcing a release every commit is not so reasonable.
Again, the model here was that of "controlled releases" of
contributions, not that of "continuous deployment". But if the majority
of contribution maintainers leans towards the second, we can re-consider.
Post by John Spackman
The only real reason I can see here is (iii), the generator, and in the
process you've enforced a release process on the contrib authors the
penalty for which is that their contrib will break. I can't see any other
reason and therefore it sounds like a solution in search of a problem to
me, and a PITA that I don't want.
I think you're overly aggressive, John.
Post by John Spackman
----------
All I'm asking for is that you find a way to remove the dependency on
having a SHA in the catalog Manifest.json,
I've yet to hear a viable alternative for this.
Post by John Spackman
and remove the duplication of
data between the catalog's Manifest.json and the contrib.'s own
Manifest.json (ideally, just have a Contrib.json or similar to specify the
URL to find the real Manifest.json).
I could see that. But if you wanted to use Github's auto-archive feature
you would still need a copy of the contribution's Manifest (as we agreed
that a file of an archive cannot contain the checksum over this
archive). So why not maintain this copy in the catalog?!

A much cheaper solution for us would be to give all contributers write
access to the catalog repo directly, which would remove all pains from
pending pull requests.
Post by John Spackman
I have a suggestion for how you might do this without costly downloads -
if it's a normal http:// file repo, use If-Modified-Since;
We can only download archives, remember?! But yes, we could implement
If-Modified-Since checks as a first measure for freshness. But this
wouldn't cover the archive integrity check you can do with checksums (to
avoid hacked file hosters or man-in-the-middle attacks). But I
acknowledge security is a lesser concern to you...
Post by John Spackman
if it's github
archive .zip URL, you can quickly check if there have been updates from an
RSS feed, e.g. see [1] for [2]. You could either auto-detect the RSS feed
or allow the catalog Manifest to specify it as an alternative to
If-Modified-Since.
You have to keep in mind that the solution should be simple and
host-agnostic. How would your proposal integrate with Bitbucket? With
Launchpad? With Google Code? I would very much try to keep the Generator
free of hoster-specific code.

Anyway, as you said yourself you're "late to the party", as the phase
for discussion was a while ago. We will not be able to do much right
now, as our hands are bound for the 3.0 release. But as I said, things
are not cut into stone, and if a majority of users wants a different
solution we can address this in a few months time. Until then give the
system a chance, work with it so we do not judge everything from first
sight (as doing things in a new way is always irritating at the beginning).

In any case the best first step is to create a bug for any issues (which
also has the advantage that others can vote for it).

T.
John Spackman
2013-06-12 13:13:38 UTC
Permalink
Post by thron7
I think there is something you still don't get.
As soon as I commit a new release, the SHA of my git archive changes and
no longer matches the catalog Manifest.json, and the generator will ignore
that archive until it matches the SHA in the catalog. My contrib is
broken until you commit the pull request. What have I got wrong about
that?

*If* the user has already downloaded an archive, then they keep the old
one (so the generator is making them run on old, out of date code) or the
generator refuses to compile. You've said you haven't decided which yet.

The only alternative is to create a new branch (aka "controlled release")
and submit a separate request to you. What have I got wrong about that?
Post by thron7
Maybe I'm overly concerned, but with the new system we truely lowered
the barrier for abuse. I'm not sure we should wait until something
happens before we take any measure.
I think you mean "raised" here but the only change is in the case where
the server storing the .zip file has been hacked and the .zip modified.
Not impossible by any means but it seems odd to focus on the one exploit -
the jquery-plugin-disaster you gave as an example was caused by people
submitting apparently legit contribs.

You said that you'd have a check to see if there's a website and Andreas
would do the same introduction as he's always done. After a few commits
(or maybe just one) anyone can add what they want into their contrib
because you're not going to be checking the contents.
Post by thron7
Post by John Spackman
With (ii) that's not an unreasonable idea,
but forcing a release every commit is not so reasonable.
Again, the model here was that of "controlled releases" of
contributions, not that of "continuous deployment". But if the majority
of contribution maintainers leans towards the second, we can re-consider.
Again, your definition of "controlled release" == you disable the contrib
after the author makes a commit until you do a merge.
Post by thron7
Post by John Spackman
The only real reason I can see here is (iii), the generator, and in the
process you've enforced a release process on the contrib authors the
penalty for which is that their contrib will break. I can't see any
other
reason and therefore it sounds like a solution in search of a problem to
me, and a PITA that I don't want.
I think you're overly aggressive, John.
I don't understand why you would say this - it is simply the only reason
that I can see. That's not aggressive, just a statement of fact...and you
haven't offered any other reason, except for the possibility of thwarting
a single type of exploit.
Post by thron7
Post by John Spackman
All I'm asking for is that you find a way to remove the dependency on
having a SHA in the catalog Manifest.json,
I've yet to hear a viable alternative for this.
I'd given one, further down. I'll try and expand my suggestion again
below.
Post by thron7
So why not maintain this copy in the catalog?!
The opposite argument also applies - why not maintain it in the contrib,
where it's relevant (and is already located)?
Post by thron7
A much cheaper solution for us would be to give all contributers write
access to the catalog repo directly, which would remove all pains from
pending pull requests.
That would be an improvement
Post by thron7
Post by John Spackman
I have a suggestion for how you might do this without costly downloads -
if it's a normal http:// file repo, use If-Modified-Since;
We can only download archives, remember?!
Yes, I do remember - and that's what I meant. By "http:// file repo" I
meant an http:// URL that references a [single] file, IE any static
archive.zip which is not backed by a VCS. EG if, as you suggested, the
user uploads a file to bitbucket or their own web server or wherever, they
will (in most cases) support If-Modified-Since.
Post by thron7
But yes, we could implement
If-Modified-Since checks as a first measure for freshness. But this
wouldn't cover the archive integrity check you can do with checksums (to
avoid hacked file hosters or man-in-the-middle attacks). But I
acknowledge security is a lesser concern to you...
Look, this is getting ridiculous - I'm not unconcerned about security in
any way but you're now saying that you're actively trying to protect
against a third party hosting service being hacked? To follow that logic
you should include SHA of the Qooxdoo archives on another platform too
then, just in case Qooxdoo's server's get hacked/targeted?
Post by thron7
Post by John Spackman
if it's github
archive .zip URL, you can quickly check if there have been updates from
an
RSS feed, e.g. see [1] for [2]. You could either auto-detect the RSS
feed
or allow the catalog Manifest to specify it as an alternative to
If-Modified-Since.
You have to keep in mind that the solution should be simple and
host-agnostic. How would your proposal integrate with Bitbucket? With
Launchpad? With Google Code? I would very much try to keep the Generator
free of hoster-specific code.
The suggestion is that catalog has a URL to the archive, and optionally a
URL to an RSS feed to detect whether the archive has changed (which would
be provided only when the If-Modified-Since is not supported). That is
host agnostic.
Post by thron7
Anyway, as you said yourself you're "late to the party", as the phase
for discussion was a while ago.
I just checked the docs at [1] and after some searching found the blog
posting at [2]; the docs at [1] is apparently obsolete, and blog [2] only
applies. My emails on this subject have only come up because you sent an
email 9 days ago asking contrib authors to actually try out the catalog
and I got stuck in as early as I could.

I'm telling you about my experiences with something you're only just
introducing and trying to help you see a flaw in the implementation and a
(hopefully simpler, better) way of fixing it before it's released, and I'm
being constructive about it (coming up with solutions rather than just
complaining).

What I get in return seems to be sarcasm, name calling, and general
intractability - all of which I find more than a little strange given that
this is supposed to be a collaborative effort. Sorry you don't see my
comments as helpful but I guess c'est la vie.

[1] http://qooxdoo.org/contrib/qooxdoo-contrib2.0
[2] http://news.qooxdoo.org/qooxdoo-contrib-2-0-the-blueprint

John
Andreas Ecker
2013-06-14 09:37:27 UTC
Permalink
Hi John,

thanks for your valuable input regarding a future qooxdoo-contrib
infrastructure. You seem to have raised a couple of valid issues,
particularly for handling contribs that might still be under development.
We've started to take your concerns into account already to see if they
could fit the proposed scope of the catalogue. Currently it might just
take us a while to look deeper at the issues and figure out if and how to
address them. As we appreciate your input, and a solid but constructive
technical discussion (sorry if there have been misunderstandings), we hope
to catch up with you (and others) at the contrib 2.0 topic soon. TTYL,

Andreas
--
Project Lead
http://qooxdoo.org
John Spackman
2013-06-14 09:45:20 UTC
Permalink
Hi Andreas

Thanks for your email, it's much appreciated.

All the best
John
Post by Andreas Ecker
Hi John,
thanks for your valuable input regarding a future qooxdoo-contrib
infrastructure. You seem to have raised a couple of valid issues,
particularly for handling contribs that might still be under development.
We've started to take your concerns into account already to see if they
could fit the proposed scope of the catalogue. Currently it might just
take us a while to look deeper at the issues and figure out if and how to
address them. As we appreciate your input, and a solid but constructive
technical discussion (sorry if there have been misunderstandings), we hope
to catch up with you (and others) at the contrib 2.0 topic soon. TTYL,
Andreas
--
Project Lead
http://qooxdoo.org
--------------------------------------------------------------------------
----
Build for Windows Store.
http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
qooxdoo-devel mailing list
https://lists.sourceforge.net/lists/listinfo/qooxdoo-devel
Stefan Andersson
2013-06-14 21:48:14 UTC
Permalink
Dear qooxdoo team,

Please, let down attitude and collaborate! The attitude has been a problem a couple of times before, though I think there is an undefined language barrier too. An excuse would be in place.

Dear John,

I appreciate your comments and ideas regarding the Contrib 2.0 It has been valuable to read and so far I see most of the obstacles to overcome. I would wish the qooxdoo team to really look into it.

Dear Andreas,

Thanks for your direct reply that you now take John's comments and ideas seriously.

Here is a trade off to keep the generator independent and not more complex. I see the same problems as John, though most we can overcome.

Keep up the good work!

Collaboration and transparency has increased and we really hope the qooxdoo team will furthermore enhance these gestures in future.

The best for the community,
Stefan

Continue reading on narkive:
Loading...