Discussion:
[Twisted-Python] Suggested plan for GitHub migration
Craig Rodrigues
2015-11-17 23:48:09 UTC
Permalink
For now, the funds were raised to migrate to GitHub, so we can not use
them to do other things.
We will stay on Trac for issues... at least for now.
I have no idea how we can migrate to any issue tracker without losing
data if we don't have full access to the database.
It is possible to migrate to another issue tracker and not lose
data. I've done Trac -> Redmine, and it works, but there was an existing
migration
script I could use.
For migrating to a cloud based bug tracker, you would need to take every
user
in the existing Trac database, and see if there would be a way to map
the existing users to the cloud database, such as GitHub. It's a lot of
work, but possible.
However, for the scope of this project, if staying with Trac for issues is
what is required, that is fine.
We don't plan to migrate to GitHub Issues / GitHub Wiki / GitHub Pages
OK.

So based on what you have listed, I would say that most of the work will be
working with Git post commit hooks.

I would say the plan should be something like this.

A.1 https://github.com/twisted/twisted will be the "repository of truth"
for Twisted.
-> Twisted releases will be done from GitHub
-> the Twisted developers who are now "core committers" for SVN,
must be
given access to be "core committers" to
https://github.com/twisted/twisted

A.2 On the Trac server, a local git mirror of the GitHub must be set up.
Every bug tracker I've seen that integrates with git needs a local
mirror of the repo
in order to parse the git history in order to update the bug
database.
This mirror should be read-only, and the only thing updating this
repo should be the Trac GitHub plugin.

A.3 On the Trac server, this plugin must be installed:
https://github.com/trac-hacks/trac-github

A.4 On the GitHub server, a post-commit web hook must be configured. The
workflow will be this:

[core committer does push to https://github.com/twisted/twisted]

-> [post commit GitHub hook will be called to poke the Trac
GitHub plugin]
-> [Trac GitHub plugin will update the local git repo on the
Trac server]
-> [Trac GitHub will parse the git history for new commits and
update tickets]

I would recommend that steps (1) - (4) be made to work in a staging
environment, with a separate
GitHub repo, and a separate copy of the Trac database. That way, you can
test things out without derailing
Twisted developers. When you are confident that this workflow works, then
the transition plan will be something
like the following.

B.1 Send an e-mail to the mailing list and pick one day for the
maintenance window.
This will warn folks when they should take a holiday from Twisted
work. :)

B.2 When maintenance is about to begin, send a [HEADSUP] mail saying that
repo will be unavailable.

B.3 Create Subversion pre-commit hook to disable all commits to
Subversion:

http://stackoverflow.com/questions/2411122/how-to-freeze-entire-svn-repository-to-make-it-read-only

B.4 Set up steps A.1 - A.4

B.5 Verify that B.4 works. Have someone (Glyph?) do a commit to
https://github.com/twisted/twisted, and
make sure that Trac works.

B.6 Once the Twisted core team are satisified that everything works, send
an e-mail to the mailing list
that the maintenance window is over, and GitHub is now where the
action is!

B.7 Update all wiki documentation to change all references to getting code
from Subversion,
to getting code from GitHub.

B.8 Update all systems which used Subversion to use GitHub. For example,
buildbots.

--
Craig


--
Craig
Adi Roiban
2015-11-29 16:20:36 UTC
Permalink
Hi Craig,

Sorry for the delay and many thanks for your plan.

I have also sent your plan to the Unofficial Twisted Software Foundation.

From what I can see we are stuck in bureaucratic process.

The plan needs to be approved by Unofficial Twisted Software Foundation and
the Unofficial Twisted Software Foundation want to have a single plan
submitted for approval.

We now have 3 plans : Amber's, Craig's and mine..... and we see which plan
to be sent to the Unofficial Twisted Software Foundation.

Also the Unofficial Twisted Software Foundation will only consider plans
for GitHub.com.

No GitLab/Bitbucket/Jira/Redmime....etc

We can try to keep tickets/wiki/website on Trac and only move the main repo
+ PR + hooks to GitHub.

This should allow us to get rid of SVN and build the infrastructure on web
hooks. Later we can consider migration to other tools... or extending the
GitHub.com usage to Issues / Wiki / GitHub Pages...etc

Until now we failed to coordinate toward creating a single plan and besides
Glyph's comments on IRC, I have not received any feedback from the Unofficial
Twisted Software Foundation for any of the plans.

Can we plan an IRC/Google Hangouts meeting to discuss the plan?

It would be great if someone from the Unofficial Twisted Software
Foundation could also join.

If you would like to see Twisted on Git and GitHub please consider joining
the meeting.

I have created this Doodle poll to help us schedule the date and time for
the meeting - http://doodle.com/poll/4ys8m8qakav9u9f9

Thanks!
--
Adi Roiban
Glyph Lefkowitz
2015-11-30 18:41:30 UTC
Permalink
Post by Adi Roiban
Hi Craig,
Sorry for the delay and many thanks for your plan.
I have also sent your plan to the Unofficial Twisted Software Foundation.
Just for everyone's information, the real name for the relevant group here is the "Twisted Project Leadership Committee for the Software Freedom Conservancy". The reason that the relevant mailing list is titled "unofficial twisted software foundation" is that we started off investigating if we could start our own foundation, and later, when we opted to have the Software Freedom Conservancy as our fiscal sponsor instead, the name "twisted software foundation" became "unofficial" because there is no such legal entity.
Post by Adi Roiban
From what I can see we are stuck in bureaucratic process.
We are stuck at the point before the bureaucratic process :). The bureaucratic process is that the PLC votes to approve the plan.
Post by Adi Roiban
The plan needs to be approved ... and [the PLC] want to have a single plan submitted for approval.
The reason the plan needs to be approved is that the plans for the fellowship are documented here - https://twistedmatrix.com/trac/wiki/Fellowship2015#WorkPlan <https://twistedmatrix.com/trac/wiki/Fellowship2015#WorkPlan> - and that says "The maintainer will develop a plan for migration of development to GitHub, and once it has been approved implement the plan".
Post by Adi Roiban
We now have 3 plans : Amber's, Craig's and mine..... and we see which plan to be sent to [the PLC].
Also [the PLC] will only consider plans for GitHub.com.
This is also because it's written in https://twistedmatrix.com/trac/wiki/Fellowship2015#WorkPlan <https://twistedmatrix.com/trac/wiki/Fellowship2015#WorkPlan>, so it was decided before the fellowship began. However, lots of the steps to move to GitHub.com <http://github.com/> are also necessary prerequisites to use any of those sites - getting rid of subversion and reducing the amount of infrastructure we are operating. Moving from Github somewhere else ought to be radically simpler than the process we've been undertaking to move to Github.
Post by Adi Roiban
We can try to keep tickets/wiki/website on Trac and only move the main repo + PR + hooks to GitHub.
This sounds good to me.
Post by Adi Roiban
This should allow us to get rid of SVN and build the infrastructure on web hooks. Later we can consider migration to other tools... or extending the GitHub.com usage to Issues / Wiki / GitHub Pages...etc
Until now we failed to coordinate toward creating a single plan and besides Glyph's comments on IRC, I have not received any feedback from [the PLC] for any of the plans.
The PLC is unlikely to be involved. Personally, I'm severely overcommitted; most of the other PLC members have a very low level of involvement with the project. Not going to point fingers specifically, but some hardly answer their email :-).

While I would prefer it if the PLC were a bit more involved, it should not be much of an issue in this case (assuming that I can herd the cats in the right direction when it's time for a vote). The PLC's job in this case is only to provide oversight, just to verify that the plan is a proper investment of the SLC's financial resources. In the same way that reviewers should not participate too closely in the authorship of patches they review, it should be fine if the PLC is not involved in the planning process. (If we had the personal resources to do so, we probably wouldn't have needed to put it into the fellowship plan!) The PLC won't decide between a set of different plans; it will just give final approval to the proposed one.
Post by Adi Roiban
Can we plan an IRC/Google Hangouts meeting to discuss the plan?
I've not had good luck with Hangouts, personally, but scheduling time on IRC sounds like a good idea.
Post by Adi Roiban
If you would like to see Twisted on Git and GitHub please consider joining the meeting.
I have created this Doodle poll to help us schedule the date and time for the meeting - http://doodle.com/poll/4ys8m8qakav9u9f9 <http://doodle.com/poll/4ys8m8qakav9u9f9>
Ultimately, it is Amber and Adi who need to coordinate on this. But I would strongly encourage anyone with an interest in this migration to participate in the planning process; clearly we need some help nailing down the specifics :).

-glyph
Amber "Hawkie" Brown
2015-11-30 14:49:41 UTC
Permalink
Hi Craig,

Thanks for this, sharing your past experience is invaluable :)

I've gone through and thought about it a bit, and rewritten it into https://github.com/twisted-infra/braid/blob/git-migration-plan/gitmigration.rst -- it is basically your plan, with some added notes about Twisted Infra specific parts. I've skimmed over the specific details, since I feel that going too in-depth in such a plan will just be wasted effort as unknown issues arise, but with enough structure that we have a clear set of overarching goals for each step.

The migration will have a handful of policy changes that we will have to resolve -- such as ensuring that all merges have a topfile -- which aren't possible under a GitHub based system. I think these issues will just involve a lot of scrutiny and double checking during the transitional period until we are confident that we are enforcing our existing quality and process standards.

If anyone has any suggestions, or any more invaluable experience to share, please do :)

- Amber
For now, the funds were raised to migrate to GitHub, so we can not use
them to do other things.
We will stay on Trac for issues... at least for now.
I have no idea how we can migrate to any issue tracker without losing
data if we don't have full access to the database.
It is possible to migrate to another issue tracker and not lose
data. I've done Trac -> Redmine, and it works, but there was an existing migration
script I could use.
For migrating to a cloud based bug tracker, you would need to take every user
in the existing Trac database, and see if there would be a way to map
the existing users to the cloud database, such as GitHub. It's a lot of work, but possible.
However, for the scope of this project, if staying with Trac for issues is what is required, that is fine.
We don't plan to migrate to GitHub Issues / GitHub Wiki / GitHub Pages
OK.
So based on what you have listed, I would say that most of the work will be
working with Git post commit hooks.
I would say the plan should be something like this.
A.1 https://github.com/twisted/twisted will be the "repository of truth"
for Twisted.
-> Twisted releases will be done from GitHub
-> the Twisted developers who are now "core committers" for SVN, must be
given access to be "core committers" to https://github.com/twisted/twisted
A.2 On the Trac server, a local git mirror of the GitHub must be set up.
Every bug tracker I've seen that integrates with git needs a local mirror of the repo
in order to parse the git history in order to update the bug database.
This mirror should be read-only, and the only thing updating this repo should be the Trac GitHub plugin.
A.3 On the Trac server, this plugin must be installed: https://github.com/trac-hacks/trac-github
[core committer does push to https://github.com/twisted/twisted]
-> [post commit GitHub hook will be called to poke the Trac GitHub plugin]
-> [Trac GitHub plugin will update the local git repo on the Trac server]
-> [Trac GitHub will parse the git history for new commits and update tickets]
I would recommend that steps (1) - (4) be made to work in a staging environment, with a separate
GitHub repo, and a separate copy of the Trac database. That way, you can test things out without derailing
Twisted developers. When you are confident that this workflow works, then the transition plan will be something
like the following.
B.1 Send an e-mail to the mailing list and pick one day for the maintenance window.
This will warn folks when they should take a holiday from Twisted work. :)
B.2 When maintenance is about to begin, send a [HEADSUP] mail saying that repo will be unavailable.
B.3 Create Subversion pre-commit hook to disable all commits to
http://stackoverflow.com/questions/2411122/how-to-freeze-entire-svn-repository-to-make-it-read-only
B.4 Set up steps A.1 - A.4
B.5 Verify that B.4 works. Have someone (Glyph?) do a commit to https://github.com/twisted/twisted, and
make sure that Trac works.
B.6 Once the Twisted core team are satisified that everything works, send an e-mail to the mailing list
that the maintenance window is over, and GitHub is now where the action is!
B.7 Update all wiki documentation to change all references to getting code from Subversion,
to getting code from GitHub.
B.8 Update all systems which used Subversion to use GitHub. For example, buildbots.
--
Craig
--
Craig
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Adi Roiban
2015-11-30 15:37:56 UTC
Permalink
On 30 November 2015 at 16:49, Amber "Hawkie" Brown <
Post by Adi Roiban
Hi Craig,
Thanks for this, sharing your past experience is invaluable :)
I've gone through and thought about it a bit, and rewritten it into
https://github.com/twisted-infra/braid/blob/git-migration-plan/gitmigration.rst
-- it is basically your plan, with some added notes about Twisted Infra
specific parts. I've skimmed over the specific details, since I feel that
going too in-depth in such a plan will just be wasted effort as unknown
issues arise, but with enough structure that we have a clear set of
overarching goals for each step.
[snit]

+1 for "master" as the main branch.

+1 for GitHub logins but I have no idea how we could migrate existing
accounts.

This is more like a Git migration plan with main repo on GitHub, but I
think that this should be the first step...and focus on this, before
looking at PR and GitHub issues.

I don't see any note about the code browser and how to migrate narrative
and apidocs source code links.

The Internet is full of links like
http://twistedmatrix.com/trac/browser/tags/releases/twisted-8.2.0/twisted/web/http.py#L475

I think that tracext.github.GitHubBrowser only redirects changeset ... and
from the docs only new changesets

I was hoping that as part of the git migration, we can implement a custom
Twisted web resource which will handle all the redirections to GitHub code
browser

Since Twisted trunk merges are not busy, I don't think that we need to
worry to much about breaking the dev process... I am more worried about
failing to gather the merges required to create the waiting/testing queue.

--
Adi
Post by Adi Roiban
For now, the funds were raised to migrate to GitHub, so we can not use
them to do other things.
We will stay on Trac for issues... at least for now.
I have no idea how we can migrate to any issue tracker without losing
data if we don't have full access to the database.
It is possible to migrate to another issue tracker and not lose
data. I've done Trac -> Redmine, and it works, but there was an
existing migration
script I could use.
For migrating to a cloud based bug tracker, you would need to take every
user
in the existing Trac database, and see if there would be a way to map
the existing users to the cloud database, such as GitHub. It's a lot of
work, but possible.
However, for the scope of this project, if staying with Trac for issues
is what is required, that is fine.
We don't plan to migrate to GitHub Issues / GitHub Wiki / GitHub Pages
OK.
So based on what you have listed, I would say that most of the work will
be
working with Git post commit hooks.
I would say the plan should be something like this.
A.1 https://github.com/twisted/twisted will be the "repository of
truth"
for Twisted.
-> Twisted releases will be done from GitHub
-> the Twisted developers who are now "core committers" for
SVN, must be
given access to be "core committers" to
https://github.com/twisted/twisted
A.2 On the Trac server, a local git mirror of the GitHub must be set up.
Every bug tracker I've seen that integrates with git needs a
local mirror of the repo
in order to parse the git history in order to update the bug
database.
This mirror should be read-only, and the only thing updating
this repo should be the Trac GitHub plugin.
https://github.com/trac-hacks/trac-github
A.4 On the GitHub server, a post-commit web hook must be configured.
[core committer does push to
https://github.com/twisted/twisted]
-> [post commit GitHub hook will be called to poke the Trac
GitHub plugin]
-> [Trac GitHub plugin will update the local git repo on
the Trac server]
-> [Trac GitHub will parse the git history for new commits
and update tickets]
I would recommend that steps (1) - (4) be made to work in a staging
environment, with a separate
GitHub repo, and a separate copy of the Trac database. That way, you
can test things out without derailing
Twisted developers. When you are confident that this workflow works,
then the transition plan will be something
like the following.
B.1 Send an e-mail to the mailing list and pick one day for the
maintenance window.
This will warn folks when they should take a holiday from
Twisted work. :)
B.2 When maintenance is about to begin, send a [HEADSUP] mail saying
that repo will be unavailable.
B.3 Create Subversion pre-commit hook to disable all commits to
http://stackoverflow.com/questions/2411122/how-to-freeze-entire-svn-repository-to-make-it-read-only
B.4 Set up steps A.1 - A.4
B.5 Verify that B.4 works. Have someone (Glyph?) do a commit to
https://github.com/twisted/twisted, and
make sure that Trac works.
B.6 Once the Twisted core team are satisified that everything works,
send an e-mail to the mailing list
that the maintenance window is over, and GitHub is now where the
action is!
B.7 Update all wiki documentation to change all references to getting
code from Subversion,
to getting code from GitHub.
B.8 Update all systems which used Subversion to use GitHub. For
example, buildbots.
--
Craig
--
Craig
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________
Twisted-Python mailing list
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
--
Adi Roiban
Tristan Seligmann
2015-12-01 07:35:15 UTC
Permalink
Post by Amber "Hawkie" Brown
The migration will have a handful of policy changes that we will have to
resolve -- such as ensuring that all merges have a topfile -- which aren't
possible under a GitHub based system.
You could make master/trunk/whatever a protected branch, and have a
required status check for this, but that would require some external CI
thing that actually performs the status check.
Jonathan Lange
2015-12-01 18:36:06 UTC
Permalink
For now, the funds were raised to migrate to GitHub, so we can not use
them to do other things.
We will stay on Trac for issues... at least for now.
I have no idea how we can migrate to any issue tracker without losing
data if we don't have full access to the database.
[snip]
B.7 Update all wiki documentation to change all references to getting
code from Subversion,
to getting code from GitHub.
Probably would be a good idea to have a list of such changes *before* the
migration.
B.8 Update all systems which used Subversion to use GitHub. For example,
buildbots.
Couldn't we do this before the migration anyway? At least partly?

jml
Glyph Lefkowitz
2015-12-02 01:01:34 UTC
Permalink
B.7 Update all wiki documentation to change all references to getting code from Subversion,
to getting code from GitHub.
Probably would be a good idea to have a list of such changes *before* the migration.
Yes. Everything should be written up and reviewed beforehand, and
B.8 Update all systems which used Subversion to use GitHub. For example, buildbots.
Couldn't we do this before the migration anyway? At least partly?
Well, I mean... this is "the migration" itself. So this should happen earlier in the process, particularly earlier than we start committing directly to git? Much of this work is already done: the buildbots are already getting the source from git, not SVN.

One thing that I'd love to see is a checklist that is kept up-to-date regarding which parts of this have already happened.

-glyph
Craig Rodrigues
2015-12-03 23:53:07 UTC
Permalink
Post by Glyph Lefkowitz
One thing that I'd love to see is a checklist that is kept up-to-date
regarding which parts of this have already happened.
I worked with Amber and Adi and we have the checklist:
https://github.com/twisted-infra/braid/blob/git-migration-plan/gitmigration.md

--
Craig
Glyph Lefkowitz
2015-12-04 03:07:49 UTC
Permalink
Post by Glyph Lefkowitz
One thing that I'd love to see is a checklist that is kept up-to-date regarding which parts of this have already happened.
https://github.com/twisted-infra/braid/blob/git-migration-plan/gitmigration.md <https://github.com/twisted-infra/braid/blob/git-migration-plan/gitmigration.md>
Thank you very much for your participation in this.

The important thing about a "checklist" though, is checking things off - how will that be kept up to date in sync with what has actually happened?

-glyph
Amber "Hawkie" Brown
2015-12-04 03:10:44 UTC
Permalink
Post by Glyph Lefkowitz
Post by Glyph Lefkowitz
One thing that I'd love to see is a checklist that is kept up-to-date regarding which parts of this have already happened.
https://github.com/twisted-infra/braid/blob/git-migration-plan/gitmigration.md
Thank you very much for your participation in this.
The important thing about a "checklist" though, is checking things off - how will that be kept up to date in sync with what has actually happened?
-glyph
When it's accepted, we can make it a GItHub issue, and then we can, well, click on the checkbox. :)

- Amber
Craig Rodrigues
2015-12-04 19:11:19 UTC
Permalink
Post by Glyph Lefkowitz
The important thing about a "checklist" though, is checking things off -
how will that be kept up to date in sync with what has actually happened?
The checklist is written using the GitHub flavored markdown syntax for
checklists:

https://github.com/blog/1375%0A-task-lists-in-gfm-issues-pulls-comments

so when the tasks are completed, this checklist will be updated by changing:

[ ] task item foobar

to

[X] task item foobar

and committing the updated checklist to GitHub.

Adi and Amber are on board with this, so it should be fine.
--
Craig
Tom Prince
2015-12-02 06:09:05 UTC
Permalink
Post by Glyph Lefkowitz
Probably would be a good idea to have a list of such changes *before* the migration.
Yes. Everything should be written up and reviewed beforehand, and
There has been a lot of words written talking about coming up with a
plan for the migration, but I have yet to see a concrete plan. I think
this (a list of all the things that depend on SVN) followed by plans to
address each of the, is probably a sensible first step. I suspect that
none of this will be contreviersial, but it seems that the disucssion
keeps dancing around things needing to be done, but nobody has taken the
time to actually come up with a list.

Tom
Glyph Lefkowitz
2015-12-16 08:40:09 UTC
Permalink
Post by Tom Prince
Post by Glyph Lefkowitz
Probably would be a good idea to have a list of such changes *before* the migration.
Yes. Everything should be written up and reviewed beforehand, and
There has been a lot of words written talking about coming up with a
plan for the migration, but I have yet to see a concrete plan. I think
this (a list of all the things that depend on SVN) followed by plans to
address each of the, is probably a sensible first step. I suspect that
none of this will be contreviersial, but it seems that the disucssion
keeps dancing around things needing to be done, but nobody has taken the
time to actually come up with a list.
This checklist is actually what will become the official plan, because investigating the exact steps required to satisfy each step is part of the plan itself :).

-glyph

Loading...