Difference between revisions of "AlliedModders Planning"

From AlliedModders Wiki
Jump to: navigation, search
m
(Phase 7)
 
(53 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
__FORCETOC__
 
__FORCETOC__
  
The complexity of SourceMod's branching (even as trivial as it is) is too much for Subversion and our build process.  There are a bunch of things at play here, and not all are related.  This is sort of a major push for AlliedModders to be a bit more streamlined as we're having trouble accommodating multiple projects, documentation sets, build systems, mirrors etc.
+
This is sort of a major push for AlliedModders to be a bit more streamlined.  We're encountering growth problems, especially as our personnel has dwindled quite low recently.  This document discusses some changes I'm proposing.  Anyone is free to discuss this on the forums or add comments into this document.
  
This document discusses some changes to the build process and a possible move to Mercurial.
+
Everything related to source code management in this document refers to Metamod:Source and SourceMod.  AMX Mod X is considered "mature" and is no longer maintained except for critical or high priority bugs.  If and when we do a move past Subversion, it is unlikely AMX Mod X will follow.
  
Some of the benefits we'd gain from the ideas in this document:
+
=The Problems=
*Much easier branching
+
==Short Term==
*Merging changesets would be FAR easier
+
*We're finding it hard to manage multiple projects in a unified way.
*Merges between branches would be tracked
+
*There are site disparities, Metamod:Source and SourceMod and AMX Mod X each have entirely different backends.
*All the benefits of a distributed system -- pulling changes from other parties is much easier
+
*Similarly, each site has a different version of the mirror system script. 
*Getting rid of revision numbers
+
*Subversion doesn't cut it -- Mercurial might handle things better.  Subversion basically has no branching or tagging support, and it has no real branch merging capabilities.  This is a real problem for SourceMod which has pretty active development.
*Making fetchdlls work seamlessly
+
*The packaging system is disorganized and fundamentally broken.  It's inconsistent, inaccurate, and takes up 1.7GB of wasted space and growing.
*Getting rid of package/symbol svn and doing 30day rotation to save disk space
 
  
 +
==Long Term==
 +
*AlliedModders has plans past SourceMod.  We need to be able to grow our project base without adding exponential amounts of work.
 +
*We still don't have a main site
 +
*MediaWiki still has authentication problems since its handling of usernames is broken
 +
*Flyspray is buggy, definitely non-production, and doesn't get maintained frequently
 +
*It's impossible to maintain the AMXX/SM/MM:S sites right now
  
=Overview=
 
Subversion doesn't let us branch nicely or create user-specific branches without forever mucking up the central repository.  It's difficult to do merges.  Mercurial seems to solve these problems, but our build process will be majorly affected.
 
  
 +
=Mercurial=
 +
Subversion doesn't let us branch nicely or create user-specific branches without forever mucking up the central repository.  It's difficult to do merges.
  
=The Revision Problem=
+
An excellent tutorial for how Mercurial works is [http://www.selenic.com/mercurial/wiki/index.cgi/UnderstandingMercurial here].
Mercurial tracks revisions with 160-bit IDs (40 digits of hex), called "changeset IDs."  Mercurial has aesthetic revision numbers only; they're not guarantees like they are in Subversion. Revision numbers are properties of a repository clone, not the overall repository. This means that if two people make changes locally, those revisions will be different, and will change again once pushed upstream.
 
  
This problem is somewhat mitigated by the fact that the central repository will have reliable revision numbers, but that's still not good enough since branching will generate multiple revisions.  Branching also makes the revision numbers awkward since users assign meaning to them where none exists (for example, "1.0.1.1234 versus 1.1.0.1231 doesn't make sense because 1.1 is later than 1.0.1").
 
  
This means we must change SourceMod to not use revision numbers, a change we should make regardless of a Mercurial merge.  The major problems to keep in mind are: '''What do we display for the version number?''' and '''How do we keep fetchdlls working?'''  (Fetchdlls needs to be able to get a changeset ID from the MINIDUMP_MODULE struct, which has a checksum and version info.)
+
=Phases=
  
==Solutions==
+
==Phase 0 - Done ==
There are a few ways to go about this. These are the two I came up with while writing this.
+
'''DONE:''' We've eliminated build numbers and are back on the x.y.z system.
  
*Revision numbers that are per major release.  We'd have a "build number server" that would dish out new numbers on request.  These numbers would be used by the build scripts.
+
*<strike>Apply new versioning scheme, whatever it may be.</strike>
**A decent bit of work but would preserve the most compatibility with fetchdlls.
 
**[svnrev] (or [hgrev] or whatever) would have mismatching revision/build # ids which could get confusing.  Thus we'd have to try and not display the revision numbers publicly (they'd only be used for linking).
 
*Simply assign 0 to the build number and ignore it entirely.
 
**Seems easy but we'd need to have a mapping between VS's "checksum" values and a changeset ID if we wanted fetchdlls to work.  Not too difficult really as this could be done with a simple HTTP push.
 
**For visual display, we'd totally eliminate the build number.  It'd only be visible on Windows from the file information.
 
**We'd add a secret date+time field to plugins that would be filled in by the compiler.
 
  
For plugins, displaying a timestamp is not a problem.  We could put some sort of "public String:__date[] = __DATE__" into core.inc.
+
==Phase 1 - Done==
 +
'''DONE!'''
  
Right now, I'm preferring the second solution.  There's no significance to a build number if we can replace it with a timestamp, and a checksum -> changeset mapping is trivial.
+
<strike>Move from Sente to Iroh so we can get away from ThePlanet and clean up/optimize some infrastructure.  This is documented at [[Iroh_Migration|Iroh Migration]].</strike>
  
 +
==Phase 2==
 +
Expected date: Mid-July
  
=Automation=
+
*<strike>Redo automation so the "packaging" versus "sending upstream" processes are fundamentally separate.</strike>
Right now we use subversion for all of the packages. This is getting to be troublesome. It makes fetchdlls extraordinarily complicated when it really does not have to be, and the hourly builds just accumulate with no way of being deleted.  The package repository is 1.7GB and growing.
+
*<strike>Redo "sending upstream" process so it simply sends over FTP.</strike>
 +
*<strike>Use an SSH tunnel to notify of new packages, so we don't have to do something idiotic like mirrors.pl for the mirror system, or using post-commit scripts on the Packages repository.</strike>
 +
*Redo the package system so only 30 days worth of builds are available.
 +
*<strike>Write a script to help users diff between packages? (Update: No one asked for it.)</strike>
 +
*<strike>Remove the Packages repository entirely (transitioning AMX Mod X to this will be trivial).</strike>
 +
*<strike>Update symbol server to include binaries and maybe even sourcecode? Removes the need for fetchdll's.</strike>
  
It would be a lot simpler if we could just throw a single package on the webserver and forget about it.  We could even keep a rotation of, say, 30 days' worth of packages in a rotation.  This would let us remove cruft and keep enough binaries around for debugging.  The problem is that people rely on the repositories to perform diffs.
+
==Phase 3 - Done==
 +
'''DONE!'''
  
I don't have any good ideas on solving that problem, because there's no way to selectively rotate things out of a versioning system. The only solution I can think of is providing people with, say, a perl or bash script that does the job for them.  This could go through two routes: using local repositories, or simply doing a HTTP fetch from the web server and then doing a diff.
+
*<strike>{{bz|3257}}: Transition SourceMod to vs2k8.</strike>
 +
*<strike>{{bz|3258}}: Transition Metamod:Source to vs2k8.</strike>
  
 +
==Phase 4 - Done==
 +
'''DONE!'''
  
=License Changes=
+
*<strike>Move the pdbstore to be centralized client-sideThis can be accomplished by making a more advanced symstore wrapper that will remove anything older than 30 days, and then (in a similar SSH tunnel method as above), rsync all changes upstream to the webserverThe exact mechanism is still up in the air, but rsync would probably suffice (as long as we can stomach cygwin).</strike>
I don't want to keep hacking up our private->public sync scriptDS has faithfully maintained it until now, but I believe he will be as glad as I to do away with itA move to Mercurial would have either of the following benefits:
+
*<strike>Remove the Symbols repository.</strike>
*We could keep the JIT closed source and shove it in a private repository, OR
+
*<strike>Rewrite fetchdlls to support the new layout.</strike>
*Just open source the thing finally, it's not very special, and have everything be together.
 
  
 +
==Phase 5 - Done==
 +
'''DONE!'''
  
=Phases=
+
*<strike>Add hg.alliedmods.net pointing to our "repository roots," which will be per-project.  For example, hg.alliedmods.net/sourcemod.</strike>
 +
*<strike>From there, import trunk as "sourcemod."</strike>
 +
*<strike>Branch every major release off as separate repositories, from the appropriate revisions.</strike>
 +
*<strike>Tag minor releases.</strike>
 +
*<strike>Update all links we can find on the site and in the docs.</strike>
  
==Phase 0==
+
==Phase 6 - Done==
*Apply new versioning scheme, whatever it may be
+
'''DONE!'''
  
==Phase 1==
+
*<strike>Import mmsource into Mercurial.</strike>
*Redo the automation so it simply sends the packages upstream to FTP
+
*<strike>Import hl2sdk/hl2sdk-ob into Mercurial.</strike>
*Use an SSH tunnel to notify of new packages, so we don't have to do something idiotic like mirrors.pl for the mirror system
+
*<strike>Update all links we can find on the site and in the docs.</strike>
*Redo the package system so only 30 days worth of builds are available
 
*Write a script to help users diff between packages?
 
*Transition C# build tool to vs2k5 during this downtime
 
*Remove the Packages repository entirely (transitioning AMX Mod X to this will be trivial).
 
  
==Phase 2==
+
==Phase 7==
*Move the pdbstore to be centralized client-side.  This can be accomplished by making a more advanced symstore wrapper that will remove anything older than 30 days, and then (in a similar SSH tunnel method as above), rsync all changes upstream to the webserver.  The exact mechanism is still up in the air, but rsync would probably suffice (as long as we can stomach cygwin).
+
'''DONE!'''
*Remove the Symbols repository.
 
  
==Phase 3==
+
*<strike>{{bz|3259}}: We need to update vBulletin to the 3.7 branchThis is potentially a huge task.</strike>
(Maybe)
 
*Add hg.alliedmods.net pointing to our "repository roots," which will be per-projectFor example, hg.alliedmods.net/sourcemod. 
 
*From there, import trunk as "sourcemod."
 
*Branch every major release off as separate repositories, from the appropriate revisions.
 
*Tag minor releases.
 
*Update all links we can find on the site and in the docs.
 
  
==Phase 4==
+
==Phase 8==
(Assuming Phase 3)
+
Expected date: Q1 2009
*Add /metamodsource, import trunk and branches.
 
*Branch every major release off as separate repositories, from the appropriate revisions.
 
*Tag minor releases.
 
*Update all links we can find on the site and in the docs.
 
*Create new repositories for the hl2sdk stuff.  This way, the AlliedModders root organization will look like:
 
**hl2sdk
 
***hl2sdk
 
***hl2sdk-ob
 
**metamodsource
 
***metamodsource
 
***metamodsource-1.0
 
***metamodsource-1.1
 
***metamodsource-1.2
 
***metamodsource-1.3
 
***metamodsource-1.4
 
***metamodsource-1.6
 
**sourcemod
 
***sourcemod
 
***sourcemod-1.0
 
  
==Phase 5==
 
 
We really, really still need a CMS to make this packaging, mirroring, etc business much more streamlined.  But this phase is so far in the future I expect other phases to creep in before it.  This is more a mind dump than anything, but here goes --
 
We really, really still need a CMS to make this packaging, mirroring, etc business much more streamlined.  But this phase is so far in the future I expect other phases to creep in before it.  This is more a mind dump than anything, but here goes --
  
*Moving from flyspray to bugzilla.  After using bugzilla, I'm convinced that its massive feature set far outweights the few permissions issues we ran into.  flyspray is severely lacking, and even though bugzilla isn't the nicest looking tool, it's templated and professionally done.  It has little things that really count, like..
+
*<strike>Moving from flyspray to bugzilla.  After using bugzilla, I'm convinced that its massive feature set far outweights the few permissions issues we ran into.  flyspray is severely lacking, and even though bugzilla isn't the nicest looking tool, it's templated and professionally done.  It has little things that really count, like..</strike>
**Not e-mailing the person who made the change to a report
+
**<strike>The input system isn't broken and convoluted like Flyspray's embedded dokuwiki (which seems to be less a product than a bunch of bugs someone coded)</strike>
**A much, MUCH better attachment system that allows for attachment reviews
+
**<strike>A much, MUCH better attachment system that allows for attachment reviews
**More focused on triaging and patch management than flyspray, which is simply a list of tasks with some basic statistics
+
**More focused on triaging and patch management than flyspray, which is simply a list of tasks with some basic statistics</strike>
 
*Moving from MediaWiki, which (I view) as a very, very outdated piece of software.  From version 1.5 to 1.10 I have noticed no real usability or feature set changes, and the installation/upgrade process is always very buggy for us.  Its administration and permissions model is laughably limited.  A possible target for this is DekiWiki which Mozilla is supposedly converting to.  It's open source and it's actually a CMS, which (perhaps) could kill two birds with one stone.
 
*Moving from MediaWiki, which (I view) as a very, very outdated piece of software.  From version 1.5 to 1.10 I have noticed no real usability or feature set changes, and the installation/upgrade process is always very buggy for us.  Its administration and permissions model is laughably limited.  A possible target for this is DekiWiki which Mozilla is supposedly converting to.  It's open source and it's actually a CMS, which (perhaps) could kill two birds with one stone.

Latest revision as of 22:03, 15 March 2011


This is sort of a major push for AlliedModders to be a bit more streamlined. We're encountering growth problems, especially as our personnel has dwindled quite low recently. This document discusses some changes I'm proposing. Anyone is free to discuss this on the forums or add comments into this document.

Everything related to source code management in this document refers to Metamod:Source and SourceMod. AMX Mod X is considered "mature" and is no longer maintained except for critical or high priority bugs. If and when we do a move past Subversion, it is unlikely AMX Mod X will follow.

The Problems

Short Term

  • We're finding it hard to manage multiple projects in a unified way.
  • There are site disparities, Metamod:Source and SourceMod and AMX Mod X each have entirely different backends.
  • Similarly, each site has a different version of the mirror system script.
  • Subversion doesn't cut it -- Mercurial might handle things better. Subversion basically has no branching or tagging support, and it has no real branch merging capabilities. This is a real problem for SourceMod which has pretty active development.
  • The packaging system is disorganized and fundamentally broken. It's inconsistent, inaccurate, and takes up 1.7GB of wasted space and growing.

Long Term

  • AlliedModders has plans past SourceMod. We need to be able to grow our project base without adding exponential amounts of work.
  • We still don't have a main site
  • MediaWiki still has authentication problems since its handling of usernames is broken
  • Flyspray is buggy, definitely non-production, and doesn't get maintained frequently
  • It's impossible to maintain the AMXX/SM/MM:S sites right now


Mercurial

Subversion doesn't let us branch nicely or create user-specific branches without forever mucking up the central repository. It's difficult to do merges.

An excellent tutorial for how Mercurial works is here.


Phases

Phase 0 - Done

DONE: We've eliminated build numbers and are back on the x.y.z system.

  • Apply new versioning scheme, whatever it may be.

Phase 1 - Done

DONE!

Move from Sente to Iroh so we can get away from ThePlanet and clean up/optimize some infrastructure. This is documented at Iroh Migration.

Phase 2

Expected date: Mid-July

  • Redo automation so the "packaging" versus "sending upstream" processes are fundamentally separate.
  • Redo "sending upstream" process so it simply sends over FTP.
  • Use an SSH tunnel to notify of new packages, so we don't have to do something idiotic like mirrors.pl for the mirror system, or using post-commit scripts on the Packages repository.
  • Redo the package system so only 30 days worth of builds are available.
  • Write a script to help users diff between packages? (Update: No one asked for it.)
  • Remove the Packages repository entirely (transitioning AMX Mod X to this will be trivial).
  • Update symbol server to include binaries and maybe even sourcecode? Removes the need for fetchdll's.

Phase 3 - Done

DONE!

  • bug 3257: Transition SourceMod to vs2k8.
  • bug 3258: Transition Metamod:Source to vs2k8.

Phase 4 - Done

DONE!

  • Move the pdbstore to be centralized client-side. This can be accomplished by making a more advanced symstore wrapper that will remove anything older than 30 days, and then (in a similar SSH tunnel method as above), rsync all changes upstream to the webserver. The exact mechanism is still up in the air, but rsync would probably suffice (as long as we can stomach cygwin).
  • Remove the Symbols repository.
  • Rewrite fetchdlls to support the new layout.

Phase 5 - Done

DONE!

  • Add hg.alliedmods.net pointing to our "repository roots," which will be per-project. For example, hg.alliedmods.net/sourcemod.
  • From there, import trunk as "sourcemod."
  • Branch every major release off as separate repositories, from the appropriate revisions.
  • Tag minor releases.
  • Update all links we can find on the site and in the docs.

Phase 6 - Done

DONE!

  • Import mmsource into Mercurial.
  • Import hl2sdk/hl2sdk-ob into Mercurial.
  • Update all links we can find on the site and in the docs.

Phase 7

DONE!

  • bug 3259: We need to update vBulletin to the 3.7 branch. This is potentially a huge task.

Phase 8

Expected date: Q1 2009

We really, really still need a CMS to make this packaging, mirroring, etc business much more streamlined. But this phase is so far in the future I expect other phases to creep in before it. This is more a mind dump than anything, but here goes --

  • Moving from flyspray to bugzilla. After using bugzilla, I'm convinced that its massive feature set far outweights the few permissions issues we ran into. flyspray is severely lacking, and even though bugzilla isn't the nicest looking tool, it's templated and professionally done. It has little things that really count, like..
    • The input system isn't broken and convoluted like Flyspray's embedded dokuwiki (which seems to be less a product than a bunch of bugs someone coded)
    • A much, MUCH better attachment system that allows for attachment reviews
    • More focused on triaging and patch management than flyspray, which is simply a list of tasks with some basic statistics
  • Moving from MediaWiki, which (I view) as a very, very outdated piece of software. From version 1.5 to 1.10 I have noticed no real usability or feature set changes, and the installation/upgrade process is always very buggy for us. Its administration and permissions model is laughably limited. A possible target for this is DekiWiki which Mozilla is supposedly converting to. It's open source and it's actually a CMS, which (perhaps) could kill two birds with one stone.