Closed Bug 517131 Opened 15 years ago Closed 14 years ago

Set up a bzr.(bugzilla|mozilla).org server

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mkanat, Assigned: arzhel)

References

Details

Attachments

(4 files, 1 obsolete file)

loggerhead init script
2.59 KB, text/plain
Details
do-index.sh
373 bytes, text/plain
Details
bzr-repo.sh
263 bytes, text/plain
Details
xinetd.d script for bzr
385 bytes, text/plain
Details 
By overwhelming vote on the Bugzilla developers list, we have decided to switch away from cvs to bzr (and not to Hg). 

bug 470570 comment 18 describes what needs to be done to set up a bzr server.

I'd be happy to perform the actual initial sysadmin tasks after the server is set up, if I can get root access to the box.

The only tricky part (for me) is syncing the current committers' ssh keys, userids, and group memberships to the system--I assume there's something in place for that that works with Despot to set this up for hg, svn, and cvs?
We have an existing bzr repository for bmo's code on dm-bugstage01.  It would make much sense to get that formalized and on a dedicated bzr server.  Combining servers for both of these would probably be doable as long as we have ACLs in place to control who can commit where like we do on SVN and CVS.  As long as Bugzilla is the only project using this (and bmo), a VM will probably be plenty.  The existing (temporary) Bugzilla bzr server is on a VM, and dm-bugstage01 is also a VM, and they're doing fine.  Getting LDAP set up to be able to grant permissions for it would probably be useful.  Then we can piggyback on SSH keys people already have in LDAP, etc.  (Not to mention that everyone who had a CVS account was supposedly granted an Hg account a while back, which means most of the Bugzilla committers should already have LDAP accounts whether they know it or not).

Note that if we're putting official Mozilla resources on it as well (like b.m.o) we're probably not going to be able to let Max in to set it up.  And my time is probably spoken for until we finish getting b.m.o upgraded to 3.4 (maybe this can wait until then, it'll probably only be a couple weeks).
FWIW, I would pitch this more as a "lets get an official home for our local b.m.o code" and "as long as we have it, we can let the Bugzilla project piggy-back on it".
I agree with dave's pitch.  I need to know server/disk requirements.
Server requirements aren't anything stellar. I imagine that a single CPU and 2GB of RAM at most would be fine.

If we had 20GB of free space after the OS, that would probably be fine.

The server needs to be open for ssh, http, https, and port 4155 (bzr server) to the world.

It needs bzr packages backported from Fedora. bzr 2.0 will be out soon, we may want to even upgrade to that on the server (since the bzr package on the server does have a lot to do with the speed of the server).

Other configuration stuff is mentioned in bug 470570 comment 18.
Oh, and the bzr repository needs to be backed up somewhere, of course.
Assignee: server-ops → ayounsi
Auth on this should be done the same way as Hg and SVN -- restricted shell accounts based on email address that can only access the bzr server, hit LDAP for the ssh keys.  We'll probably need a new group added to LDAP to support it.

EPEL has bzr 1.3.1 it looks like (that's what we're running on the current internal bzr server).  Fedora 11 has 1.14.  The Fedora 11 SRPM refuses to unpack on RHEL5, citing MD5SUM mismatch errors.
Does the native bzr protocol do SSL?  Does it support hooking LDAP for passwords?
(In reply to comment #6)
> The Fedora 11 SRPM refuses to unpack on RHEL5, citing MD5SUM mismatch errors.

  Yeah, I mentioned in #bmo a few days how to get around that:

  rpm -ivh --nomd5 *.src.rpm

  And then do "rpmbuild -ba" on the spec file.

(In reply to comment #7)
> Does the native bzr protocol do SSL?  

  No. That's what bzr+ssh:// is for.

> Does it support hooking LDAP for passwords?

  Possibly, but I wouldn't worry about it. bzr:// will be write-only, and bzr+ssh:// will be the one that can write.
(In reply to comment #8)
> Possibly, but I wouldn't worry about it. bzr:// will be write-only, and
> bzr+ssh:// will be the one that can write.

Assuming you mean bzr:// is read-only, yeah, that'll do.

Wrote: /usr/src/redhat/SRPMS/bzr-1.14-0.2.rc1.rhel5.src.rpm
Wrote: /usr/src/redhat/RPMS/i386/bzr-1.14-0.2.rc1.rhel5.i386.rpm
Wrote: /usr/src/redhat/RPMS/i386/bzr-debuginfo-1.14-0.2.rc1.rhel5.i386.rpm

I don't see an RPM for bzrtools in Fedora, do I care about that one?
(In reply to comment #9)
> Wrote: /usr/src/redhat/SRPMS/bzr-1.14-0.2.rc1.rhel5.src.rpm

  It would be ideal to have a newer version than that, ideally 1.18 (which is the latest F11 update) or 2.0.0 when it comes out. They're both much faster than 1.14.

> I don't see an RPM for bzrtools in Fedora, do I care about that one?

  Um, maybe. On the server side of things, maybe not. There is an rpm in Fedora, though, for sure.
(In reply to comment #10)
> (In reply to comment #9)
> > Wrote: /usr/src/redhat/SRPMS/bzr-1.14-0.2.rc1.rhel5.src.rpm
>   It would be ideal to have a newer version than that, ideally 1.18 (which is
> the latest F11 update) or 2.0.0 when it comes out. They're both much faster
> than 1.14.

Ah, yeah, it's in updates.

> > I don't see an RPM for bzrtools in Fedora, do I care about that one?
> 
>   Um, maybe. On the server side of things, maybe not. There is an rpm in
> Fedora, though, for sure.

That was in updates, also.  They didn't have it in base.  I should have looked there first, actually, fall back to base if they haven't updated it. :)
Wrote: /usr/src/redhat/SRPMS/bzr-1.18.1-1.rhel5.src.rpm
Wrote: /usr/src/redhat/RPMS/i386/bzr-1.18.1-1.rhel5.i386.rpm
Wrote: /usr/src/redhat/RPMS/i386/bzr-debuginfo-1.18.1-1.rhel5.i386.rpm
Wrote: /usr/src/redhat/SRPMS/bzrtools-1.18.0-1.rhel5.src.rpm
Wrote: /usr/src/redhat/RPMS/i386/bzrtools-1.18.0-1.rhel5.i386.rpm
The above RPMs have been added to the mozilla internal repo.
Max, you said in bug 470570 comment 18:
    Copy and install my modified loggerheadd initscript from landfill (it lives
in /etc/init.d)
    Copy and install my Apache configuration for loggerhead (in the
bzr.bugzilla.org VirtualHost).
But I don't have access to this box. Could you attach or send me these conf files?
Thanks!
Max, ping
Attached file loggerhead init script — 
Oh, sure! Here's the script.

You may wish to create a "loggerhead" user instead of running loggerheadd as "bzr". In fact, given all my experience running bzr+loggeread, I'd recommend that (running it as its very own user), at this point. You may have to fix permissions on the /var/log/loggerhead directory to get it working right.
So the final hostname will be bzr.mozilla.org or bzr.bugzilla.org?

Also, Max, could you attach your special apache configuration file for loggerhead?

Thanks!
I think my understanding was that it was going to be bzr.mozilla.org.
Here's the Apache config for a bzr.mozilla.org vhost:

  ServerName   bzr.mozilla.org
  DocumentRoot /var/www/html/bzr/
  Options      -Indexes +SymlinksIfOwnerMatch

  RewriteEngine On
  RewriteCond %{REQUEST_URI} !\.bzr
  RewriteCond %{REQUEST_URI} !^/robots.txt
  RewriteRule (.*) http://localhost:8080%{REQUEST_URI} [P]
  ProxyPassReverse / http://localhost:8080/
Assignee: ayounsi → aravind
@Arzhel: the executable you should serve on ssh login is 

"/usr/bin/bzr serve --inet --directory=/repo/bzr/mozilla --allow-writes"

You will have to change that directory path based on what we are using.  I will leave this assigned to me for now, feel free to grab it if you want to work on this stuff.
Assignee: aravind → ayounsi
What's the status on this? We're just about to branch for another line of development and it would be the ideal time to switch.
It's a low priority right now with everything else in the quarter wrapping up.  When's your cutover going to be?  Maybe that's something we can work towards.
(In reply to comment #22)
> It's a low priority right now with everything else in the quarter wrapping up. 
> When's your cutover going to be?  Maybe that's something we can work towards.

  Well, the freeze cutover is technically today, though we could hold off on checking anything in to trunk and keep trunk as the branch on CVS for a while.
Hey hey. Any ideas on a date for this? I'd at least like to be able to tell the Bugzilla community what the plan is.
Hi Max, The server is almost ready, I'm waiting for some ldap changes. I'll check with oremj to finish it quickly.
(In reply to comment #25)
> Hi Max, The server is almost ready, I'm waiting for some ldap changes. I'll
> check with oremj to finish it quickly.

  Great! If you really need to reach me quickly for something and I'm not online, justdave has my cell number and can text me.
Okay, so the server is basically functional now! :-) That is, folks who want to use bzr.mozilla.org basically can, now.

There are a few plugins that should be installed, though, and I will need to write/look up two other plugins, as well. The plugins that should be installed globally on the server are:

Ability to email a list when a commit hits:
http://doc.bazaar.canonical.com/plugins/en/email-plugin.html

Enable fulltext searching in loggerhead:
http://doc.bazaar.canonical.com/plugins/en/search-plugin.html

The plugins that I need to write/find are:

* Enforce that the committer's email address matches the SSH account they're using.

* Sync back the Bugzilla bzr to CVS.
By the way, here are some sample installation instructions for installing the "search" plugin:

cd /usr/lib/python2.4/site-packages/bzrlib/plugins
bzr co lp:bzr-search search

(If bzr.mozilla.org is a 64-bit machine, though, that first directory will be in /usr/lib64/ instead.)
(In reply to comment #27)
> * Enforce that the committer's email address matches the SSH account they're
> using.

Why? In Hg, for example, I can commit as myself but change the "commit user" (the display portion) to say something else so I can correctly attribute patches written by others.
(In reply to comment #29)
> Why? In Hg, for example, I can commit as myself but change the "commit user"
> (the display portion) to say something else so I can correctly attribute
> patches written by others.

  That's --author in bzr. The "committer" should always be the person committing.
Okay, I've written a plugin to enforce committer==user. 

To store our custom plugins, I think we should create a repository on the bzr server called "bzr-plugins". So here's the commands to create a repository called "foo":

cd /wherever/the/bzr/repos/are/
bzr init-repo --2a --no-trees foo/
chown -R mkanat:bugzilla foo/
find foo/ -type d -exec chmod 775 {} \;
find foo/ -type d -exec chmod ug+s {} \;
find foo/ -type f -exec chmod 664 {} \;
chmod 755 foo/

Where "mkanat" is the user who can create new branches in that repository, and "bugzilla" is the group of users who can commit to that repository.

So, to create this new "bzr-plugins" repository, replace "foo" with "bzr-plugins" in the above commands.

Once the repository is created, I'll upload my plugin and you can install it from there.
Attached file bzr-to-cvs.pl (obsolete) — 
Okay, here is a script that can be run to sync a single revision from bzr to cvs, for a specific branch.

I'm thinking that instead of running this in a post_commit hook on the bzr server (which would seriously slow down bzr commits), I'll run this in a cron job so that it happens asynchronously.
Oh, also, because I'm not quite sure about how correct my original instructions that I have to Arzhel on IRC were, for creating the repos, I'd like to delete and re-create the bugzilla/ and bugzilla/qa/ repos using the instructions in comment 31.

(Both repositories should be owned by me for now. After the initial setup is done in a few days, bugzilla/qa/ should be switched to being owned by LpSolit.)
Seems like repos should be owned by a generic 'bzr' user rather than a specific user (similar to what is done for hg repositories).
The server now has bzr 2.0.3, courtesy of rpmforge, who just picked it up.  They however, don't have bzrtools, so we have our home-built bzrtools 2.0.1 plugin still.  Hope that doesn't conflict too badly.

The repositories from dm-bugstage01 have all been imported and upgraded to 2a format.
(In reply to comment #34)
> Seems like repos should be owned by a generic 'bzr' user rather than a specific
> user (similar to what is done for hg repositories).

  No, the owner of the repository determines who can create new branches. A generic "bzr" user wouldn't give any advantage.
Okay, I have uploaded a plugin called "enforcecommitter" to bzr-plugins/enforcecommitter. 

It should be installed as a global bzr plugin, by checking it out of the repository (bzr co /path/to/bzr/repo/bzr-plugins/enforcecommitter) in  /usr/lib64/python2.4/site-packages/bzrlib/plugins/
Okay, one other thing that should be done is that for every new branch, "bzr index" should be run in that branch's directory at some point, which initializes the fulltext index of the branch's contents, which allows loggerhead to do full-text searching of all commits and files.

Probably the best would be to write a simple cron script that does something like:

find /path/to/bzr/repo/ -type d -not -path '*.bzr*' -exec nice bzr index {} \; 2> /dev/null

And that can run once a day, which should be fine, since I'm pretty sure it will only operate on branches that have just been uploaded to the server.
Okay, so the indexing is working now, but there's a problem. Whenever I commit, I get an error like:

bzr: ERROR: Permission denied: "bzr-plugins/enforcecommitter/.bzr/bzr-search/upload/3787a819ab0bb734200efc346dcb48e2.pack": : [Errno 13] Permission denied: '/var/www/html/bzr.mozilla.org/bzr-plugins/enforcecommitter/.bzr/bzr-search/upload/3787a819ab0bb734200efc346dcb48e2.pack

It looks like the search files aren't getting their permissions set correctly, which is strange. Could I see an ls -lR of /var/www/html/bzr.mozilla.org/bzr-plugins/enforcecommitter/.bzr/bzr-search/ ? And then also an ls -ld /var/www/html/bzr.mozilla.org/bzr-plugins/enforcecommitter/.bzr/bzr-search/ ?
----------------------------
ls -lR /var/www/html/bzr.mozilla.org/bzr-plugins/enforcecommitter/.bzr/bzr-search/
/var/www/html/bzr.mozilla.org/bzr-plugins/enforcecommitter/.bzr/bzr-search/:
total 24
-rw-r--r-- 1 root root   27 Jan 26 02:36 format
drwxr-xr-x 2 root root 4096 Jan 26 02:36 indices
-rw-r--r-- 1 root root  198 Jan 26 02:36 names
drwxr-xr-x 2 root root 4096 Jan 26 02:36 names-lock
drwxr-xr-x 2 root root 4096 Jan 26 02:36 obsolete
drwxr-xr-x 2 root root 4096 Jan 26 02:36 upload

/var/www/html/bzr.mozilla.org/bzr-plugins/enforcecommitter/.bzr/bzr-search/indices:
total 40
-rw-r--r-- 1 root root 39423 Jan 26 02:36 720ceb5a7bb62db76b74134039199ebe.pack

ls -ld /var/www/html/bzr.mozilla.org/bzr-plugins/enforcecommitter/.bzr/bzr-search
drwxr-xr-x 6 root root 4096 Jan 26 02:36 /var/www/html/bzr.mozilla.org/bzr-plugins/enforcecommitter/.bzr/bzr-search
----------------------------

It looks like the +s do not apply to subdirectories:
----------------------------
bzr-plugins]# ls -al
total 16
drwxr-xr-x 4 mkanat@bugzilla.org bz_bugzilla 4096 Jan 25 02:59 .
drwxr-xr-x 6 root                root        4096 Jan 24 15:24 ..
drwsrwsr-x 4 mkanat@bugzilla.org bz_bugzilla 4096 Jan 24 15:12 .bzr
drwxr-xr-x 3 mkanat@bugzilla.org bz_bugzilla 4096 Jan 25 02:59 enforcecommitter
bzr-plugins]# ls -al enforcecommitter/
total 12
drwxr-xr-x 3 mkanat@bugzilla.org bz_bugzilla 4096 Jan 25 02:59 .
drwxr-xr-x 4 mkanat@bugzilla.org bz_bugzilla 4096 Jan 25 02:59 ..
drwxr-xr-x 5 mkanat@bugzilla.org bz_bugzilla 4096 Jan 26 02:36 .bzr
----------------------------



Also: enforcecommitter]# bzr up
bzr: ERROR: Tip change rejected: You are committing as "mkanat@bugzilla.org", but you are logged in to this bzr server as "root". Use the "bzr whoami" command to set your bzr committer name properly.

Does this comes from your enforcecommitter plugin?
> Does this comes from your enforcecommitter plugin?

  Ah, yes, yes it does. For now the workaround is to delete the directory and just get it again using "co".
(In reply to comment #40)
> bzr: ERROR: Tip change rejected: You are committing as "mkanat@bugzilla.org",
> but you are logged in to this bzr server as "root". Use the "bzr whoami"
> command to set your bzr committer name properly.

  Okay, I've resolved this problem--branches now only enforce the committer being correct if they have an enforce_committer option set to True in .bzr/branch/branch.conf.
Attached file do-index.sh — 
Okay, here's a script to run "bzr index" nightly in the cron, and also fix permissions on all the files appropriately. You run it like:

do-index.sh /var/www/html/bzr.mozilla.org
Comment on attachment 423332 [details]
bzr-to-cvs.pl

Okay, I'm storing the bzr-to-cvs script here now:

http://bzr.mozilla.org/bzr-plugins/bzr-to-cvs/files

I've updated it so that it can do multiple revisions from bzr at once, and so that it actually stores a file in the CVS branch that tells us what the current bzr revision is of that CVS branch.
Attachment #423332 - Attachment is obsolete: true
BZR ready, now nagios & backups
Attached file bzr-repo.sh — 
Okay, here's another script--this one creates a new repository and sets permissions appropriately. There's currently something wrong with the "bugzilla" repository, I'm not sure what--maybe the permissions got messed up by the old indexing command before I wrote do-index.sh.

You can use the script like:

bzr-repo.sh bugzilla mkanat@bugzilla.org bz_bugzilla

So, for now, let's rm -rf /var/www/html/bzr.mozilla.org/bugzilla/

and then:

bzr-repo.sh bugzilla mkanat@bugzilla.org bz_bugzilla
bzr-repo.sh bugzilla/qa mkanat@bugzilla.org bz_bugzilla
I didn't realize this until just now, but it looks like the "bzr" server isn't running (or isn't accessible) on port 4155. Attached is an xinetd.d script for the bzr server. (This is for read-only access.) It's specified to run as a "bzr" user, which should be created as a system user if it doesn't exist.

Also, if required, port 4155 on the firewall will have to be opened.
Attachment #408761 - Attachment description: v1 → loggerhead init script
It was ready but commented out. Started.
I think this is done!
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: