The idea for an extra test run with only the modules available from python-minimal came up during the last developer meeting.

• The current check for missing modules in python-minimal is already done, and the build breaks if the list of modules isn't complete.

• The separate test suite run for the python-minimal modules works, but the test cases and test suite need more modules than are contained in python minimal. The approach taken is to modify each test case to import the needed modules:

  import sys
  sys.path.append(<path with all modules>)
  import <module not in -minimal>
  sys.path.append(<path with all modules>)
  del sys

After a test run, the imported modules have to be removed from sys.modules. Unfortunately unloading the modules doesn't work for every module, so you end up with test cases in the test suite, where an import succeeds, because the module is already imported, but not in sys.path (anymore).

The approach to patch all tests works, but seems to be a maintenance burden. There are however some results: time.strptime is currently unusable in -minimal. The C extension imports the _strptime module, depending on gettext, locale and _locale, copy, calendar. Having locale imported and referring to encodings, we will end up including these in -minimal as well. What to do?

1. Include all the referenced modules (150k).
2. Write a time.strptime, based on the C implementation, handling the same as the Python implementation does.
3. Document it and leave it unimplemented.

First solution can be taken by adding the modules, but increases the size of -minimal, second solution maybe adds some incompatibilities, when the complete module set is not available. Third thing could be a solution, if it's documented somewhere.

For the test of -minimal it should be sufficient to specify a list of tests from the complete testsuite, which have to pass or else the build of the package will fail.

Another outcome: copy and operator are used very often in the testsuite, so maybe add them to -minimal as well.

New live CDs for i386, amd64 and powerpc are available here:

• http://cdimage.ubuntu.com/daily-live/current/
• rsync://cdimage.ubuntu.com/cdimage/daily-live/current/

For some reason, network interface detection is failing at the moment, but this is the first time that the live CD is working on all three architectures, so I thought it would be worth an announcement for those who would like to try it out.

More information here: http://www.ubuntulinux.org/wiki/LiveCD (toward the end)

As we all know many people are targeting yelp/scrollkeeper as technologies by which users will be able to access the Ubuntu Documents. This is fine for now since Ubuntu currently only ships a GNOME Desktop. However, I doubt it will always remain this way and wish to warn against lock-in to the GNOME Help system.

In evaluating ways to do things under yelp/scrollkeeper I have found a number of problems that IMO are cause for concern.

1. Limited support for some very powerful DocBook features. For example: Glossary, Bibliography, Index, Profiling.
2. Limited ability to brand and customize help features.
3. Poorly formatted display of xrefs, a key cross-reference mechanism.
4. Implementation of external cross references using ghelp:fooapp is incompatible with HTML and FO outputs.

My concern with these issues is point-to-point:

1. The unsupported features limit our ability to produce comprehensive works and to publish them in both electronic and print formats. To do so we would need to perform an extensive amount of pre-processing prior to transformation to presentational targets.
2. As far as I can see, other than adding a logo here an there, there is not a great deal of flexibility to customize yelps output.
3. A small problem that could probably be fixed in the short-term is that yelp does not render xrefs properly.
4. The method used for implementation of cross document references in GNOME requires a combination of Yelp+ScrollKeeper. When calling an external resource the attribute value supplied is something like this url="ghelp:fooapp". naturally this does not work except under yelp and scrollkeeper. So any target presentational formats would contain broken links.

In view of these problems and the high probability that Ubuntu will support other desktops, I would like to refrain from using anything that causes lock-in to GNOME. Please note, I am not saying we don't support GNOME. I am saying that the docs must run across desktops, GNOME included.

Having said all this, I would therefore like to motion that we target chunked HTML/XHTML as the target format of choice for Ubuntu Documents. The advantages are as follows:

1. Ability to use all DocBook features without regard to platform or user agent technologies.
2. Ability to customize as much as we need or desire.
3. Portability across desktops.
4. Flexibility in deployment under Web based applications (locahost or central server)
5. Reduced technology lock-in.

This morning somebody pointed out that a small change (kernel rev) on the Ubuntu LiveCD images was producing a major churn and resulting in rsync needing to fetch down 175MB (nearly a third of the image).

aaaabbbbccccdd|   |ddeeeeffffgggg   <--- old image
             /     \
aaaabbbbcccc|NEWDATA|eeeeffffgggg   <--- new image

Even when new information is added to a file, rsync has the magic and intelligence to look backwards and forwards through the old file to see it can find a match and avoid any re-downloading. In the example above, rsync copies slightly less of the first half, inserts the NEWDATA, skips a bit and finds a match in the second half. Rsyncing uncompressed files works great.

However, in the case of the Ubuntu Live CD, 95% of the disk is actually a single huge file which is itself compressed (this is what is mounted using the kernel 'cloop' driver. For reference 'cloop' is read-only).

aaaabbbbccccddddeeeeffffgggg      <--- old image data
A4(^4)*6                          <--- old image compressed
|----|
A4(^4)*3NEWDATAE4^4^4             <--- new image compressed
aaaabbbbccccNEWDATAeeeeffffgggg   <--- new image data

What you can here is that whilst the actual images (old+new) are not much different (only the 'NEWDATA' would need copying), in fact the compressed versions are vastly different and after the first 6 bytes there is no apparent correlation at all.

So one solution that would work is to rsync an /uncompressed/ version of the 'big file' on the Live CD. This could potentially be done by using a loop driver at each end of the rsync. The second is to ensure that the image that is produced does not suffer like the example above.

aaaabbbbccccddddeeeeffffgggg      <--- old image data
A4^4^4^4^     4^4^4               <--- old image compressed (ignore spaces)
|----|        |---|
A4^4^4NEWDATAE4^4^4               <--- new image compressed
aaaabbbbccccNEWDATAeeeeffffgggg   <--- new image data

Here, the decompresser has not changed at all, but the compressor has been told to synchronize the stream every 4 input bytes. This has made a larger compressed version (about 0.07% larger in the real-world), but rsync can now see matches before and after the New Data in the middle.

This idea is already built into 'gzip' and can be enabled with the '--rsyncable' argument. In Ubuntu and Debian, all the '.deb's are built this way to reduce rsync-traffic. The install CD ISO images are built by Jigdo by concatenating (connecting together) all these '.deb's and adding the filesystem magic around them. This is why the install CD syncs fine between revisions.

The gzip 'deflate' algorithm is used all over the place and the library 'zlib' that does the actual work has spread further than gzip; it's used by PNG, compressed webpages and in the kernel for PPP, compressing the boot kernel and by 'cloop'---the compressed loopback driver that handles the Live CD filesystem. (Gzip is compiled with it's own copy of 'deflate.c' and is not linked against the shared zlib library).

Since there are no changes to the decompresser, only zlib itself needs patching for the compression stage (and only the copy linked to the cloop tools); the 'create_compressed_fs' command in 'cloop-utils' can then use this without any modifications. Matt added that the compressor in use for Ubuntu images is really the newer 'advfs' installed with a different name.

A Hungarian distribution hit something similar (with apt-rsync) a couple of years ago, ported the original '--rsyncable' patch and added a simple hack to test an environment variable to transparently enable the sychronisation code without the application needing to know or be recompiled: http://lists.debian.org/debian-devel/2003/07/msg00462.html

Patches 2a, 2b & 3 are the patches to pull in, the last of which is the environment variable patch, but I'm not sure if that would get accepted upstream, whereas adding an extra flag to the 'level' parameter of zlib's compress2() might be more palatable.

Lamont says he's going to try and work on it (I guess not having broadband is a motivator!)

This is untrue, .deb files are not rsyncable and can change wildly between version-to-version.

dpkg-deb does not call gzip, but is statically linked to zlib which does not support 'rsyncable' images. Colin and I actually talked about this only a week or so ago, and he filed a bug on the zlib Debian package to include support for rsyncable output.

Thanks to some new autodetection logic from Daniel Stone, inspiration from Joey Hess, and support from Colin Watson and LaMont Jones, the current crop of live CDs will now automatically configure the X server using the same tools used during the installation process.

This consolidation makes it simpler for us to maintain this component, maximize our support for graphics hardware, and facilitate testing in the community. If the live CD detects your hardware correctly, there's an excellent chance that the installer will as well.

This also makes the live CD a great way to get a preview of some of the new features available in the Hoary desktop, without changing your Warty system.

Check it out here:

http://cdimage.ubuntu.com/daily-live/current/

Let us know how it works for you.

Caveats:

• The amd64 CD suffers from a mismatch of kernel and modules; this should be fixed with the next daily build
• There is a known bug where some nVidia cards are configured with the UseFBDev option enabled, when it needs to be disabled. Daniel Stone is working on a fix for this.

The new Planet Ubuntu is ready to rock!

• http://planet.ubuntu.com/

It now aggregates member and developer feeds, and includes a cool aggregate of various important Ubuntu news feeds (which you can subscribe to with RSS and OPML). This ends up being more useful than the news feed on the website (which is one of the sources it aggregates) so we'll work on putting it up there, too.

If there are any bugs or problems with it, let me know. Probably best to reply on this thread. :-)

• http://planet.ubuntu.com/rss2.0.xml
• http://planet.ubuntu.com/rss1.0.xml

Today we are announcing the most exciting competition since Willy Wonka put heavy metal in chocolate bars and purged rude children with coloured sugar.

No, we're not giving away gold tickets for a trip to the ISS, although that would be cool... We're giving YOU the chance to give www.ubuntu.com a total look'n'feel makeover, so it can be the most attractive and usable project website EVER!

Canonical is sponsoring the competition, offering a US$1000 first prize for the winning design!

Competition Guidelines

• Goal: We're looking for a fantastic, new look'n'feel for the current www.ubuntu.com site, which is based on Plone 2.
• Entries do not have to be completed Plone skins! You may send mockup images, html+image samples, or if you really want to, an example Plone skin. They will all be judged equally - we're looking for an innovative look'n'feel more than ability to create Plone skins. :-)
• Designs should cover as many Plone widgets and structural elements as possible, so we know how to apply it to a complete skin. See the first documentation link below for more about this.
• Entries must be entirely your own work, or based on clearly licensed, credited, DFSG compatible works. The final website look'n'feel may be based on a combination of entries.
• Entries must include the Ubuntu logo. :-)
• Send your entry to jeff.waugh+webcomp@ubuntu.com or simply reply to this mail.
• Entries close on January 31st, 2005.
• sabdfl's judgement of the competition is final, even if he polls the community for input. :-)

Documentation and Resources

Here's some documentation and inspiration to get you started with Plone themes...

• http://plone.org/documentation-old/howto/CreatingACustomSkin
• http://plone.org/documentation/howto/creating-custom-skins (Plone 2)
• http://plone.org/development/teams/ui/p2uicookbook/ (Plone 2)
• http://www.ploneskins.org/ (for inspiration, mostly Plone 2)

Array CD 3 is ready. This is the third in a series of milestone CD images, released when they're known to be reasonably free of showstopper CD-build or installer bugs, while representing very current snapshots of Hoary. You can download it here:

• http://cdimage.ubuntu.com/releases/hoary/array-3/
• rsync://cdimage.ubuntu.com/cdimage/releases/hoary/array-3/

See http://www.ubuntu.com/wiki/Archive for access instructions. I recommend rsync if possible, as you can then download future images based on this one to save bandwidth.

Pre-release versions of Hoary are not encouraged for anyone needing a stable system or anyone who is not comfortable running into occasional breakage. They are recommended for Ubuntu developers and those who want to help in testing, reporting, and fixing bugs.

Some notable installer improvements and bug-fixes in this release:

• End-to-end secure netboot installation: all our Release files are signed using the archive signing key (1024D/437D05B5) or the CD image key (1024D/FBB75451), and the installer verifies these signatures at each step. (The netboot kernel and initrd themselves are contained in the CD images, which are accompanied by MD5SUMS and MD5SUMS.gpg files).
• Timezone and username/password questions moved to the first stage. The question about installing packages from the network remains in the second stage for now, but will eventually be moved as well.
• Rudimentary rescue mode added: boot with the 'rescue/enable=true' parameter to use it. Its UI still needs significant polishing work.
• Default debconf priority dropped from critical to high, and several questions adjusted; this fixes some automatic installation scenarios, makes it easier to merge changes back and forward with Debian, and makes it possible to share code between the first and second stages.
• Example sources.list lines for security/universe fixed.
• Support for Smart Boot Manager should work better now, although I haven't tested it personally.
• Size requirement for a USB drive used to boot the installer reduced to 8 megabytes.
• Fixed installation on large filesystems with long device names (#4875).
• Much improved and more flexible kernel selection logic.
• Fixed default hostname when network configuration is skipped in first stage (#2844).
• Fix part of Array CD 2 erratum: the framebuffer should now reliably be loaded.
• Several fixes to the ia64 installer; much work remains to be done.

Known installer issues:

• On my amd64 system, grub enters an infinite loop trying to load stage 1.5. I've never seen this before, and suspect hardware problems, but if it affects other people too I'd like to know about it.

If you're interested in following changes as we further develop Hoary, have a look at the hoary-changes list: http://lists.ubuntu.com/mailman/listinfo/hoary-changes

Bug reports should go here: https://bugzilla.ubuntu.com/

This request for comment is the start of a proposal to define a flexible and easy to navigate infrastructure for the Documentation Team Web site.

The core of my proposal is centered on the first page of the Documentation Team https://www.ubuntulinux.org/community/teams/documentation and on addressing the information requirements of its users (audience).

An old tarball of our HTML-rendered documentation is online at: http://people.ubuntulinux.org/~mako/docteam/

This is an old tarball, and the upload is done by hand with me generating the tarball and sending it to Mako who then puts it there.

Next step is making a nice alias for that page (docteam.ubuntu.com is the first proposal, which could also be the one that stays unless someone has further input).

Then I'll try to see with Mako how to automate that.

In the meantime, me and Mako will keep those pages updated by hand every now and then.