Mail: richard ätt 2009.atterer.net
Last change: 2009-02-26

Jigsaw Download

Jigsaw Download, or short jigdo, is a tool designed to ease the distribution of very large files over the internet, for example CD or DVD images. Its aim is to make downloading the images as easy for users as a click on a direct download link in a browser, while avoiding all the problems that server administrators have with hosting such large files.

Article about jigdo on heise.de (c't-Magazin; German)

jigdo is Free Software, distributable under the GNU GPL.

________________________________________

Download

Help

________________________________________

Why is it needed?

jigdo was inspired by the problems that Debian Linux had with the distribution of its CD images. Debian is a 100% volunteer-driven organization which depends on donations to finance its infrastructure. For this reason, it also doesn't own any of the 300 servers around the world on which the distribution is mirrored.

For some time, the size of Debian has been a problem - Debian supports 11 different processor architectures, more than any other Linux distribution, and for every single one of these architectures, there are more than 10000 software packages - again more than any other distro. Today, a full Debian mirror needs roughly 100 GB of disc space!

Debian also offers the distribution in the form of CD images (and more recently DVD images). If these images were distributed as full .iso images on the mirrors, the size of a Debian mirror would double to 200 GB (or even triple to 300 GB if there were also DVD images) - nobody is prepared to mirror such a huge amount of data!

jigdo solves this problem by not requiring that the full image be stored on the servers. Instead, it can download the individual files from a normal Debian mirror and only assemble the image on the user's machine. Additionally, it can also do other useful things like helping with the mirror selection and upgrading images.

________________________________________

How does it work?

When someone has produced a large file they wish to distribute (for example, a CD image), they process this file with the jigdo-file tool. Among other things, jigdo-file creates a file with a .jigdo extension. To download the original file, a user only needs to tell the jigdo-lite download tool the URL of the jigdo file, it will extract all the necessary information and perform the download, fetching the data in many small pieces, possibly from several different servers.

So far, this is not exciting; there are already several other programs which allow you to split and reassemble large files, or which make their download easier. Also, at the moment only the shell script jigdo-lite can be used for downloading - the final jigdo download manager shown below will be much more comfortable to use.

[Screenshot]
The jigdo download tool (under development - not usable at the moment)

However, jigdo's approach is unique due to the way it identifies the small pieces that the large file consists of: For instance, if the file is a CD image, the individual files on the CD are contained somewhere within the image file. jigdo-file is capable of finding these files, so if copies of the individual files on the CD are stored separately on the server, jigdo-lite can be made to download these files one by one, and assemble the CD image on the fly on the user's computer. This scheme has several advantages:

  • The large image does not need to be stored on the server, instead only the many small files contained in the image. This works with CD images, DVD images (both ISO9660 and UDF format), uncompressed zip files, tar archives... jigdo-file is format independent.
  • In spite of the above, jigdo creates a bit-exact copy of the original image on the user's machine. (To achieve this, the directory data, boot block etc. of the image is stored in a special .template file which is distributed alongside the .jigdo file.)
  • There is full control over where jigdo-lite will download the individual parts. It is possible to define mirrors so users can choose their nearest mirror.
  • jigdo relies on standard HTTP/FTP, no support for special protocols like rsync, and no installation of special software is required on the mirrors.
  • jigdo-lite supports resuming aborted downloads, or continuing the download with another mirror if the current one proves slow.
  • It is possible to "upgrade" a CD image: If a new version of an image is released, jigdo-lite can download only the data that has changed, the remaining data is read from the outdated version.
  • It is possible to release modified versions: If someone distributes a CD image and you have made a few small modifications, e.g. added some files, then you only need to upload the data for your modifications to your webspace. All the data from the original CD image is still fetched from the original site.

If you have read this far, you may be confused now, asking yourself what jigdo does and doesn't do. :-) It seems that the jigdo concept is a bit difficult to grasp at first. If you're still interested, have a look at the respective section of the HOWTO and at the Examples section of the jigdo-file manual.

________________________________________

Further jigdo resources


To top - Unless noted otherwise, all graphics, programs and text © Copyright 2009 Richard Atterer. All trademarks are the property of their respective owners.