X-Files: A RISC OS image filing system

Last Wednesday I finally cracked. In my long relationship with Acorn
machines I'd seen "Directory full" one too many times. Here's the result.

X-Files is an image filing system. An image filing system is an FS which can
open a file and treat it's contents as a nested filesystem within the
filesystem containing the file.

X-Files supports long filenames (up to 256 characters internally, although
you'd be hard pressed to create a file with a 256 character name under RISC
OS) and an unlimited number of files per directory. Other than that it does
the things you'd expect an image filing system to do.

PROS AND CONS

If you want to have long filenames and directories which contain lots of
files you can either a) write a set of patches which emulate this in an
existing filesystem using hidden files and the like or b) write an image
filing system.

Solution "a" is transparent to the user and can benefit from the underlying
filing system's proven ability to keep track of directories and files
without getting in a muddle.

Solution "b" requires a whole new filing system to be written, isn't
transparent to the user and is more likely to lose all your data if the
person who wrote it screwed up.

Obviously I've chosen solution "b".

Purists may be offended by the fact the X-Files is written in C. Tough. I
needed it quick and although ARM assembler is a dream to write I don't
reckon I could have had it working in two and a bit days if I'd written it
in assembler.

IMPLEMENTATION

X-Files consists of a storage allocator which keeps track of which bits of
an image contain what, a directory handler which keeps track of the file
hierarchy and a cache manager which keeps recently access bits of the image
in memory in case they're needed again (they frequently are).

The storage allocator is pretty simple minded. If an object (file or
directory) can grow in place it will otherwise it moves to the end of the
image so that extending it is simply a matter of extending the image file.
This leaves big holes are squashed out when the wasted space exceeds 1/3 of
the image's total size. In the normal course of events compression will not
occur more frequently than once every thirty seconds.

PERFORMANCE

Most things are a bit slower than the underlying filesystem, although raw
data transfer rates are not measurably affected and a few operations are
actually quicker. At the moment the cache manager writes through in an
attempt to try to avoid losing your data. It has been designed to be equally
viable in delayed write mode which will I expect make X-Files quicker
overall than the underlying filesystem.

*** STOP PRESS ***

0.54 includes write behind caching which does speed things up a lot.

IN USE

When X-Files gets out of beta (and I have time) I'll write some spiffy
extensions to the filer or something. For now though the only thing you
really need is:

How to create a new X-Files image

Create an empty file (length 0) and set it's type to X-File like this

   *Create SpogMunge
   *SetType SpogMunge X-File

Now you can open it and start sticking files in it. Easy no?

Actually, you can only set the file's type to X-File is you've arranged for 

   *Set File$Type_B23 X-File

to have been executed as well. You could of course just set the file's type
to &B23.

This is very much development code. In particular it has a load of asserts
and other debug code in it which will, when things get sticky, attempt to
produce debug output. To see this output you'll need to run !Tracker
(included) when you're running X-Files. When reporting bugs I'd be grateful
if you could send any associated output (you can save the contents of the
Tracker window -- incidently don't let the poor quality of Tracker prejudice
you against X-Files -- it's the first desktop program I wrote when RISC OS 2
appeared and I haven't updated it in years).

DISCLAIMER

At the time of writing X-Files is three days old. I use it to store really
important files pertaining to the project I'm working on. I trust you're not
that reckless, but if you are and it all turns nasty, don't come crying to
me.

HISTORY

 0.50  25.03.96  First release to beta testers

 0.51  26.03.96  Second release to beta testers. Found and fixed a bug
                 whereby compaction could cause broken directories and
                 other 'things moving when other things didn't expect
                 it' sort of bugs.
                 
 0.52  27.03.96  Fixed problems with cache index table getting munged.
                 When entries became invalid (image closed or shrunk),
                 I was setting their owner and offset fields to zero
                 which made them compare incorrectly with their
                 neighbours. The knock-on effect of this was potentially
                 catastrophic!

 0.53  30.03.96  Various bug fixes. Nothing data threatening -- just
                 silliness really. The worst one was due to me not
                 having fully considered the implications of zero
                 length chunks floating around the filesystem.

 0.54   1.04.96  Write behind caching enabled. The cache is flushed two
                 seconds after the last access.

 0.55   2.04.96  Humble pie time. Didn't run all the tests did I?
                 I think it's fixed now...

 0.57  22.02.97  Recompiled with write behind turned off again which seems
                 to improve reliability somewhat :-) Release now includes
                 Nicholas Clark's X-Recover utility.
                 