L'essentiel

CRIL: Anciens projets de Marc SCHAEFER

Introduction

Certains projets prédatent ma nouvelle nouvelle organisation, et n'ont pas été retravaillés récemment. Ils sont alors stockés ici, mais seulement les plus importants. Dans le doute contactez-moi.

Certains seront probablement repris plus tard par moi. Encore qu'à voir, 4 ans après, ce n'est pas gagné.

Anciens projets

mfs: the migration file system (FIST)

Introduction

The goal of the mfs project is to implement a new Linux filesystem which presents files to the applications. Those files are either present on the local disk (on e.g. a standard ext2 filesystem), or on migrated storage.

The latter includes tape, tape libraries, slower disks, network file servers, CD-ROMs, etc. The mfs filesystem supports writing, which is implemented by first deleting or unmigrating the file from storage depending on the operation (create, append, etc) if it was migrated, and then absolutely normal I/Os.

When the underlying filesystem gets full or below a certain threshold, a migration process is called which will atomically migrate files to tape (this will support, by design, read or read/write access to the file while it's mirrored, but in that case the migration will be aborted later and the index for that file not created).

Only an index (containing inode and location information) is stored in the mfsd user-level process for migrated files.

Implementation

The first implementation has the following assumptions and design goals:

mfs is implemented as a stackable-filesystem (using FIST), with ext2 or any other POSIX filesystem supporting attributes (lsattr, chattr) as underlying filesystem. By POSIX we mean that the standard assumptions of atomicity of rename(), link() and others apply. Implementing as a stand-alone filesystem (e.g. by modifying ext2) might be done later, e.g. to implement a slightly faster system, or to diminush the complexity. However that's not a priority, especially now that new advanced filesystems are being implemented (ext3, etc). By using the concept of filesystem stacking (or wrapping), we can benefit from all this, and from very well tested code.
mfs only migrates and unmigrates complete files. Some previous work by me (e.g. the tapefs read only userfs-implemented fast-indexed tape filesystem) have implemented partial unmigration, but forgetting about this special case (only useful for very big files only accessed partly) makes the design much cleaner.
everything else than the wrapping itself is done as a user-process. However, data I/O is done as almost the same speed as the underlying filesystem (assuming that the file is unmigrated), except for a few microseconds for stacking overhead. Countrary e.g. to tapefs where all data has to transit through this user-level process. Implementing the work in a user process makes it easier, and safer. Implementation of the mfs filesystem itself (in the kernel) will ensure that if that user process dies (except because of a kernel problem of course), all running operations are cleanly finished with an error, and that the fs can e.g. be unmounted.
migrated files can reside anywhere, the unmigration and migration processes are done through external programs or scripts, possibly finally to a tape library migration/queueing/reordering daemon, etc. This is not in the initial design, apart from the fact that the user-level process (mfsd) is not supposed to do the migration/unmigration by itself, since it must handle index queries very fast, and able to launch multiple unmigration/migration requests at the same time (assuming they are from independant sources).
this implementation will not support versioning (snapshots).
backing up is not in the initial specification. It will of course be implementable by dumping the mfs itself. Better ways including tape reclamation may be implemented later.

More information can be found in the SPEC file of the mfs tarball.

Source download

Flash! The mfs initial-public-test-release source is available, along with a more recent snapshot. Note that this is ALPHA quality software. Please read the README in the archive.

PLEASE only download the archive if

you are willing to patch a 2.2.13 kernel with FIST
you want to risk to install/compile ALPHA quality software which has not all features, by far, implemented yet
you are willing to spend some time tracking problems, providing patches and ideas.
you are willing to join the mailing-list.

At this time no binary packages are available. Please do not request for them yet.

Current status

The general concept has been proven by implementing the mfs fs in the kernel and a very small mfsd daemon. At this time only a very simple test was done (migration and unmigration without too many constraints, listing of a migrated file, and mfsd not yet there, and mfsd died unexpectedly).

We also implemented the index in mfsd, the tar importer (mfs_import) which brings a tar file or file on tape's references in the index, a simple unmigration process from .tar.gz files, and the ability to start a migration when low on space conditions (errors) occur in write().

Next steps in order:

Port to the new bug-fixed FIST release.
Make mfsd multithreaded.
Implement the remaining truncate/open(O_CREAT)/rename/unlink functions
Test the low-on-space recovery.
Update migration/unmigration so that we can use filed.
Fix permissions, owners, and checks (e.g. mfs_migrate, etc).

Then, maybe:

Implement pre-migration (copy to tar, but no truncate()) based on a regular df.
Fix the use of attr flags somehow (e.g. request new attr in the standard kernel).
Cleanup the way we do data I/O and commands in mfsd.
Quota support in a way which works and has a migrated quota too.

Support

If you want to contribute (ideas, code, bug reports) or if you need commercial support with Linux and/or mfs, you can contact me.

Other related software

A few related links:

The filed (File Storage Daemon), which provides the actual storage services for mfs migrated and cached data.
The list of GPL software I created or contributed to recently, which notably contains links to tapefs and userfs.

filed: the file storage daemon

Introduction

filed's goal is to abstract the file concept for clients. Clients can send or receive files (stream of data, with known, guessed, or unknown size when creating, size and offset when reading) which are stored within filed's hierarchical storage. A database of where and how the files are stored is kept within filed and is unknown to clients. Clients can however specify policies (I/O priority, bandwidth even maybe, collocation strategies: same medium, contiguous, and retension strategies: how long to keep some data). Clients can also discard unused data.

filed manages the hierarchical storage transparently: for example, some parts of a file can be stored on a local harddisk, some on a remote file server, and some in a tape changer. The most important part of filed is its scheduler, which decides what to do next. This might include deleting old files according to policies or client orders, and do reclamation of sequential medium.

Applications for filed include: backend for backup software (Amanda will be ported into filed, thus solving many of its shortcomings), and for migration filesystems such as mfs. More applications could involve guaranteed rate I/O: while this is not an initial goal, filed's scheduler could be amended to also support this class of problems.

Security (authentification, encryption, data integrity) are one of the design goals of mfs, allowing clients to store data even in an unsafe (ie standard) environment.

Implementation

Not yet documented.

Source download

Not yet available.

Current status

Specification was written. Quick test implementation is being written. The server can now store and retrieve parts, and schedule operations. Don't expect anything before september. Integration with mfs or Amanda not yet planned.

In details, what remains to be done till first alpha release:

Integrate the changer and the taper in the server system, including pipe protocol.
Implement server TCP commands (status, label, etc)
Implement user commands (label, etc)

Then, we will need to:

Evaluate what to do for Amanda support
Evaluate what to do for mfs support
Implement it.

Amanda: extensions

Introduction

Amanda is not my project. It was really developped at the University of Maryland, and is supported by the Amanda group. I have used Amanda and I have found it very useful. However, I think there are a few lacking features to be the best free software for backup on *NIX (especially Linux), and also a few others to be the best backup software of all plateforms and software types.

For example, the problems I have with the stock Amanda:

Inability to compare file-by-file backups. This is a very important feature when you want to trust backup software. Because Amanda can use e.g. tar, you can always do the comparison yourself (manually), but it's best when it's integrated, and also there are a few problems with the way Amanda uses tar (infact it's a bug with tar and the --sparse option) which makes --compare inoperational anyway with the stock Amanda.
Inability to append to a running tape. This may look suspicious at first sight: appending is inherently dangerous, and may prove of little benefit since Amanda's schedule will smooth backups. However, in some cases, for example when the size of the complete run divided by the number of runs is much smaller than the size your tape (e.g. you backup daily 1 GB on a 20 GB DDS-4 device, not accounting compression), this may prove to be a waste.
Inability to backup filesystems bigger than a tape without tricks. Tricks involve tar and exclude lists. Complicated, error-prone, not possible for very big files. Corollary: inability to span tapes (a backup which doesn't fit is retried completely on the next tape unless it's too big anyway).
Inability for backups to use the spool partly (ie a 20 GB backup will have to be streamed directly to tape through network, if the Amanda spool is less than 20 GB, where an approach doing alternates of dumps and saves could be nice (faster)).
Inability to control autochangers for data retrieving. Corollary: a few issues with data retrieving.
Security model either too complex or problematic (Kerberos), or very soft (BSD rhosts).

Some of those shortcomings might be fixed in the upcoming Amanda 2.5.x release (the security model and dump API).

Additionnally, I have done a few patches (with impulsion from DILOG SA, a hardware manufacturer of DAT autochangers I worked with, usually developping firmwares) to make the situation better.

My patches are available here. They include: support for tape appending, support for changer control also for retrieve, and support for file by file verifying, along with a few patches. They are to be applied on top of Amanda 2.4.2-beta1. I also provide Debian potato and slink packages, along with full source or patched source. This is fully unsupported and without warranty, unless you want commercial support for it, please contact me in case you are interested.

Now, I am in the writing, for the mfs project, of a generic file storage daemon, filed, which I hope will also offer useful services to Amanda (infact it should close all the remaining issues, including the security ones). Implementation

Please look at the patches, especially in the doc/ subdirectory, where all the patches are documented.

Source download

Look here.

Current status

The current release is ready for production on Debian GNU/Linux 3.0, with kernel 2.4.20. It is being used at least at three sites with DILOG autochangers.

Support

CRIL provides commercial support (you have to pay for this). If you buy a DILOG autochanger from DILOG SA, please specify you want Linux support too.

However, I am also willing to fix problems caused by my patches, feel free to send me bug reports (or post them to the amanda-hackers mailing-list). However I can't offer any warranty. That's a BSD license, after all.

If you want to contribute (ideas, code, bug reports) or if you need commercial support with Linux and/or Amanda, you can contact me.

vgetty: voice mail and extensions

Introduction

Welcome. This is the vgetty maintainer homepage. The current vgetty maintainer is vgetty-maintainer@alphanet.ch vgetty is a UNIX package, which, coupled with mgetty allows to do many things to your modem, such as:

sending and receiving faxes (sendfax)
receiving UNIX logins, FTN (FidoNet^?) and PPP calls automatically
send and receive voice messages like an autoresponder (vgetty)
build a complete voicemail system, including WWW gateway (mvm)
callback
discriminate the called or caller number and take actions on it, such as presenting a different service on different numbers, not answering calls not coming from trusted sources, etc. This works notably with ISDN.

More details

vgetty handles all the voice-modem specific handling and presents to application programs an uniform, modem-independant, voice library. Supported modems include many standard serial modems, such as ZyXEL^? (1496, 2864), and many more. Administrivia

We suggest you get on the mgetty mailing-list. Any patches for mgetty should be submitted to the mgetty maintainer. Any patches for vgetty should be submitted to the vgetty maintainer.

Archives for the mgetty mailing-list, which are fully searchable, are available. Please use that facility before asking questions. Available patches Here you will find patches on some specific versions of mgetty/vgetty, possibly the latest.

Current vgetty patch tree. Please apply to mgetty-1.1.27. All patches contained here in the main patch file have made their way into mgetty-1.1.28, except maybe some of the secondary patches files. Secondary patch files:
- called-ID discrimination. This allows, notably on ISDN with a ZyXEL^? 2864I, to ignore specific calls depending on called or caller MSN by adding this information to the cnd script parameters. This patch is temporary until complete CND support is implemented in mgetty. I use it and it works. This patch shrinks and shrinks as more and more features are merged into the stock distribution. Note that although the patch is old and doesn't apply cleanly, I am still using it on recent versions, too.

Reste

Original software

This is the UNIX software I have written recently and which I put under the GPL license. This list is incomplete, but not too far from complete. Software which is integrated to other software such as ptylogin, for example, which is in mgetty, are not on this list.

mvm: The Minimal Voice Mail system, supporting multiple mailboxes, mail forwarding/copying, WWW bidirectionnal interface, on standard vgetty-supported voice modems.
serial monitors: Two small programs to debug a serial link with a PC and two serial ports. This can be used to analyze protocols, for example.
lsdir: File download browsing utility, and area lister.
IP accounting scripts: Two versions of a set of IP accounting scripts for Linux, to execute actions when traffic thresholds are reached, and to do save data. Also see another version.
virtual_connection: Software which redirects a modem connection to TCP, or which can act like a TCP software proxy.
benchio: A raw IO SCSI read/write (destructive) benchmark, to evaluate the speed of a device compared to LBA (can be used with gnuplot to plot a decreasing speed over block address graph). Includes a Linux kernel driver. This was developped under the GPL, with support from Distributed Logic SA.
mb86606: A Linux SCSI low-level initiator driver for the Fujitsu MB86606 SCSI chip. The driver works (but is suboptimal). Note that because of bugs in the Fujitsu MB86606 SCSI chip when used in PCI mode (ie with Linux), I strongly suggest you do not buy this chip. This was developped under the GPL, with support from Distributed Logic SA.
plt: This software was originally written within Distributed Logic SA to test DILOG-manufactured autochanger products, on Linux. Now it is available on Linux and on Solaris to control autochangers. Especially on Solaris, to my knowledge it's the only free software which supports SCSI-ID based autochangers (as opposed to LUN based autochangers). It's preliminary goal is still testing, but you can use it in conjunction with e.g. multivol to change tapes.
multivol: Originally written in 1995 to fix a bug with Linux's cpio (unable to restore from cpio written multi-tapes), it has been extended to provide to any software the ability to write or read to multiple tapes in sequence efficiently, and with the support for autochangers. The latest version supports volume labels. Although it is obsoleted by fixes in cpio and tar, it is still useful for using with autochangers. Although using the name of a RISCos's utility, this bears no link (either implementation or specification) with it.
UPS daemon: Because I didn't have any more serial port on my system, I have used the mouse serial port also for UPS signalling. I have created a special cable with DTR, RXD and GND from the computer to the mouse (DTR is there for mouse power), and with RTS, CD RI and GND to the UPS in a special way: RI is connected to the ``power not present'' signal, with a 18kOhms pullup to the RTS signal. CD is connected directly to the ``battery low signal'' (this might prove wrong, maybe we also need a pullup here). RTS is forced high by the UPS daemon software. When RI drops, a shutdown of 4 minutes is initiated (my UPS lasts 10). If it rises again, we cancel the shutdown. If CD goes low, we immediately shutdown. We tested it and it works, except we didn't test yet for battery low (CD). Additionnally, we could connect TXD and reverse-engineer my UPS protocol to be able to shutdown it by software. Note that for many UPSes, as long as you have a free serial port, you don't need those contorsions nor this software: see the UPS-Howto for details.
tapefs: At this time, this implements a read-only filesystem using a tape device as underlying device. It supports forward and backward fast seeks, DDS-2 locate, and has an index mechanism so that the file system can be mounted very quickly. The end goal for this software is a read/write tape filesystem, with index, deletion, kind of versioning, and data being cached with a filesystem. At this time implemented with userfs. Maybe will be implemented with a stackable filesystem. If you want, this is your poor man's DMF, in a way.
mfs: The migration file system. An implementation of an HSM-like system under Linux (virtual storage) at the file-level.
filed: The File Storage Daemon. A daemon which abstracts the concept of files for client processes. Data for those files might be cut in pieces, stored partly or completely on remote, local or slow (on, near and off-line) medium. This is a companion project for mfs and Amanda.
fake_io: Demonstration code to show how to intercept open()/close()/read() with a preloaded shared library (no kernel code change).
auth: Demonstration code of a MD5 hash-based non public key (server, client) pair authentification system.
crypt: Demonstration code of (lib)DES crypt use.
lp: Direct I/O port access example, reading and writing from lp port, bypassing the driver.
cd_changeur: Control of a home-made hardware one CD CD changer through Perl and a parallel Port. Project realized with jordy (hardware), see the MPEG. Manque malheureusement les schémas.
cd_changeur_v2: Control of a home-made hardware multiple CD (sequential only) CD changer through Perl and a parallel Port. This version is better, and has better documentation (e.g. partial schematics).
start/stop SCSI disk: Using the START/STOP ioctl(), allows to start or stop a SCSI disk.
webstat: Give server and OS statistics from a list of domains.
no-ip-stack: Very simple IP stack for embedded systems, interfaceable to a UNIX OS through ptys and slattach, this is very easy to test. At this time, the stack answers to ICMP ping, that's all.
search_engine: A simple and efficient search engine for mailing-lists and newsgroups.
sondage: A simple system to create, administer, summarize and reply to polls. Written in Perl for PostgreSQL^?.
Contact DB: A simple system to manage, through PostgreSQL^?, contact lists and to generate either labels or enveloppes. Provided with a small Perl abstraction module, Perl and shell script examples, templates and documentation.
Paykreator Demo: A simple system to create payment lists in a PostgreSQL^? database, and export them to the OPAE format. Includes a simple GUI.

Patches

vgetty

I am the new official maintainer for vgetty, and I fixed a few bugs and added new features since I took over (some time in 1998).

xmodem-128

Patch to make lszrz support XModem-128.

buffer-patch

Patch to make the SuSE-distributed buffer program able to display the speed with the -S option. A slightly modified version of this patch is now in the Debian version of the buffer software.

ssh-uucp-pipe-diff.gz

Patch to allow ssh password exchange with UUCP over ssh in pipe mode.

IP-over-SCSI

Enhancements and fixes to Linux's IP over SCSI so that it works in

1. 2.x, and so that it works better (fixed a few bugs). This is still a work in progress, especially in the low-level SCSI drivers area.

userfs

Patches to make userfs work with SuSE^? 6.0-libc6 (and many general fixes). Userfs is basically dead and full of bugs. Look at podfuk now.

samba-pipe

Patch to samba to allow smbclient to get and send to/from files which are pipe, so that tar .... | smbclient -c 'put /dev/stdin' works without temp files. It looks that the current version of Samba implements this standard, so this patch hasn't any more use.

scsiinfo-segments

Patch to command line and X11 scsiinfo tool for changing number of segments of a SCSI drive. May be integrated now.

Amanda

Various patches for the Amanda backup software (tape append, verify support, support of autochanger also for restore, work-around for tar sparse bug). This was developped under the Amanda license, with support from Distributed Logic SA.

Some more stuff

I have written other software, but I have not bothered archiving it here for download. I just give the list of the software and a short description. If something interests you, request it from me.

coroutines: A quick and dirty implementation of cooperative multitasking within a UNIX process or an embedded system using standard C.
uumaps: An attempt to redraw a USENET map.
bsmtp_rcv: uncompressing script for BSMTP and sendmail.
choosemenuitem: Something in the line of dialog.
compute_fees: Something to compute e-mail fees.
datenews: Compute article transmission delays.
localmail: Multiple mailbox on one user ID (e.g. BBS) with sendmail.
enforce_limits: Enforce connection time limits.
explain_inpath: Draw indented texts based on inpath.
guest_system: A whole BBS as shell scripts.
hayes_esp: RISCos 5 Hayes ESP multiserial fast ISA board driver.
ioport: Utility to set/read values in ioports.
ip_probe: Firewall tester.
latex_to_nroff: Converter.
life: Game of life, under curses.
mailtonews/newstomail: Gaters.
micro_clock: More precize time.
modem_to_inet: Like virtual_connection, but as a server/client process with bandwidth limitation.
mywho: who command.
news_compute_fees: Charge based on news traffic.
rfcize: Mostly RFC-822 ize an address.
wrapper: Generic security wrapper.
faxserv: A complete mail to fax solution, with accounting, password, text formatting with special keywords, ability to send enclosed images, based on diverse utilities, and sendfax (mgetty) or efax (also Class 1, then), and a LaTeX^? engine.

Autres sources

http://archive.alphanet.ch/local

Liens

Local