Software that should exist #7: File-Allocation without Nullifying

Sun, 16 May 2010 03:26:26 +0000

I dont know about you, but I had this problem more than once: I just need a large file on my disk with no specific content to create some other fs on it. For example when creating Live-CDs, additional Swapfiles or just Test-Volumes for exotic filesystems or virtual machines, I need a big file on which I can then perform filesystem-creation and stuff. The default way to do this is to run dd with appropriate blocksize and blockcount options, let it read from /dev/zero, and write into the file. The problem here is that I then not just allocate the file but also overwrite it with zeroes. In many cases, this would not be necessary. The main reason for using /dev/zero is that /dev/zero is the fastest device one can get to get some data – but actually, I mostly dont care and the only reason for not using /dev/urandom is that /dev/urandom is a lot slower.

So, it would be nice to be able to say „give me a file with size … and random content“, such that the kernel does this by just allocating free blocks into a file, but not overwriting them, thus, the only write-accesses on the disk will be the ones for filesystem-headings like inode-tables, etc.

Problematic with this approach – and therefore probably the reason why this is no default mechanism – is that if every user could do this, the user would possibly be able to access blocks of files that shouldnt be seen by this user, i.e. blocks of files which have already been deleted but needed higher read-permissions. On the other hand, as root, there should be no such problem at all.

One possible solution which sometimes suffices is the creation of sparse-files, but only if the underlying filesystem supports sparse files, and even then, for most of the problems mentioned above the access becomes painfully slow, since the blocks have to be allocated ondemand, while the programs assume to get a blockdevice. Most mkfs-instructions will at least require some kind of „force“-option to create a filesystem on a sparse-file anyway. Loopmounting will most probably fail. Using a sparse file as swap-file isnt allowed at all (at least without strange kernel-patches).

Another solution comes – as far as I read – with ext4, which allows creating files which are nulled „virtually“ in the beginning, without having to be overwritten first. Except that I dont really like or trust ext4, since it doesnt bring the features btrfs would bring, but also doesnt seem to be a lot more stable, this is a solution on the filesystem-level. Unfortunately, such problems mostly arise in situations when you didnt choose the filesystem with respect to this or cant choose your filesystem anyway. There has got to be some more general solution.