is a tool to change a file-system from one format to another, for example from jfs, xfs, or reiser to ext2, ext3, or ext4, in-place and without the need for backup.
| README (fstransform-0.9.3-src) | : | README (fstransform-0.9.4) |
|---|
|
| ############################################################################### | | ############################################################### |
| ############################################################################### | | ############################################################### |
| ############################################################################### | | ### |
| ######### ######### | | ### DISCLAIMER |
| ######### THIS DOCUMENT EXPLAINS HOW TO USE ######### | | ### |
| ######### RISKY PROGRAMS AND PROCEDURES ######### | | ### THIS DOCUMENT EXPLAINS HOW TO USE |
| ######### THAT MAY COMPLETELY AND IRREVERSIBLY DELETE ######### | | ### RISKY PROGRAMS AND PROCEDURES |
| ######### ALL THE DATA ON YOUR DISKS ######### | | ### THAT MAY COMPLETELY AND IRREVERSIBLY DELETE |
| ######### ######### | | ### ALL THE DATA ON YOUR DISKS |
| ######### THE AUTHOR DECLINES ALL RESPONSIBILITIES ######### | | ### |
| ######### FOR ANY DAMAGE THAT MAY DERIVE ######### | | ### THE AUTHOR DECLINES ALL RESPONSIBILITIES |
| ######### FROM USING THE PROGRAMS AND PROCEDURES ######### | | ### FOR ANY DAMAGE THAT MAY DERIVE |
| ######### DESCRIBED IN THIS DOCUMENT ######### | | ### FROM USING THE PROGRAMS AND PROCEDURES |
| ######### ######### | | ### DESCRIBED IN THIS DOCUMENT |
| ############################################################################### | | ### |
| ############################################################################### | | ############################################################### |
| ############################################################################### | | ############################################################### |
| | | |
| This document explains how use the programs 'fstransform', 'fsmove' | | This document explains how use the programs 'fstransform', 'fsmove', |
| and 'fsremap' to transform the contents of a Linux device | | 'fsmount_kernel' and 'fsremap' to transform the contents of a Linux |
| - usually a disk partition - from a filesystem type to another | | device - usually a disk partition - from a filesystem type to another |
| while preserving its contents. | | while preserving its contents. |
| | |
| For example, a disk partition can be transformed from 'jfs' to 'ext4', | | For example, a disk partition can be transformed from 'jfs' to 'ext4', |
| or from 'ext2' to 'xfs', or many other combinations. | | or from 'ext2' to 'xfs', or many other combinations. |
| | |
| Currently, the programs mentioned above have been tested on Linux | | Currently, the programs mentioned above have been tested on Linux |
| with the following filesystems, both as source and as target: | | with the following filesystems, both as source and as target: |
|
| ext2, ext3, ext4, reiserfs, jfs, xfs. | | ext2, ext3, ext4, jfs, reiserfs, xfs. |
| | |
| Do NOT use these programs with other filesystems | | Do NOT use these programs with other filesystems |
| unless you are willing to LOSE your data. | | unless you are willing to LOSE your data. |
| | |
|
| In particular, they do NOT (yet) support ntfs, msdos and vfat file systems. | | In particular, they do NOT (yet) support ntfs, msdos, vfat and exfat file system
s. |
| | |
| Common sense and experience tell that you should ALWAYS have a backup | | Common sense and experience tell that you should ALWAYS have a backup |
| of your valuable data: while the programs do NOT need to backup your data | | of your valuable data: while the programs do NOT need to backup your data |
| to operate, YOU need a backup in case something goes wrong. | | to operate, YOU need a backup in case something goes wrong. |
| | |
| The programs have been tested carefully, yet there is ALWAYS a possibility | | The programs have been tested carefully, yet there is ALWAYS a possibility |
| that they will irreversibly delete ALL the data on the device you run them on, | | that they will irreversibly delete ALL the data on the device you run them on, |
| even if you use a tested combination of filesystems. | | even if you use a tested combination of filesystems. |
| | |
| All this foreword means only one thing: | | All this foreword means only one thing: |
| | |
| IF YOU LOSE YOUR DATA, IT IS YOUR PROBLEM. | | IF YOU LOSE YOUR DATA, IT IS YOUR PROBLEM. |
| | |
| The author declines ALL responsibilities for ANY damage that may derive | | The author declines ALL responsibilities for ANY damage that may derive |
| from using the programs and procedures described in this document. | | from using the programs and procedures described in this document. |
| | |
|
| ############################################################################### | | ############################################################### |
| ############################################################################### | | ############################################################### |
| ############################################################################### | | |
| | ### Introduction |
| | |
| Enough legalese... now let's get to the interesting part. | | Enough legalese... now let's get to the interesting part. |
| | |
| The program 'fstransform' does the following: | | The program 'fstransform' does the following: |
| | |
| it takes a device with a filesystem on it (even if almost full) | | it takes a device with a filesystem on it (even if almost full) |
| and transforms the device to a different filesystem type, | | and transforms the device to a different filesystem type, |
| in-place (i.e. without backup) and non-destructively | | in-place (i.e. without backup) and non-destructively |
| (i.e. it preserves all your data). | | (i.e. it preserves all your data). |
| | |
| It works even if the filesystem is almost full | | It works even if the filesystem is almost full |
| - several empyrical tests have succeeded even with a 95% full filesystem - | | - several empyrical tests have succeeded even with a 95% full filesystem - |
| and even if it contains very large files, for example if some files | | and even if it contains very large files, for example if some files |
| are larger than half the device or larger than the available space. | | are larger than half the device or larger than the available space. |
| | |
|
| There are four requirements for fstransform to have a chance to succeed: | | ### REQUIREMENTS |
| | |
| | There are five requirements for fstransform to have a chance to succeed: |
| | |
| | 1. the device must be unmountable, i.e. `umount DEVICE` must work. |
| | In particular, if some running programs are using the device, |
| | you must first close or kill them. |
| | |
| | Transforming the current root directory does not work. For that, you should |
| | boot from a different installation (for example a live CD, DVD or USB). |
| | |
|
| 1) the device must have a little free space, typically at least 5% | | 2. the device must have a little free space, typically at least 5% |
| | |
| WARNING: transforming an almost full device to 'xfs' file-system | | WARNING: transforming an almost full device to 'xfs' file-system |
| can be tricky: | | can be tricky: |
| * you need either slightly more free space, typically at least 10%, | | * you need either slightly more free space, typically at least 10%, |
| * or you must be VERY quick at suspending fstransform | | * or you must be VERY quick at suspending fstransform |
| when either the source or the target (or both) file-system is almost full | | when either the source or the target (or both) file-system is almost full |
| and run 'xfs_fsr' on the source or target device (or both) | | and run 'xfs_fsr' on the source or target device (or both) |
| before resuming fstransform. | | before resuming fstransform. |
| A future fstransform version may automate this operation. | | A future fstransform version may automate this operation. |
| | |
|
| 2) the filesystem on the device must support SPARSE FILES, i.e. files with holes | | 3. the filesystem on the device must support SPARSE FILES, i.e. files with holes |
| (see for example http://en.wikipedia.org/wiki/Sparse_file for an explanation
of what they are) | | (see for example http://en.wikipedia.org/wiki/Sparse_file for an explanation
of what they are) |
| and at least one of the two system calls "ioctl(FS_IOC_FIEMAP)" or "ioctl(FIB
MAP)" | | and at least one of the two system calls "ioctl(FS_IOC_FIEMAP)" or "ioctl(FIB
MAP)" |
| (see the file Documentation/filesystems/fiemap.txt in any recent Linux kernel | | (see the file Documentation/filesystems/fiemap.txt in any recent Linux kernel |
| for an explanation, or search for the same file on Internet) | | for an explanation, or search for the same file on Internet) |
| | |
|
| 3) the initial and final filesystems must be supported by the Linux kernel | | ioctl(FIBMAP) is limited by design to 2G-1 blocks, which typically translates |
| | to 8TB - 4kB. |
| | To transform file systems equal or larger than 8TB, ioctl(FIEMAP) is required |
| | . |
| | |
| | 4. the initial and final filesystems must be supported by the Linux kernel |
| (i.e. it must be able to mount them) | | (i.e. it must be able to mount them) |
| and by the tools 'mkfs' and 'fsck' | | and by the tools 'mkfs' and 'fsck' |
|
| (i.e. it must be possible to create them and check them for errors) | | (i.e. it must be possible to create them and check them for errors). |
| | |
|
| 4) the following programs must be available: | | Support through FUSE (userspace) drivers is acceptable in theory, |
| the two custom-made programs 'fsmove' and 'fsremap' (distributed with the scr | | but there are currently no tested FUSE filesystems that can be converted |
| ipt) | | reliably with fstransform. |
| and several common Linux tools: | | |
| | 5. the following programs must be available: |
| | the three custom-made programs 'fsmove', 'fsmount_kernel' 'fsremap' |
| | (distributed with the script) and several common Linux tools: |
| which, expr, id, blockdev, losetup, mount, umount, | | which, expr, id, blockdev, losetup, mount, umount, |
| mkdir, rmdir, rm, mkfifo, dd, sync, fsck, mkfs | | mkdir, rmdir, rm, mkfifo, dd, sync, fsck, mkfs |
| | |
|
| FOREWORD: | | ### KNOWN LIMITATIONS |
| | |
| | 1) As stated above, at a certain step during the conversion, fstransform needs |
| | to unmount the device being transformed. |
| | For this reason, running fstransform on the device currently mounted as / |
| | (i.e. the root directory) fails. |
| | For the same reason, running fstransform on the device currently mounted |
| | as /usr, /home or /var or similar heavily-used directories is difficult, |
| | because quite often there are programs using those, which prevents |
| | them from being unmounted. |
| | |
| | 2) If the device contains a HUGE number of files with multiple hard links, |
| | fstransform will be very slow and consume a LOT of memory. |
| | Devices with more than one million files with multiple hard links |
| | can cause fstransform to crash with "out of memory" errors. |
| | |
| | 3) JFS file systems equal or larger than 8TB cannot be converted |
| | due to missing support for ioctl(FIEMAP) in the kernel: |
| | the fallback ioctl(FIBMAP) is limited by design to < 8TB (assuming 4k blocks) |
| | |
| | Also, ioctl(FIBMAP) must be called for _each_ block so the conversion |
| | will be a bit slower. |
| | |
| | 4) REISERFS file systems using format "3.5" (the default) and equal or larger th |
| | an 2TB |
| | cannot be converted due to their maximum file size = 2TB - 4k: |
| | fstransform needs to create a sparse file as large as the device itself. |
| | |
| | REISERFS file systems using format "3.6" are immune to this problem. |
| | |
| | 5) for the same reason, a device cannot be converted _to_ REISERFS format "3.5" |
| | if it contains some files larger than 2TB - 4k. |
| | |
| | ### DETAILS TO KNOW |
| | |
| If the original device is almost full, the program 'fsremap' | | If the original device is almost full, the program 'fsremap' |
| will create a relatively small backup file ("secondary storage") | | will create a relatively small backup file ("secondary storage") |
| inside the directory /var/tmp/fstransform. | | inside the directory /var/tmp/fstransform. |
| This secondary storage file will be at most as large as half your free RAM. | | This secondary storage file will be at most as large as half your free RAM. |
| | |
| You can pass the option '-s <size>[k|M|G|T|P|E|Y|Z]' to the tool 'fsremap' | | You can pass the option '-s <size>[k|M|G|T|P|E|Y|Z]' to the tool 'fsremap' |
| in order manually set the secondary storage size, | | in order manually set the secondary storage size, |
| but please understand that using a too small secondary storage | | but please understand that using a too small secondary storage |
| can slow down the procedure. | | can slow down the procedure. |
| | |
| To pass the same option to 'fstransform', you must execute something like | | To pass the same option to 'fstransform', you must execute something like |
| fstransform --opts-fsremap='-s <size>' <other-options-and-arguments> | | fstransform --opts-fsremap='-s <size>' <other-options-and-arguments> |
| | |
|
| PROCEDURE: | | ### PROCEDURE |
| | |
|
| 0) compile fsmove and fsremap. | | 0. compile fsmove, fsmount_kernel and fsremap. |
| Running "./configure" then "make" should suffice on any recent Linux machine, | | Running "./configure" then "make" should suffice on any recent Linux machine, |
| as long as g++ is installed. | | as long as g++ is installed. |
| | |
|
| You will get two executables, fsmove and fsremap. | | You will get three executables, fsmove and fsremap. |
| They will be located at | | They will be located at |
|
| fstransform-{version}-src/fsmove/Release/fsmove | | ./fsmove/build/fsmove |
| and | | ./fsmount_kernel/build/fsmount_kernel |
| fstransform-{version}-src/fsremap/Release/fsmap | | ./fsremap/build/fsremap |
| | |
| You are suggested to either run "make install" or to copy them to a simpler p
ath. | | You are suggested to either run "make install" or to copy them to a simpler p
ath. |
|
| Below, they will be referred as {fsmove} and {fsremap} | | Below, they will be referred as {fsmove}, {fsmount_kernel} and {fsremap} |
| | |
|
| 1) OPTIONAL - CAN BE SKIPPED | | 1. OPTIONAL - CAN BE SKIPPED |
| mount read-write the device you want to remap to a new file-system type | | mount read-write the device you want to remap to a new file-system type |
| | |
| mount {device} {device-mount-point} [your-options] | | mount {device} {device-mount-point} [your-options] |
| | |
| if the device is already mounted, check that it is mounted read-write | | if the device is already mounted, check that it is mounted read-write |
| and that no process is using it. | | and that no process is using it. |
| | |
|
| 2) decide the target file-system type. | | 2. decide the target file-system type. |
| | |
| For some combinations of the initial and final filesystems | | For some combinations of the initial and final filesystems |
| it is not necessary to use 'fstransform', | | it is not necessary to use 'fstransform', |
| as the same result can be obtained with much simpler - and SAFER - tools. | | as the same result can be obtained with much simpler - and SAFER - tools. |
| | |
| For example, an 'ext2' or 'ext3' filesystem can be transformed into 'ext3' | | For example, an 'ext2' or 'ext3' filesystem can be transformed into 'ext3' |
| or 'ext4' using the program 'tune2fs'. | | or 'ext4' using the program 'tune2fs'. |
| | |
| Explaining how to use 'tune2fs' is beyond the scope of this document, | | Explaining how to use 'tune2fs' is beyond the scope of this document, |
| just read its man-page or search on the Internet for one of | | just read its man-page or search on the Internet for one of |
| "convert Linux File System ext2 to ext3" | | "convert Linux File System ext2 to ext3" |
| "convert Linux File System ext2 to ext3" | | "convert Linux File System ext2 to ext3" |
| "convert Linux File System ext3 to ext4" | | "convert Linux File System ext3 to ext4" |
| | |
|
| But for most combinations, the only way is either to do a full backup + forma | | But for most combinations, the only way is either to do a full backup + |
| t + | | format + restore the data, or use 'fstransform' |
| restore the data, or use 'fstransform' | | |
| | |
|
| 3) execute the program | | 3. execute the program |
| | |
|
| fstransform {device} {target-file-system-type} | | fstransform {device} {target-file-system-type} |
| | |
|
| 4) follow the instructions - the program will tell you what it is doing, | | 4. follow the instructions - the program will tell you what it is doing, |
| and will also call 'fsmove' and 'fsremap' which show progress percentage | | and will also call 'fsmove' and 'fsremap' which show progress percentage |
| and estimated time left. | | and estimated time left. |
| | |
| Note that 'fsmove' and 'fsremap' need approximately the same time to run, | | Note that 'fsmove' and 'fsremap' need approximately the same time to run, |
| so if 'fsmove' tells you that it will need 2 hours, 'fsmove' will | | so if 'fsmove' tells you that it will need 2 hours, 'fsmove' will |
| likely need a similar amount of time, for a total of 4 hours. | | likely need a similar amount of time, for a total of 4 hours. |
| | |
| In case there are errors, you can even try to fix them instead of | | In case there are errors, you can even try to fix them instead of |
| aborting the execution (if you know what you are doing). | | aborting the execution (if you know what you are doing). |
| | |
|
| 5) be PATIENT. Transforming a large device takes a LONG time... | | 5. be PATIENT. Transforming a large device takes a LONG time... |
| On a fairly fast disk, it takes about 1 minute per 1 gigabyte. | | On a fairly fast disk, it takes about one minute per gigabyte. |
| It means transforming 250GB of takes about 4 hours. | | It means transforming 1000GB takes about 16 hours. |
| Solid state disks (SSD) can be much faster. | | Raid disks can be somewhat faster, and solid state disks (SSD) |
| | can be _much_ faster. |
| | |
| 6) if something goes really wrong, check in /var/tmp/fstransform | | 6) if something goes really wrong, check in /var/tmp/fstransform |
| for the log files | | for the log files |
| fstransform.log.<NNN> and fsremap.job.<MMM>/fsremap.log | | fstransform.log.<NNN> and fsremap.job.<MMM>/fsremap.log |
| they are ABSOLUTELY necessary if you want someone to analyze the problem | | they are ABSOLUTELY necessary if you want someone to analyze the problem |
| - but unless you are very lucky you can forget about recovering your data... | | - but unless you are very lucky you can forget about recovering your data... |
| | |
| 7) if for some reason the execution is interrupted while 'fsremap' is running, | | 7) if for some reason the execution is interrupted while 'fsremap' is running, |
| for example due to a power failure, it is possible to resume it | | for example due to a power failure, it is possible to resume it |
| by running 'fsremap --resume-job=<MMM> {device}'. | | by running 'fsremap --resume-job=<MMM> {device}'. |
| Also, 'fsremap' will show at its startup the exact command line | | Also, 'fsremap' will show at its startup the exact command line |
| needed to resume its execution. | | needed to resume its execution. |
| | |
|
| | The loop file created by fstransform must NEVER be as argument to |
| | 'fsremap --resume-job=<MMM> {...}'. You would IRREVERSIBLY LOSE YOUR DATA! |
| | |
| | SOME REAL-WORLD TESTS |
| | |
| | 1) 1000GB encrypted disk, 52% full, ext2->ext4: SUCCESS, took 12 hours |
| | despite one system crash and two manual interruptions (CTRL+C) |
| | (Yes, I know ext2->ext4 can be done with tune2s, but I wanted to |
| | test fstransform) |
| | |
| | 2) 1540GB encrypted raid0 (3 disks), 56% full, ext2->ext4: SUCCESS, took 8 hours |
| | |
| Good luck! | | Good luck! |
| | |
| Massimiliano Ghilardi | | Massimiliano Ghilardi |
|
| <paperinik@users.sf.net> | | |
| | | |
| End of changes. 25 change blocks. |
|---|
| 55 lines changed or deleted | | 118 lines changed or added |
|---|
"Fossies" - the Fresh Open Source Software Archive 