Contributed by Alexander Todorov
Splitting big files into pieces is a common task. Another common task is to create a tar archive, and split it into smaller chunks that can be burned onto CD/DVD. The straightforward approach is to create the archive and then use ‘split.’ To do this, you will need more free space on your disk. In fact, you’ll need space twice the size of the created archive. To avoid this limitation, split the archive as it is being created.
To create a tar archive that splits itself on the fly use the following set of commands:
First create the archive:
tar -czf /dev/stdout $(DIRECTORY_OR_FILE_TO_COMPRESS) | split -d -b $(CHUNK_SIZE_IN_BYTES) - $(FILE_NAME_PREFIX)
To extract the contents:
cat $(FILE_NAME_PREFIX)* >> /dev/stdout | tar -xzf /dev/stdin
The above shown set of commands works on the fly. You don’t need additional free space for temporary files.
A few notes about this exercise:
- ‘tar -L’ prompts you on every chunk created. Compression can not be used with -L option. The above command is not interactive and does not prompt for anything. Compression can be used.
- The number of separate files is 100. This is because we use numerical suffixes – ‘split -d.’ If the specified chunk size is small you will get ‘split: Output file suffixes exhausted’ error. Try with bigger chunk size or with alphabetic suffixes.
- ‘cat’ will concatenate the files properly if they are not renamed. This is due to the fact that the sort order is retained by the appended chunk suffixes.
- Replace ‘tar -z’ with ‘tar -j’ for bzip2 compression or try your favourite compression program. Almost all ‘tar’ and ‘split’ options should be possible.
- The resulting chunk files are not valid tar archives. They can not be extracted separately. If you want such functionality use ‘split-tar,’ which also needs more free space.