Using zorder to pack files in repeatable order

Scenario

A customer has a large number of small files that do not change often. The nature of the application causes the files to be regularly rewritten, without the content being changed. This causes the file modification dates to change.

Issues posed by scenario

AhsayOBS may use the modification date to determine whether a file has changed. In this scenario, a change will be detected in every file; but because there is a minimum delta size cutoff, every small unchanged file will be reuploaded, incurring an unexpectedly large upload.

You may attempt to work around this by packing the source folder into a zip file or similar as part of a pre-backup command. However, most zip tools (tar, 7z etc) do not make any attempt to consistently order the input files. If the source files are regenerated periodically without any actual data change, the file inodes may differ, the operating system will provide the files in a different order, and the generated zip file may be ordered entirely differently, once again incurring an unexpectedly large upload.

Introducing zorder

zorder is a tool that zips a folder using a consistent file order, to alleviate the above issues. MyClient makes the zorder tool freely available for Windows; versions for OS X and Linux are available upon request.

To maximise the efficiency of AhsayOBS's own delta and compression systems, zorder does not compress the generated zip file.

Download

zorder-1.1.zip (2015-12-14)

• Feature: Support zip files larger than 2GB
• Enhancement: More user-friendly help message

zorder-1.0.zip (2014-04-30)

• Initial public release

Command-line Usage

ABOUT
 zorder 1.1, Copyright (2014-2015) MyClientGlobal.com Ltd
 Pack a directory in deterministic order, to minimise incremental deltas.

USAGE
 zorder.exe source-directory output-file.zip