Creating archive > 4GiB from script

[Bogdan]

This may be a bit off-topic in this thread, but why use compression tools for backup purposes?

There seems to be a bit of confusion between archiving and compressing, Neither the original PKZip nor Info-ZIP's Zip are compression tools but archivers. Tar is a tape archiver with block operation. The standard compression tools under OS/2 are PACK and PACK2, or under UNIX you have compress and pack and much more incompatible, mostly non-portable tools.

Disk space is fairly cheap nowadays. Rsync and Keith's Easy Synch are imho far better tools for that, not in the least because they can do differential/incremental backups, it's easily possible to restore individual files etc. Are network copying times an issue?

That's only data duplication or synchronisation. And how fast you can restore from those "backups"?

[shmuel]

The screen update time is negligible.

No it's not.

Why do you believe that it takes hours to change a few dozen characters on the screen?

I also use ANPM to manage that stuff, and the version installed by ANPM is broken.

What is the problem? They work well, for me.

The problem is that I need to backup my data, not yours, and the zip provided with ArcaOS truncates the archive at 4 GiB - 1.

Have you made sure, that no other versions are sitting around in your system, just waiting to cause problems?

I don't understand how any of the other versions could have generated the date in "Compiled with emx+gcc 4.9.2 for OS/2 2.x/3.x (32-bit) on Sep 28 2018."

Have you reported your problem (if you really do have one)?

Who are you and what have you done with the real Doug Bissett? What is "See also http://trac.netlabs.org/ports/ticket/194", chopped liver? Please read what I've already posted here before accusing me of making up the problem.

[shmuel]

This may be a bit off-topic in this thread, but why use compression tools for backup purposes? Disk space is fairly cheap nowadays. Rsync and Keith's Easy Synch are imho far better tools for that, not in the least because they can do differential/incremental backups, it's easily possible to restore individual files etc. Are network copying times an issue?

I want off-site backups, and the only devices that I have for removable media are DVD drives and USB ports, both slow. The capacity of a DVD is too small, so that leaves thumb drives or USB hard drives.

With zip or other compressed archive there are fewer data to write, which reduces the time required. I've rarely needed to restore an entire volume; usually I just need to restore one file to get the version prior to a finger check.

Of course, zip is also nice for porting my environment to another machine when my house loses power for weeks 

[Doug Bissett]

Why do you believe that it takes hours to change a few dozen characters on the screen?

I know that PMMail has a serious problem populating the display container, when there are a few thousand entries to display. I see no reason why the Archive tool doesn't have the same problem. It is part of the way that OS/2 works, unless a programmer takes steps to work around it.

The problem is that I need to backup my data, not yours, and the zip provided with ArcaOS truncates the archive at 4 GiB - 1.

It occurs to me, that you may have a media problem (I know that ZIP/UNZIP will do it). What, exactly, are you backing up to? I know that FAT32 has some sort of maximum file size (probably 4 GB, and that is possibly fixed in the latest FAT32 versions). Try using JFS on the backup media.

[Alex Taylor]

FAT32 does not support files larger than 4GB, period. This is a limitation of the filesystem, not the driver.

If you have to zip large archives onto a FAT32 drive, create a multivolume archive split at 4GB.

For example,

Code: [Select]

zip -r -S -AC -s 4g t:\mybackup_c.zip c:\ -x SWAPPER.DAT

[Andi B.]

FAT32 does not support files larger than 4GB, period.

And older versions of our driver only supports up to 2GB. IIRC you need a pretty new version of fa32.ifs if you need >= 2GB.

[shmuel]

Why do you believe that it takes hours to change a few dozen characters on the screen?

I know that PMMail has a serious problem populating the display container, when there are a few thousand entries to display. I see no reason why the Archive tool doesn't have the same problem. It is part of the way that OS/2 works, unless a programmer takes steps to work around it.

The only significant display updating is in the progress box; the update for each file is subsecond. There are only a bit over 4K files.

The problem is that I need to backup my data, not yours, and the zip provided with ArcaOS truncates the archive at 4 GiB - 1.

It occurs to me, that you may have a media problem (I know that ZIP/UNZIP will do it). What, exactly, are you backing up to?

M: is formatted JFS.

I know that FAT32 has some sort of maximum file size (probably 4 GB, and that is possibly fixed in the latest FAT32 versions). Try using JFS on the backup media.

The backup medium is formatted JFS.

[shmuel]

http://hobbes.nmsu.edu/download/pub/os2/util/archiver/p7zip-9.20.1-os2.zip seems to be a viable workaround; the newer http://hobbes.nmsu.edu/download/pub/os2/util/archiver/p7zip-16.02-os2-1.zip seems to be broken.

[Alex Taylor]

Last I checked, the 7zip archive format did not support EAs.  That renders it unsuitable for backing up certain things, like OS/2 system volumes.

BTW, I use info-zip (v3) to back up my volumes all the time.  I've never had any problem creating zips much larger than 4 GB, as long as the target volume supports it. 

[shmuel]

Last I checked, the 7zip archive format did not support EAs.  That renders it unsuitable for backing up certain things, like OS/2 system volumes.

I wound up using zip with exclude and include; that leaves me with more archive files for the volume, but the elapsed time is under an hour, far faster than with 7za. I'd still like a URL for a working version of zip.

BTW, I use info-zip (v3) to back up my volumes all the time.  I've never had any problem creating zips much larger than 4 GB, as long as the target volume supports it.

With what provenance? The one I'm using is from the ArcaOS and Netlabs repositories:

Code: [Select]

Installed Packages
Name        : unzip
Arch        : i686
Version     : 6.0
Release     : 7.oc00
Size        : 202 k
Repo        : installed
Summary     : A utility for unpacking zip files
URL         : http://www.info-zip.org/UnZip.html
License     : BSD
Description : The unzip utility is used to list, test, or extract files from a
            : zip archive.  Zip archives are commonly found on MS-DOS systems.
            : The zip utility, included in the zip package, creates zip
            : archives.  Zip and unzip are both compatible with archives created
            : by PKWARE(R)'s PKZIP for MS-DOS, but the programs' options and
            : default behaviors do differ in some respects.
            :
            : Install the unzip package if you need to list, test or extract
            : files from a zip archive.

Name        : zip
Arch        : i686
Version     : 3.0
Release     : 8.oc00
Size        : 524 k
Repo        : installed
Summary     : A file compression and packaging utility compatible with PKZIP
URL         : http://www.info-zip.org/Zip.html
License     : BSD
Description : The zip program is a compression and file packaging utility.  Zip
            : is analogous to a combination of the UNIX tar and compress
            : commands and is compatible with PKZIP (a compression and file
            : packaging utility for MS-DOS systems).
            :
            : Install the zip package if you need to compress files using the
            : zip program.


UnZip 6.00 of 20 April 2009, by Info-ZIP.  Maintained by C. Spieler.  Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with emx+gcc 4.9.2 for OS/2 (32-bit) on Jun 12 2018.

UnZip special compilation options:
        ASM_CRC
        COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
        OS2_EAS
        SET_DIR_ATTRIB
        SYMLINKS (symbolic links supported, if RTL and file system permit)
        TIMESTAMP
        UNIXBACKUP
        USE_EF_UT_TIME
        USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
        USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
        LARGE_FILE_SUPPORT (large files over 2 GiB supported)
        ZIP64_SUPPORT (archives using Zip64 for large files supported)
        USE_BZIP2 (PKZIP 4.6+, using bzip2 lib version 1.0.6, 6-Sept-2010)
        VMS_TEXT_CONV
        [decryption, version 2.11 of 05 Jan 2007]
A zip of the entire volume creates

Code: [Select]

12-23-18  10:37   4,294,967,295    124  M.zip
Attempting to unzip it gives

Code: [Select]

Archive:  K:/M.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
Doing the zip in pieces gives

Code: [Select]

12-23-18   3:13   2,252,560,732    124  M.pdf.Galaxy.zip
12-23-18   3:41   3,468,746,085    124  M.pdf.IBM.zip
12-23-18   3:59          45,183      0  M.pdf.IBM370.log
12-23-18   4:22          28,300      0  M.pdf.IBM370.tlog
12-23-18   4:11   3,199,967,659    124  M.pdf.IBM370.zip
12-23-18   4:24     269,681,811    124  M.pdf.IBMz.zip
12-23-18   4:26     198,084,369    124  M.pdf.Math.zip
12-23-18   2:52   2,642,968,249    124  M.pdf.zip
12-23-18   2:33     676,719,403    124  M.zip

[Alex Taylor]

The version on my backup partition reports itself ("zip -v") as:

Code: [Select]

Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon.  Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip,
as of above date; see http://www.info-zip.org/ for other sites.

Compiled with emx+gcc 4.4.6 for OS/2 2.x/3.x (32-bit) on Jul 17 2013.

This is a slightly older-than-the-latest build of the bww code.  IIRC, I built it myself from the bww sources, after adding a patch to implement OS/2 support for the archive attribute switches (-AS/-AC).  I believe bww incorporated my patch in their subsequent releases. As such, there should be no real difference between my build and theirs, beyond theirs being based on slightly newer upstream code; still, if you want to try it, it's http://altsan.org/os2/utils/zip_300_os2_archive.zip.

My usual production volume uses the currently-latest version from the Netlabs yum repo, although I don't think I've tried zipping any massive archives with that build.

[shmuel]

The version on my backup partition reports itself ("zip -v") as:

Code: [Select]

Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon.  Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip,
as of above date; see http://www.info-zip.org/ for other sites.

Compiled with emx+gcc 4.4.6 for OS/2 2.x/3.x (32-bit) on Jul 17 2013.

This is a slightly older-than-the-latest build of the bww code.  IIRC, I built it myself from the bww sources, after adding a patch to implement OS/2 support for the archive attribute switches (-AS/-AC).  I believe bww incorporated my patch in their subsequent releases. As such, there should be no real difference between my build and theirs, beyond theirs being based on slightly newer upstream code; still, if you want to try it, it's http://altsan.org/os2/utils/zip_300_os2_archive.zip.

My usual production volume uses the currently-latest version from the Netlabs yum repo, although I don't think I've tried zipping any massive archives with that build.

The bad news is that the same error exists in that build; the good news is that if I remove the working directory (-b) option, the archive is no longer truncated, although zip does take more than twice as long.

[Andreas Schnellbacher]

Last I checked, the 7zip archive format did not support EAs.

The version used by Zippy or ArcaOS Archive Tool by "GlassMan" seems to support EAs. See the table "Archive Management Software Comparison", "Supports OS/2 EAs".