Author Topic: Creating archive > 4GiB from script  (Read 1099 times)

Shmuel (Seymour J.) Metz

  • Newbie
  • *
  • Thank You
  • -Given: 1
  • -Receive: 0
  • Posts: 8
    • View Profile
Re: Creating archive > 4GiB from script
« Reply #15 on: December 03, 2018, 10:29:21 pm »
Quote
What happens when using a different unarchiver?

At least python -m zipfile -e should be available for this purpose.

Where do I download it and the documentation? Thanks.

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 9
  • -Receive: 112
  • Posts: 1644
    • View Profile
Re: Creating archive > 4GiB from script
« Reply #16 on: December 04, 2018, 12:42:41 am »
Python will be installed in ArcaOS or any system using YUM. Lots of documentation out there, perhaps start with https://www.python.org/. Just be aware that we're still on Python 2 and a lot of documentation is for Python 3. They're close but different in some ways.

Bogdan Szmalcownik

  • Jr. Member
  • **
  • Thank You
  • -Given: 0
  • -Receive: 2
  • Posts: 65
    • View Profile
Re: Creating archive > 4GiB from script
« Reply #17 on: December 04, 2018, 02:38:16 am »
Quote
What happens when using a different unarchiver?

At least python -m zipfile -e should be available for this purpose.

Where do I download it and the documentation? Thanks.
It should be already installed as a requirement for this yum/rpm variation. Please don't ask for documentation. Non-OS specific things should work as long the runtime permits.

But under OS/2 zip archives have only a defined behaviour up to 2GB-1B. The compatibility problems start with files sizes between 2GB and 4GB-1B. And starting with 4GB the zip format cannot longer be used. That's the reason I was asking for the used storage access method. It seems you try to do some kind of backup & restore with two different tools which don't offer the same features. Please use a real backup tool to create a backup archive and do the compression later with the commands you're using.

Doug Bissett

  • Hero Member
  • *****
  • Thank You
  • -Given: 1
  • -Receive: 39
  • Posts: 1082
    • View Profile
Re: Creating archive > 4GiB from script
« Reply #18 on: December 04, 2018, 06:18:20 am »
Quote
And starting with 4GB the zip format cannot longer be used.

Please don't tell my computer that. 12 GB (and larger) ZIP files work, with no problems, when you use the proper version of ZIP/UNZIP.

Please, update your software, properly, and stop posting incorrect information. Thanks...

Bogdan Szmalcownik

  • Jr. Member
  • **
  • Thank You
  • -Given: 0
  • -Receive: 2
  • Posts: 65
    • View Profile
Re: Creating archive > 4GiB from script
« Reply #19 on: December 04, 2018, 01:52:43 pm »
Quote
And starting with 4GB the zip format cannot longer be used.

Please don't tell my computer that. 12 GB (and larger) ZIP files work, with no problems, when you use the proper version of ZIP/UNZIP.
Please read the appropriate documentation. I wrote about the zip format not about the special case of files. Of course you can rename large files to ZIP but that's just ridiculous.

Quote
Please, update your software, properly, and stop posting incorrect information. Thanks...
To 9 years old code with undocumented patches? ...and you will have the problems the original poster is talking about.

Doug Bissett

  • Hero Member
  • *****
  • Thank You
  • -Given: 1
  • -Receive: 39
  • Posts: 1082
    • View Profile
Re: Creating archive > 4GiB from script
« Reply #20 on: December 04, 2018, 06:40:57 pm »
Please, update your software, properly, and stop posting incorrect information. Thanks...

Shmuel (Seymour J.) Metz

  • Newbie
  • *
  • Thank You
  • -Given: 1
  • -Receive: 0
  • Posts: 8
    • View Profile
Re: Creating archive > 4GiB from script
« Reply #21 on: December 07, 2018, 08:38:09 pm »
 1. Arctool can create and read a 12 GiB zip file, although it takes more than 4 times as long

 2. what are the URLs for the zip and unzip builds that support > 4GiB in OS/2?

 3. Yes, of course ArcaOS includes Python, but what about the zipfile module and its documentation?

 4. Several of the other compression tools, e.g., gzip, xz, are intended to compress in place rather
     than to build a compressed archive.

Doug Bissett

  • Hero Member
  • *****
  • Thank You
  • -Given: 1
  • -Receive: 39
  • Posts: 1082
    • View Profile
Re: Creating archive > 4GiB from script
« Reply #22 on: December 07, 2018, 10:18:31 pm »
Quote
1. Arctool can create and read a 12 GiB zip file, although it takes more than 4 times as long

It reads the ZIP file (I have done it with 60 GB files), just as fast. What takes the time, is formatting the screen output, so the user can see, and select, the files that are inside the zip file. Overall, much easier to use, but slower.

Quote
2. what are the URLs for the zip and unzip builds that support > 4GiB in OS/2?

I don't know, offhand. I use Arca Noae Package Manager (ANPM) to manage that stuff, so I don't need to do it manually. I expect that they are wherever they post the RPM packages. Whether they will work properly, without all of the other stuff that ANPM (RPM/YUM) installs, I don't care to know.

Quote
4. Several of the other compression tools, e.g., gzip, xz, are intended to compress in place rather than to build a compressed archive.

That doesn't make sense. How can you compress a folder (for example) "in place"? And what good are they, if they don't produce an archive file? Gzip and xz do exactly what ZIP does, they create an archive file out of (a) source file(s).

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 9
  • -Receive: 112
  • Posts: 1644
    • View Profile
Re: Creating archive > 4GiB from script
« Reply #23 on: December 07, 2018, 10:33:10 pm »
The Python zip documentation and link to module, https://docs.python.org/2/library/zipfile.html

With gzip, bzip2, xv, the way to do it is to create a tar archive and then gzip, bzip2 or xz the tar archive. Tar usually supports both steps, read the documentation. Problem with these is no EA support unless you use really old versions, which don't support large files.
There is star, http://hobbes.nmsu.edu/download/pub/os2/util/archiver/star-1.5.73-os2.zip, an alternative tar program that does support EAs, large files and gzip and IIRC, bzip2. Worth looking at as Yuri did a good job of porting, not just rebuilding.
In order of compression, from worst to best, gzip, bzip2 and xz. Note that xz is not considered as stable though I've never had a problem with xz compressed tarballs I've downloaded.

Shmuel (Seymour J.) Metz

  • Newbie
  • *
  • Thank You
  • -Given: 1
  • -Receive: 0
  • Posts: 8
    • View Profile
Re: Creating archive > 4GiB from script
« Reply #24 on: December 14, 2018, 08:00:34 pm »
Quote
1. Arctool can create and read a 12 GiB zip file, although it takes more than 4 times as long

It reads the ZIP file (I have done it with 60 GB files), just as fast. What takes the time, is formatting the screen output, so the user can see, and select, the files that are inside the zip file. Overall, much easier to use, but slower.

It's the zip speed I'm concerned with, and that's 4 times as slow. The screen update time is negligible.
Quote

Quote
2. what are the URLs for the zip and unzip builds that support > 4GiB in OS/2?

I don't know, offhand. I use Arca Noae Package Manager (ANPM) to manage that stuff, so I don't need to do it manually. I expect that they are wherever they post the RPM packages. Whether they will work properly, without all of the other stuff that ANPM (RPM/YUM) installs, I don't care to know.

I also use ANPM to manage that stuff, and the version installed by ANPM is broken. That's why I asked for the  URL to a working zip version. I don't care whether it is an rpm, a wpi or a zip, as long as correctly builds a large zip.

Quote
Quote
4. Several of the other compression tools, e.g., gzip, xz, are intended to compress in place rather than to build a compressed archive.

That doesn't make sense. How can you compress a folder (for example) "in place"? And what good are they, if they don't produce an archive file?

You can't. The default behavior of those utilities is to create a compressed file and delete the original.

Quote
Gzip and xz do exactly what ZIP does, they create an archive file out of (a) source file(s).

Not even close. See the Dec 7 reply by David Yeo

Shmuel (Seymour J.) Metz

  • Newbie
  • *
  • Thank You
  • -Given: 1
  • -Receive: 0
  • Posts: 8
    • View Profile
Re: Creating archive > 4GiB from script
« Reply #25 on: December 14, 2018, 08:15:18 pm »
The Python zip documentation and link to module, https://docs.python.org/2/library/zipfile.html

Thanks.

Quote
With gzip, bzip2, xv, the way to do it is to create a tar archive and then gzip, bzip2 or xz the tar archive. Tar usually supports both steps, read the documentation. Problem with these is no EA support unless you use really old versions, which don't support large files.
There is star, http://hobbes.nmsu.edu/download/pub/os2/util/archiver/star-1.5.73-os2.zip, an alternative tar program that does support EAs, large files and gzip and IIRC, bzip2. Worth looking at as Yuri did a good job of porting, not just rebuilding.
In order of compression, from worst to best, gzip, bzip2 and xz. Note that xz is not considered as stable though I've never had a problem with xz compressed tarballs I've downloaded.

I definitely need both EA and large archive support. I'll take a look at star, but in the meantime if you can give me a URL for a zip build that can creat a large archive under ArcaOS, I'd appreciate it.

Doug Bissett

  • Hero Member
  • *****
  • Thank You
  • -Given: 1
  • -Receive: 39
  • Posts: 1082
    • View Profile
Re: Creating archive > 4GiB from script
« Reply #26 on: December 14, 2018, 08:20:57 pm »
Quote
The screen update time is negligible.

No it's not.

Quote
I also use ANPM to manage that stuff, and the version installed by ANPM is broken.

What is the problem? They work well, for me.

Have you made sure, that no other versions are sitting around in your system, just waiting to cause problems?

Have you reported your problem (if you really do have one)? It won't even be thought about, until you do. Posting here is NOT reporting a problem to anyone.

Quote
You can't. The default behavior of those utilities is to create a compressed file and delete the original.

Well, that isn't going to do you much good, as a backup method.  It does have limited uses, but not for backups.

Dave Yeo

  • Hero Member
  • *****
  • Thank You
  • -Given: 9
  • -Receive: 112
  • Posts: 1644
    • View Profile
Re: Creating archive > 4GiB from script
« Reply #27 on: December 14, 2018, 08:53:24 pm »


Quote
You can't. The default behavior of those utilities is to create a compressed file and delete the original.

Well, that isn't going to do you much good, as a backup method.  It does have limited uses, but not for backups.

As I said, it's a two step process, tar which basically concats the files together with a table of contents, then compress the tarball. The old unix way is to pipe the commands, the new unix way is to use tar with a filtering option. tar --help | less for details.
Gzip, Bzip2 and xv are great for compressing individual files.
Compressing a 175MB log file, build log from TB, highly compressible.
gzip,  21.14 sec 5.67MB
bzip2 66.92 sec 4.42 MB
xz      26.79 sec 3.58 MB (Using 1 core)
xz      9.11 sec   3.61MB (using 4 cores) 
zip     5 sec        5.67MB (same compression method as gzip)


Bogdan Szmalcownik

  • Jr. Member
  • **
  • Thank You
  • -Given: 0
  • -Receive: 2
  • Posts: 65
    • View Profile
Re: Creating archive > 4GiB from script
« Reply #28 on: Today at 04:42:12 am »
As I said, it's a two step process, tar which basically concats the files together with a table of contents,
I don't know what's the reason of this misconception. No known tar implementation works in this way.

Quote
then compress the tarball. The old unix way is to pipe the commands, the new unix way is to use tar with a filtering option. tar --help | less for details.
Of course it's possible to use tar under UNIX. But it's not a tool for backup or restore. To label pipes old-stylish sounds strange. Btw. the given command will not work with any (standard) UNIX implementation. A simple "man tar" should be done, to read at least the section "Limitations" and to check its convenience for the desired process.

For backups usable in recovery situations there should be dump, cpio and pax. Unfortunately OS/2 is not UNIX and additional restrictions can occur.