PMMail: 2.00.xx to 3.22 migration...what to do?

[Dariusz Piatkowski]

Alright, as the subject says, are there any known "problem spots" and/or preferred steps in attempting this?

My current 2.00.1500 install is ages old...about 1.8 Gig in total disk space, 3 accounts, the big one (mine) about 10k msgs in the SENT folder alone). Lots of different built-in filters (SPAM and various folder sorting), some stuff in the address books, etc, etc.

Sooo...I do not have 3.22 installed yet, but it will be done through the WarpIn package. The current version was a ZIP based install, so if I understood what I had read about this so far I should not try installing OVER the current install. Not sure yet what the migration process is like, is it done during the install, or is this a separate process altogether?

I would appreciate any & all hints and suggestions you can make. My intent is to migrate as much of my current emailstore as possible.

[Doug Bissett]

First BACK UP what you have, just in case something goes wrong (and it could, simply because the version you are using is very old).

You may want to join the mail list. See http://www.os2voice.org/mailinglists.html. That is the best place to ask questions, and find answers to problems, but you need a working mail program to use it.

Next, be sure to READ the information that is presented by the installer, and it would probably be a good idea to look through the FAQ http://pmmail.os2voice.org/index.php?title=PMMail_FAQ. Setting up secure server support is explained in the FAQ, in a different way than it is in the Help. Some people find one, or the other, to be confusing, but they usually figure it out by using both.

I would suggest that you think about reducing the number of messages in the primary folders (Inbox, Sent, and any that you filter messages to). I usually put old messages into sub folders, by year, or month, depending on the amount of activity. Sub folders of the Sent folder take on the characteristics of the Sent folder. You will get much better performance by doing that, and migration will probably be better too.

Then, Re-Index the important folders, to be sure that they don't have surprises in them. The new PMMail can Re-Index whole accounts, but the old one can't (I assume that your version can do folders, but it may not).

Okay, now that you have done some preparation, try it. It will ask for a new location, and basically copy all of your mail etc. to the new location (it is a new directory structure, and some files are modified to match), so be sure that you have enough space. The old PMMail will remain operational, until you decide that the new one is working. Again, READ the instructions, and remember that any mail that you SEND with one, will not be in the other. The new accounts are set to NOT delete mail from the POP server, until you set it to do so, so Incoming mail can end up in both places, as long as you process it with the new program first. If something goes wrong, you can ask what to do about it, and, worst case, erase what was done, and start over.

Another thing, is that the "Migration Tool" is badly named. It does the migration, but it also does a couple of other things, like make PMMail the default mail program, and remove the old PMMail (it actually ZIPs it up), when you select those options. It should never try to do a new migration, when the source is the new PMMail, However, if you don't have a standard setup, odd things can happen. Please ASK what to do if something goes wrong.

Oh yeah, the new spell checker needs to have the appropriate files beside the main installer, when you do the install. Page down on the PMMail home page to get the ASpell packages that you need. They are used by the main installer.

Be especially careful with existing filters. Some fairly serious problems were fixed, and that means that some filters now actually do what they are supposed to do. A few people got caught using incorrect filter operation, which now doesn't do what it did before.

You mention SPAM. PMMail now uses the BogoFilter program, so you will probably want to use that, and get rid of your own SPAM filters. I guarantee that BogoFilter will do a better job, after you properly train it (usually takes a few days, but it is not difficult to do).

That should cover most of it, but I may have forgotten something. Try it, and see what happens.

[Dariusz Piatkowski]

Hi Doug!

Wow man, Thank You, that is a massive write-up and I appreciate ever single part of it.

Alright, let me focus on some key points as I need a little clarification.

...I would suggest that you think about reducing the number of messages in the primary folders (Inbox, Sent, and any that you filter messages to). I usually put old messages into sub folders, by year, or month, depending on the amount of activity. Sub folders of the Sent folder take on the characteristics of the Sent folder. You will get much better performance by doing that, and migration will probably be better too...

Are you suggesting that I put NEW child folders into the account specific default SENT parent folder? I thought of doing this before but I always assumed that it was a bad idea to mess with this folder. It certainly makes a lot of sense, and for starters I would be able to do a search in the SENT folders, which today causes my PMMail to simply crash.

...Okay, now that you have done some preparation, try it. It will ask for a new location, and basically copy all of your mail etc. to the new location (it is a new directory structure, and some files are modified to match), so be sure that you have enough space....

OK, so a specific issue I have is that I only have about 1.5Gig left on my application drive (also happens to be my boot drive, but that's the legacy of the original install I did). So the option I have is to move my current pmmail directory structure to a different drive, where I have a bunch of space, wipe it off of my boot drive and then do the WarpIn based install. Will the installer itself attempt to migrate the old PMMail stuff during the install, or is this a separate process one does later once the NEW PMMail install is completed?

The 2nd question is: does the migration tool/process simply need to find a disk directory structure of the old PMMail, or is it looking for some application specific install/config data?

Basically I am wondering if my plan to move the current install to the spare drive has a chance of working?

...Be especially careful with existing filters. Some fairly serious problems were fixed, and that means that some filters now actually do what they are supposed to do. A few people got caught using incorrect filter operation, which now doesn't do what it did before....

Alright, good point. The filters issue is that they are not complex, but rather I have a whole pile of them, so just a sheer volume. But OK, good to know, I had intended to review all of them one-by-one and possibly get rid of some of them to clean things up a tad.

Thanks again!

[Dariusz Piatkowski]

Alright, so a quick update.

I did the following:

1) split up the contents of SENT folder into multiple child folders, that got me several with approx 1000-1500 msg each, a lot better than the single massive SENT folder

2) I re-indexed all the current folders, no errors found

3) I shut PMMail down, backed it up as both dir copy and ZIP

4) I installed the NEW application into a separate directory (so no overwrite over the old one) ...and....drumroll...here is where I ran into an issue...

During the install it appears that the new application process completed successfully. I do have the PMMail folder with all icons present. However, during the said install the Migrate tool was automatically kicked-off. I pointed it to the old PMMail install and told it to COPY stuff, do not delete the old one.

It is during this processing that I am getting an error message, see attachment for a screenshot of the error itself and some variable values in the inspection window.

Specifically, what I am wondering is whether the Migration tool crashes because it's attempting to copy a file which in the NEW install is currently 0 (zero) bytes. After all, it gave me an option to Migrate during the install, and as such I have not actually ran the new PMMail even once by this point in time.

The specific command that fails is: {"copy "'dst_dir'\Tools\group.db" "'dst_dir'\Tools\group.db.BAK" '}, where: dst_dir = G:\APPS\TCPIP\PMMail3\Accounts, and where the matching contents of that directory are:

Directory of G:\apps\tcpip\pmmail3\accounts\tools

10-13-17   2:10p     31118           0  addr.db
10-13-17   1:29p      3474           0  assoc.db
 3-26-17   5:35p       289           0  books.db
 2-08-11   8:19a        92           0  csetalii-default.db
10-13-17   1:36p         0         124  csetalii.db
10-05-98   2:00a       350           0  DirSvcs.db
 7-18-17   3:56p         0           0  GROUP.DB
        7 file(s)      35323 bytes used

Sooo....I have a half-baked install at the moment, and starting the NEW PMMail produces a good looking account structure, however, with zero messages. To be expected since the migration failed. I can certainly re-do all of this, however, one thought is troubling me: even in my old PMMail install that group.db file is 0 (zero) bytes as well.

Here are the matching entries for this directory in the old install:

Directory of G:\apps\tcpip\pmmail\tools

9-29-17   2:53p     31118           0  ADDR.DB
10-03-17  12:36p      1137           0  ASSOC.DB
 3-26-17   5:35p       289           0  BOOKS.DB
10-05-98   2:00a       350           0  DirSvcs.db
 7-18-17   3:56p         0           0  GROUP.DB
        5 file(s)      32894 bytes used

Given the date/time stamp of the group.db file present in the NEW directory it certianly appears that the Migration process successfully copies it over in the first place, but then fails later on as it attempts to create a backup???

What is stored in that file anyways???

EDIT
====
Also attached the migratePMM.log-20171013-14-58-17, which I believe is the Migration Tool log. Unfortunately I do not see the failure point in there, but that could be just due to a timing issue?

I will try to log a ticket for this issue...

[guzzi]

Error 16 - Label not found

Explanation:

A SIGNAL instruction has been executed or an event for which a trap was set with SIGNAL ON has occurred, and the language processor could not find the label specified. You might have mistyped the label or forgotten to include it.

The associated subcodes are:

001

    Label "label_name" not found

Doesn't mean much to me.  The fact that the file is 0 bytes might be because you don't have any groups defined? Wouldn't effect a copy operation anyway. Something triggers an error condition in the rexx script. A bug report is a good idea, and drop it in the mailinglist too.

[ivan]

Have you tried just copying the folders over to your new install rather than running the migration tool?  You might need to re-index the folders after.

I don't know if this would work but since you have a backup it shouldn't be a problem to try and if it does work you are up and running.

[Dariusz Piatkowski]

ivan,

I'm afraid that is not going to work given how old my version of PMMail is. Reading through the documentation it specifically states that the Migration tool makes certain changes to the data files in order to make them usable by the new release.

I did file a request for a new ID on the bugtracker, but it's been a couple of hours now and I still have not received anything in my email. I'll give it a bit more time.

[Doug Bissett]

Are you suggesting that I put NEW child folders into the account specific default SENT parent folder? I thought of doing this before but I always assumed that it was a bad idea to mess with this folder. It certainly makes a lot of sense, and for starters I would be able to do a search in the SENT folders, which today causes my PMMail to simply crash.

Yes. Child folders in Sent are okay, but they should all be sent files, or you might get some strange entries in the list. That is, don't try to put both sent, and received, files in the same folder. The display is a little different.

OK, so a specific issue I have is that I only have about 1.5Gig left on my application drive (also happens to be my boot drive, but that's the legacy of the original install I did). So the option I have is to move my current pmmail directory structure to a different drive, where I have a bunch of space, wipe it off of my boot drive and then do the WarpIn based install. Will the installer itself attempt to migrate the old PMMail stuff during the install, or is this a separate process one does later once the NEW PMMail install is completed?

It is really a separate process, but the migration tool is started after the WarpIn install is done, and before it terminates. In this case, I would suggest installing to the new location, and just let the tool do the copy. That is supposed to work, and it should give you the default setup, with all of your accounts, and files. Don't touch your old install, at this point. It should still work.

The 2nd question is: does the migration tool/process simply need to find a disk directory structure of the old PMMail, or is it looking for some application specific install/config data?

It actually looks for the INI entry that the old PMMail puts in one of the INI files (USER.INI, I think). That should point to the old PMMail, but many users have managed to lose that entry over the years, so i would be surprised if it is there, and correct. Just point the migration tool to the old base PMMail directory, and it should do the job.

Basically I am wondering if my plan to move the current install to the spare drive has a chance of working?

I think it should work, but I don't remember doing a copy to a different drive. Let's see what happens.

UHOH! something has come up. I will have to follow up later...

[Dariusz Piatkowski]

Well, in the end I found out that the GROUP.DB file being 0 (zero) bytes in length is what was causing the problem. Simply including a digit (I used a 0) in the contents of the file allowed the migration tool to successfully pass that check and eventually produced a working PMMail application.

On my system attempting to do a CLI copy of that very same file produces an error, which I believe is to be expected. XCOPY on the other hand, which is also used during the migration process, does successfully copy the file over to the NEW install. At least that's what I have been able to surmize given the results I see in the filesystem folders.

OK, problem solved. I am happy to report that SMTP Authentication works well and that other than this one "gotcha" the massive amounts of my emails across three separate accounts were migrated without any other problems. This is pretty impressive given the amounts of data!

[Doug Bissett]

Well, in the end I found out that the GROUP.DB file being 0 (zero) bytes in length is what was causing the problem. Simply including a digit (I used a 0) in the contents of the file allowed the migration tool to successfully pass that check and eventually produced a working PMMail application.

Well, I am glad that you got it. GROUP.DB  is supposed to contain the address groups. If there are no address groups, It can be zero bytes.

On my system attempting to do a CLI copy of that very same file produces an error, which I believe is to be expected. XCOPY on the other hand, which is also used during the migration process, does successfully copy the file over to the NEW install. At least that's what I have been able to surmize given the results I see in the filesystem folders.

That would seem to be somewhat strange. I can copy a zero byte file, with no troubles, so there must be something wrong with that file. Possibly a bad EA, or something like that. Stranger things have happened during a migration.

Given the date/time stamp of the group.db file present in the NEW directory it certianly appears that the Migration process successfully copies it over in the first place, but then fails later on as it attempts to create a backup???

The backup copy is made after the main xcopy, and before the tool modifies the file. In this case no modifications are needed, because it was empty. The copy should have worked though. The only conclusion is that the file was somehow defective. I do know, that there is a flaw in the kernel handling of defective EAs, but trying to prove that now, is probably impossible. If it is the kernel EA problem, ZIP would also crash on it, but adding the "0" seems to have fixed the problem, somehow. Now, you don't have a broken file.

OK, problem solved. I am happy to report that SMTP Authentication works well and that other than this one "gotcha" the massive amounts of my emails across three separate accounts were migrated without any other problems. This is pretty impressive given the amounts of data!

Happy to hear that you got it. You probably should do a Re-Index all, in each account, before doing too much more. If GROUP.DB  was not being used, it should be empty. You may want to open the file, in the editor, and remove the "0" that you put into it, otherwise, you should be good to go. Don't forget to turn off the "Leave messages on the server" flag, in each account (Receive tab), if you got that turned on. If you don't mail will pile up in the server, and that usually causes mail to be rejected.

Don't assume that everything still works exactly the same as before, but most of it will. The Help is much better, but not perfect, and the FAQ (there is a link in the PMMail Help menu) should be mostly correct.

Have you tried just copying the folders over to your new install rather than running the migration tool?  You might need to re-index the folders after.

That would be something to do, when nothing else will work. The problem was not in copying the folders anyway, it was in the GROUP.DB file, which is part of the address book. As I recall, the only changes are to adjust the paths to point to the new location, but there is nothing to adjust if it is empty. Copy works okay, on zero length files, so it should have worked on that one too.

I'm afraid that is not going to work given how old my version of PMMail is. Reading through the documentation it specifically states that the Migration tool makes certain changes to the data files in order to make them usable by the new release.

Actually, it will work, but there may be a few things that need to be manually updated. I wrote the migration tool, and I don't remember all of the things that it does. I would need to go and read the code (it is done with DrDialog, which is REXX), and the last change was in January of 2015.

I did file a request for a new ID on the bugtracker, but it's been a couple of hours now and I still have not received anything in my email. I'll give it a bit more time.

Hmmm. I am not sure who is doing that now. If it is who I think it is, it may take a day or two, but I will rattle his chain. I am not sure if it would be worth while to make a bug report, or not. I am the one who would need to look after it, and all I can do is write it off as a bad file in the original PMMail. This is definitely not the first time something like that has happened. Then again, it probably wouldn't hurt to document what happened, but I don't think there is anything to do to fix the problem, short of completely rewriting the tool, and that isn't likely to happen. It isn't perfect, but it does the job (sometimes with a few glitches).

Enjoy...

[Doug Bissett]

I did file a request for a new ID on the bugtracker, but it's been a couple of hours now and I still have not received anything in my email. I'll give it a bit more time.

Did you get a response to this? Apparently the ID was created, and sent. If you didn't get it, I will see if I can figure out what happened.

[Dariusz Piatkowski]

Hi Doug!

Did you get a response to this? Apparently the ID was created, and sent. If you didn't get it, I will see if I can figure out what happened.

Yes, yes I did, a bit later on in the day though. No problem, I will get the ID confirmed on Mantis though, otherwise what's the point of creating one?

You are right, I will not be logging this as a bug though. I was able to come up with a work-around, although the fact that my GROUP.DB file is empty is somewhat strange, because if I understood correctly this stores the Address Book Groups...and I certainly do have them present in my old PMMail...sooo??? LOL

I will just toss this up to the ancient version of PMMail I have been running. No sense spending any more time on something that's suspect to start off with.

Doug, thank you once again for all the insights, suggestions and help, greatly appreciated!

[Doug Bissett]

You are right, I will not be logging this as a bug though. I was able to come up with a work-around, although the fact that my GROUP.DB file is empty is somewhat strange, because if I understood correctly this stores the Address Book Groups...and I certainly do have them present in my old PMMail...sooo??? LOL

Probably easiest to just recreate the groups. They probably needed trimming anyway. 

I will just toss this up to the ancient version of PMMail I have been running. No sense spending any more time on something that's suspect to start off with.

It would be interesting to know what the real problem was. Obviously, there was something wrong with that file, since you couldn't do a CLI copy. However, trying to figure it out could be an exercise in frustration.

Doug, thank you once again for all the insights, suggestions and help, greatly appreciated!

Glad to help. I hope PMMail works well for you, and I hope we can find somebody to do more work on it. The project is mostly paused, at the moment, but it is definitely not dead.