Author Topic: Sloooooooooow Samba Server  (Read 2393 times)

djcaetano

  • Full Member
  • ***
  • Posts: 205
    • View Profile
Sloooooooooow Samba Server
« on: 2011.12.30, 04:19:42 »

  Hi There,

  I have been using Samba Server on eCS 2.0 for a while, but had never used it to transfer large amounts of data... until yesterday.
  I decided to make a backup using my local 100mbps network connection and mapped all OS/2 drives on the network so I could read them on another computer, which runs Windows 7.
  As expected, the drives are shown on Windows 7 "Network" and I am able to copy files ... but the copy is rather slow. The transfer rate is about 300KB/s... :/
  Also, my eCS box CPU goes 100% while transfers are being carried. Sometimes windows loses the connection also, and I am obligated to restart Samba Server so the copy may continue.

  Any clues on how to improve the transfer performance?

  Regards,

  Daniel Caetano
 

 

RobertM

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2034
    • View Profile
    • A.I.BuiltPC - using OS/2 Warp Server & eComStation for Custom Web and Database Solutions
Re: Sloooooooooow Samba Server
« Reply #1 on: 2011.12.30, 05:24:21 »
Ugh, I wish. I've seen the same issues with WSeB (on a virtually all Warp and eCS network) and with Windows Server 2003 R2 (on an entirely Windows network). I always suspected it was an issue with the underlying protocol (namely NetBIOS (stand alone or over TCP/IP)).

Not sure where Samba maintains its logs, but that may be a starting point - or if the other machines are using LanMan, you can check its logs - where you might find indications of the issues.
|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


DougB

  • Sr. Member
  • ****
  • Posts: 407
    • View Profile
Re: Sloooooooooow Samba Server
« Reply #2 on: 2011.12.30, 07:40:43 »
Quote
Any clues on how to improve the transfer performance?

Unfortunately, no. I too have been using SAMBA (various versions, and combinations) for quite a while. I have yet to find a combination that is really usable, although using Win7 as a client seems to work reasonably well, compared to using EVFS in eCS.

Actually, I do see a reasonable transfer rate of roughly 17,000 KB/s (IP Monitor widget) when transferring very large files to Win7, using Win7 as the client on a 1GB connection. It must be very large files, because it seems that things like directory listings take "forever" (roughly 50 KB/s, even on a 1 GB connection), and, if you have lots of small files, it spends more time looking up the directory entries, than it does transferring files. When I use EVFS as the client, it seems to top out at about 8,000 KB/s, which is about the same as what I see on a 100 MB/s connection. The slow directory access still applies, and long pauses (30 seconds) while doing it are all too common (usually locking out the desktop when it happens - sometimes, it never comes back, and a reboot is required).

Sadly, I find that I need to restart SAMBA every few minutes, when trying to do large numbers of files, and using RSync across the SAMBA network share is impossible. DSync works much better, but it takes far too long to be of much use. Using the RSync daemon, rather than SAMBA is probably much better, but that takes a lot of trial and error to get it working. Another option is to use Peter Moylan's FTP server and Firefox, or SeaMonkey, to do the file transfer.

djcaetano

  • Full Member
  • ***
  • Posts: 205
    • View Profile
Re: Sloooooooooow Samba Server
« Reply #3 on: 2011.12.30, 19:39:21 »
It must be very large files, because it seems that things like directory listings take "forever" (roughly 50 KB/s, even on a 1 GB connection), and, if you have lots of small files, it spends more time looking up the directory entries, than it does transferring files.
(...)
Sadly, I find that I need to restart SAMBA every few minutes.

  Well, it is exactly what I´ve just discovered. One of harddisks had only large files, and it was transferred in almost no time.
  The most bizarre thing is the CPU usage when Samba Server is sending files to a client. It just doesn't make much sense to me.

  Regards,

  Daniel Caetano

DougB

  • Sr. Member
  • ****
  • Posts: 407
    • View Profile
Re: Sloooooooooow Samba Server
« Reply #4 on: 2011.12.30, 23:16:34 »
Actually, I have never noticed unusual CPU activity while using SAMBA. In fact, I am always amazed at how little CPU usage there is, especially when nothing is happening, when it should be sending something. High CPU usage probably indicates that something is not right, somewhere, but I have no idea where to start looking.

RobertM

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2034
    • View Profile
    • A.I.BuiltPC - using OS/2 Warp Server & eComStation for Custom Web and Database Solutions
Re: Sloooooooooow Samba Server
« Reply #5 on: 2011.12.31, 18:43:11 »
Actually, I have never noticed unusual CPU activity while using SAMBA. In fact, I am always amazed at how little CPU usage there is, especially when nothing is happening, when it should be sending something. High CPU usage probably indicates that something is not right, somewhere, but I have no idea where to start looking.

That problem is probably JFS. Under certain intensive operations (large files or lots of files simultaneously) I have found that JFS for some reason will peg the CPU (or at least increase CPU use dramatically). I have found HPFS386 does NOT - even when (up to the 2GB limit) performing the same copy/move/write operations.

At first, I thought it was an IDE/SATA/they-use-the-CPU-not-dedicated-controller issue - but alas, the same results occur on SCSI U160 and U320 when using very high end controllers (though to a lesser extent since it's only JFS' use of the CPU and minimal CPU use from the SCSI activity).

I've not tested under HPFS, so I've got no idea how that would perform. The oddest thing is, I'd have expected those results from HPFS38 (and not JFS), since HPFS (386 and non) have a bunch more overhead when it comes to writing files.

For some reason, it's worse via network stuff. Perhaps the large file support was kludged onto the network support and the issue is that JFS is using that API and thus running into the poor performing kludge? Dunno...
|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


sXwamp

  • Full Member
  • ***
  • Posts: 228
    • View Profile
    • OS/2 Release Notes
Re: Sloooooooooow Samba Server
« Reply #6 on: 2011.12.31, 22:47:50 »
Quote
Any clues on how to improve the transfer performance?

Using the RSync daemon, rather than SAMBA is probably much better, but that takes a lot of trial and error to get it working.


It should take only about 5 minutes to setup Rsync (OS/2 Server) & Rsync (Win client).

http://www.os2notes.com/os2rsync.html

1) Download - 'Rsyncbackup'.
2) Add this to the rsyncd.conf and create the 'Windows' directory in the 'Backups' directory, start the Rsync Server:

[RsyncWin]
# hosts allow = 192.168.2.101
comment = backup storage area
path = /RsyncBackup/Backups/Windows
read only = false
list = yes
gid = root
uid = root

3) On Windows 7, download & install DeltaCopy

4) Start client, type in server IP and that's it.

Add the directories to backup and enable the scheduler,  set & forget !!!

« Last Edit: 2011.12.31, 22:58:37 by sXwamp »

DougB

  • Sr. Member
  • ****
  • Posts: 407
    • View Profile
Re: Sloooooooooow Samba Server
« Reply #7 on: 2012.01.01, 06:06:39 »
Quote
It should take only about 5 minutes to setup Rsync (OS/2 Server) & Rsync (Win client).

Well, I want an eCS client, and server. I have looked at RSyncBackup, but I haven't had the time to actually try to make it work. In fact, I don't really want to use that for backup, my preference is to use it to sync my laptop to my main system. That becomes a little more complicated, because I don't want to screw up my laptop (or the main system), so I need to take my time to determine what to do. I do use RSync to backup my main system to my USB drive, but that doesn't involve using the daemon, and I use a batch file that I built a couple of years ago. I dread the day when I need to restore the while thing, because USB (2) is very slow.

I never heard of DeltaCopy. I will look it up, it may be useful. I also would want to use it on WinXP, if it works. I see that there is also Syncrify, that does something similar, but it appears that there is no OS/2 version of that. I see some work ahead of me, next year.

DougB

  • Sr. Member
  • ****
  • Posts: 407
    • View Profile
Re: Sloooooooooow Samba Server
« Reply #8 on: 2012.01.01, 19:07:28 »
Quote
That problem is probably JFS.

I don't think so. I use JFS (as supplied by eCS 2.1) for all of my disks, except for one volume that is FAT32 for sharing with windows). I suppose the old JFS, as supplied by IBM, could have problems, especially if you attempt to use a cache of more than about 64 meg. I am using 256 meg, with no apparent problems.

Quote
It should take only about 5 minutes to setup Rsync (OS/2 Server) & Rsync (Win client).

That is pretty optimistic. 5 minutes, after trying 14 ways to set it up, maybe.

I am sorry, but I find your RsyncBackup package to be very complicated, and configured to run only on YOUR setup, when it should be very simple, and capable of running on anybodies setup. It took me a couple of hours, just to figure out what you did, and another couple of hours to sort out what you were really trying to do. After that, I got it to work, but only with the basics. More work is required.

Quote
3) On Windows 7, download & install DeltaCopy

4) Start client, type in server IP and that's it.

Add the directories to backup and enable the scheduler,  set & forget !!!

3) went well.

4) took a LOT more than just plugging in the IP.

Then, I needed to figure out that "Virtual directory" actually means the "Module" name in the RSync daemon config.  "Set and forget" is not something that can be done, until the user figures out what it is all about. I set it (incorrectly), it triggered, and did it's thing. Then, I went looking to see that it worked. It didn't do anything and there was no indication, that I could find, that it had failed. I am also not so sure that windows scheduler will trigger the action, unless the system is booted, and ready to go, at the assigned time, but that is a problem with windows scheduler.

I had slightly better results trying to use RSync on eCS as the client, but there are still some anomalies that I need to figure out.

Thanks anyway, but I need to find some time to figure out what you are trying to do, and suggest better ways of doing it. One of the most obvious is that you seem to assume that the %HOME% directory will always be C:\HOME (or, at least <boot drive>:\HOME). That is a very BAD assumption. You should use %HOME% so that it uses the environment variable, which should always be right, no matter where the user puts the directory. You also assume that SysBootdrive() doesn't work with classic REXX. It does, as long as the user has the appropriate (very OLD) fix pack installed. There is no need to know what the boot drive is anyway.

Anyway, most of this really belongs in another thread.

sXwamp

  • Full Member
  • ***
  • Posts: 228
    • View Profile
    • OS/2 Release Notes
Re: Sloooooooooow Samba Server
« Reply #9 on: 2012.01.01, 20:23:43 »

Thanks anyway, but I need to find some time to figure out what you are trying to do, and suggest better ways of doing it. One of the most obvious is that you seem to assume that the %HOME% directory will always be C:\HOME (or, at least <boot drive>:\HOME). That is a very BAD assumption. You should use %HOME% so that it uses the environment variable, which should always be right, no matter where the user puts the directory. You also assume that SysBootdrive() doesn't work with classic REXX. It does, as long as the user has the appropriate (very OLD) fix pack installed. There is no need to know what the boot drive is anyway.

Anyway, most of this really belongs in another thread.

1) Using it RsyncBackup for local copies is the easiest , just run install and your set.

   Here is an article that goes over RsyncBackup: http://www.os2notes.com/os2rsync.html

2) The WPS & scripts are only templates and the c:\home directory is only used as an example only( what other example should I use ?).  Instead of trying to figure out each different command for Rsync just look at the examples.  I guess the install script should just make them as real templates - do you know how to do that ? I only made one example for each different copy operation, so I'm not sure how 10 or 20 would make things easier to figure out.

3)  As for the layout & setup, I'm open to suggestions -  but there is only the server part, local copies & network copies.  

As for the couple of files copied to the boot partition, that's the easiest way to get both the WPS objects & scripts to work the same for everyone.  The main reason as I see it for not copying anything to the boot partition is that it can be wiped and reinstalled without losing anything, right ???  Well, there is nothing to lose - just run install again for RsyncBackup.

4) As a matter of fact everything is a template:

 a) if you want to do local copy, look in the local copy folder => WPS object, 'Local copy - c:\Home'
 b) make a copy of it & change the name, and modify the directories.
 c) the correct commands are there for each operation needed - as templates to modify to your system.


Quote
You also assume that SysBootdrive() doesn't work with classic REXX.

As for my programming skills and methods, they consist my me looking and copying what I see ( and have zero experience starting from a couple weeks ago).

Since you brought it up what's the correct way to do it now ???


Thanks,

Greggory



    
              
« Last Edit: 2012.01.01, 21:39:55 by sXwamp »

DougB

  • Sr. Member
  • ****
  • Posts: 407
    • View Profile
Re: Sloooooooooow Samba Server
« Reply #10 on: 2012.01.02, 23:27:54 »
Quote
1) Using it RsyncBackup for local copies is the easiest , just run install and your set.

Well, not quite. First, the user needs to figure out what you are doing, so they don't get some surprises. In the local section, you use the long form of the parameters, and they seem to be correct, except for "--dry-run" which is documented (but probably not understood by most users). In the Network section, you have used the short form of the parameters. You use "-aqzx" which is probably wrong. I would suggest using "-aqzX", or "-aqX", which might be faster, depending on the system, and the network link. "x" is not the same as "X". From the docs:
-X, --xattrs                preserve extended attributes
-x, --one-file-system       don't cross filesystem boundaries
The second one doesn't make sense in OS/2, while the first one is essential. It may also be a good idea to add the "-u" parameter, so that newer files in the target are not replaced, although that may not be the desired action, in some cases.

It would also be good to demonstrate the use of the "--delete" parameter. Without that, all of the old stuff that has been removed from the source directory will still exist in the target directory. That can result in a real mess, eventually.

Next, you should NOT put RSync.EXE and RSync.log, into the %ETC% directory (\MPTN\ETC). You should have RSync, properly extracted, into it's own directory, then put that directory into the Working directory field in the icons. The logs end up where they should anyway (<boot drive>:\VAR\LOG).

Quote
Since you brought it up what's the correct way to do it now

Replace this:
Code: [Select]
/***********************************************************************/
/* query the bootdrive  (didn't use SysBootdrive 'cause of OREXX)      */
/***********************************************************************/
IF BootDrive='' THEN
  DO
  path_to_os2_ini=VALUE(USER_INI,,"OS2ENVIRONMENT")
  BootDrive=FILESPEC(D,path_to_os2_ini)
  END
with this:
Code: [Select]
/***********************************************************************/
/* query the bootdrive  (didn't use SysBootdrive 'cause of OREXX)      */
/***********************************************************************/
IF BootDrive='' THEN
  DO
  BootDrive=sysBootDrive()
  END
This method will not work on any ancient REXX (pre Object REXX time frame), but then the user has the option of inserting the proper drive into the script.

It is also not good to make the user read through ALL of the code, to find places where they need to change things like the IP address of the backup server. You should probably use a config file, where a user can define things like that, then import the information into your scripts at run time. One config file, with all of that stuff would be sufficient, it doesn't matter if some of it isn't used in various scripts.

It would also be helpful, if you put a sample of how to copy something that is in the root of a drive, to the root of a drive on another system. That helps if a user wishes to use RSync to synchronize two systems.

I need to do more work with this. It is a good idea, but I think it can be done in a better way. A GUI interface, with check boxes, or radio buttons, for the options, might work a lot better.

sXwamp

  • Full Member
  • ***
  • Posts: 228
    • View Profile
    • OS/2 Release Notes
Re: Sloooooooooow Samba Server
« Reply #11 on: 2012.01.03, 03:51:25 »
Anyway, most of this really belongs in another thread.


Thanks DougB, move this to it's own post:

http://www.os2world.com/component/option,com_smf/Itemid,63/topic,4733.0/


Greggory