Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - Doug Bissett

Pages: [1] 2 3 ... 99
Storage / Re: JFS cache sizing, and system "speed-up"
« on: November 26, 2021, 11:10:59 pm »
Is XUL marked to load high? Otherwise the only follow-up is to see if it's been LXLITE compressed? (not sure if DLLs get processed by LXLITE, I think they do...?)

They are all marked for high code. LXLITE is whatever they were shipped as. I am going to stay with /cache:132000 on my main machine. That seems to work as well as anything else. Changing Lazy write to /LW:16,60,12 doesn't seem to increase performance much. but it does seem to smooth out some of the peaks and valleys. I think I will put that back to /LW:8,30,6 and see what happens. /MINBUFFER:4500 /MAXBUFFER:15000 seem to be good numbers, for me.

Right click on the preference and choose reset would probably work. Otherwise with the browser closed, edit prefs.js and delete the line, after backing up prefs.js.

Okay, that works. It is not exactly obvious what Reset is going to do.

Storage / Re: JFS cache sizing, and system "speed-up"
« on: November 26, 2021, 05:58:22 pm »

I tried that, with the appropriate changes. I really don't notice any difference, but I do have Firefox set to clear everything at shut down anyway. My RAMDISK doesn't retain information over a reboot (my choice, and I never tried it), but that shouldn't change anything. I didn't spend much time with it, but now, how do I remove that entry properly?

I did up my cache to 256000, on my main machine. It created the cache, but then Firefox won't start, complaining that XUL is defective. I put it back to 132000 and it is working.

Then, I have been playing with LW, MIN and MAX buffers. That is possibly making some difference. I will know more on Monday morning, when I do my backups. There is an indication that the single processor machine doesn't like it much, so that will go back to what it was (the LW part anyway).

Storage / Re: JFS cache sizing, and system "speed-up"
« on: November 24, 2021, 09:50:54 pm »
I'm curious what you guys see for your running systems?

This is from my main system, with cache:132000:
Code: [Select]
Nonswappable Memory analysis:
Apps & DLLs      = 0003C000 ->     240K -> 0.234M
Process overhead = 002F1000 ->    3012K -> 2.941M
DD allocated     = 0D453000 ->  217420K -> 212.324M
DOS              = 0001B000 ->     108K -> 0.105M
VDisk            = 00000000 ->       0K -> 0.000M
File system      = 00051000 ->     324K -> 0.316M
Kernel code      = 000B1000 ->     708K -> 0.691M
Kernel data      = 01166000 ->   17816K -> 17.398M
Kernel heap      = 00190000 ->    1600K -> 1.563M

Total            = 0EB93000 ->  241228K -> 235.574M

I am a bit puzzled about the DOS entry. DOS/WINOS2 is not installed on this system.

Storage / Re: JFS cache sizing, and system "speed-up"
« on: November 24, 2021, 07:24:04 pm »
Physical Memory:  2793 mb

Yeah. That seems to be common on newer machines (I have heard of one, that only leaves about 1 GB for the user). They fill up memory with stuff, and that leaves less room for the user. I don't think that has anything to do with what we are talking about though (I could be wrong). I should check to see what is left in UEFI mode.

In any case, it seems that all of this is very machine dependent, and the results of making changes can vary widely. The main problem is to determine if it is actually worth the effort.

Storage / Re: JFS cache sizing, and system "speed-up"
« on: November 24, 2021, 04:05:52 am »
This was the latest beta with the updated NSPR and NSS so that may be part of it.

Yeah. That version is not doing well. This is a typical ExceptQ report:

Storage / Re: JFS cache sizing, and system "speed-up"
« on: November 22, 2021, 08:56:42 pm »
The cachesize field implies that actual data cache is only 256M, which is what I'm struggling with.

That would appear to be the number of 4K buffers (but that is only a guess). If true, it matches your defined cache size.

Quote from: Doug Bissett on November 01, 2021, 08:25:47 pm

        /MINBUFFER:16000 /MAXBUFFER:84000

    Did you find a description of what these actually do? I have run out of places to look, and can't find anything.

Yes and no. There were some Warpstock presentations that touched on this. See "Dynamically Tuning the JFS Cache for Your Job" by Sjoerd Visser from 2009. P22 of that deck starts getting into the details of the JFS cache design, which is really about the logic of how the different buffers are handled, and the differences between actual data and metadata.

Interesting. It seems that they are all numbers referring to 4K data blocks.

Anyways, previously I had found my system to run best with VAL=3072. However, that meant my JFS cache would only go to about 64M, any attempt at bigger value would produce that "out of memory" boot message and a default cache size being substituted.

This doesn't make any sense. It implies that the cache, over 64M, goes into unreserved upper shared memory space (above what VAL reserves). Could be possible, I suppose, and it might explain some of the weird crashes that I see when I try to use larger values for VAL.

I think we can safely assume that it does not use (or even know about) PAE memory, so any memory above about 3.5G is likely out of the picture, although it is quite likely that it could use PAE memory, if somebody programmed it to use it.

I use VAL=2560, and going larger causes instability in my system (don't know why). The biggest JFS cache, that I can use, seems to be 132M, no matter what larger value I set in the IFS startup line. However, I just tried setting it to 256M, in a new install that defaults VAL to 1536 (way too small for actual use), and it did take it. The Sentinel memory watcher (XCenter widget) appears to show that the memory was allocated (from somewhere). I tried 512M (then 384M, then your number 1048567), VAL is still 1536, but now cachejfs shows only 132M, and Sentinel seems to confirm that. I never see an "out of memory" boot message.

So, it seems that 256M is the largest value that doesn't default to 132M, for me (that is probably a bug, I expect that 132M is the maximum acceptable). I don't see any indication of where the cache memory is allocated (private low, private high, shared low, or shared high).

Since I seem to be able to use 8 times the default cache size, I would think that changing MAX and MIN to 8 times their default value, would make sense, but that is only a guess. I need to do more reading. Thanks for the reference.

Which version of JFS are you using? The one that I am using is v1.9.9 from AN.


I found that that is only needed by VBox. It can cause problems elsewhere. I also gave up on VBox 5.0.51, and went back to VBox 5.0.6. For me, it is much more stable (meaning usable). I still need that setting, or I get the COM message. I do not put it in CONFIG.SYS. I do put it in the DragText Environment tab (without "SET "), in the VBox icon. That way, it is only used with VBox.

Also, you have VAL=1536, which may be too low. Thry increasing up to 3072 or whatever.

Yes, 1536 is way too low (even 2048 is too low). I use 2560, for all of my systems (even my antique IBM ThinkPad A22e, which is maxed out at 256 MB of real memory, but I don't try to run VBox on it). I have two machines that will run with 3072, but I still use 2560. The rest of my systems become very unstable if I use more than 2560. I also monitor the free upper, and lower, shared memory space. If I run win XP in VBox, lower shared memory drops to the 100 KB range, and upper shared memory still has about 1,200 KB. If I run win 7 in VBox, lower shared memory drops to the 50 KB range, and upper shared memory still has about 800 KB available. If anything else tries to run, lower shared memory can (and does) drop to zero. If that happens NOTHING works any more. I had to eliminate the AOO Quick Start thing, and Firefox Turbo, to clear enough shared memory space to be able to run win 7 in VBox. Even then, I sometimes need to reboot, before running VBox, or lower shared memory can be fragmented enough that something can't get what it needs, and bad things happen.

VBox, under OS/2 , is a delicate balancing act, and almost anything can push it over the edge.

Storage / Re: JFS cache sizing, and system "speed-up"
« on: November 01, 2021, 08:25:47 pm »
so I bumped the cache up to 768M

Since I haven't tried this, for a long time, I decided to give it a shot. How did you get it to take 768m (I would assume that you used 768000 (K) as the cache size. When I try that, CACHEJFS shows me:
Cache Size:  131072 kbytes
which I believe is the allowed max now.


Did you find a description of what these actually do? I have run out of places to look, and can't find anything.

Storage / Re: JFS cache sizing, and system "speed-up"
« on: October 30, 2021, 07:08:14 pm »
In other words: it's not about sheer performance, rather it's about how all the various controls available to us can produce better application performance.

There are a lot of things, that can affect application performance. Most of them make so little difference that they are not possible to measure. For instance, it takes less time to repaint the desktop, if it is a single, solid, color, than if it is a complicated pattern (like a photo). A small picture paints faster than a large picture. A large screen resolution takes more time, than a small screen resolution. Two screens take a lot longer than one screen. This becomes much more obvious, if you are using a program, like VNC,  to operate another computer over the internet.

You can also position data on your disk, so it is easier to get to it (more effective on a spinning disk, than a SSD). The outer edge of the disk spins faster than the inner edge (modern disks take advantage of that, so it doesn't make as much difference). Once you get there, the data transfer is a little faster (although modern disks also use internal cache, so it probably doesn't make any difference, as long as it is enabled).  File fragmentation also enters this equation.

If you use a RAMDISK, it is faster to read/write directly, than to need two steps to write to cache, then to the program. This may also apply to devices like NVME drives.

Various formats operate at different speeds. FAT is probably the fastest. FAT32 is likely the slowest (not counting optical devices). Enabling Lazy Write (all formats) often slows it down, especially when the cache fills up. Of course, each format has it's own uses.

Eliminating background processes can speed things up, until you need one of them.

There are many more things that can affect application performance. One, that most people forget about, is that it takes a lot longer to run a program, when something crashes, and you need to restart the program, or, worse, reboot the computer. Overall stability is one of the most important performance considerations. I always found that making the cache size too big, contributes to instability.

Storage / Re: JFS cache sizing, and system "speed-up"
« on: October 29, 2021, 02:44:05 am »
FAT is not FAT32. They are different drivers, with different parameters, and different cache. Using incorrect cache size probably just uses the default size, but it could also just drop the higher bits, and use whatever is left over. If that turns out to be something usable, it will likely work.

No, I haven't tried it. I no longer use FAT, except for the ArcaOS USB stick installer. Working with USB is affected by far too many outside factors, to even consider doing a copy to test speed. Do it 1,000 times, after rebooting each time, with consistent results, and I may find a reason to try it.

Storage / Re: JFS cache sizing, and system "speed-up"
« on: October 28, 2021, 08:24:45 pm »
I'm doing cache testing lately. With interesting results. But the difference with you is that I am testing with this:

swappath=c:\ 0 4096


Uhmm. That is totally wrong. From Help DISKCACHE:
Code: [Select]
DISKCACHE Command: n Parameter

Specifies a number from 48 through 14400 that indicates the number of 1024-byte blocks (or 1KB blocks) of storage to be used for control information and programs in the disk cache buffer. The default value is d, which is set during installation and based upon the amount of system memory. You can reset this value to a numerical value.

To set your disk cache size to 128KB, type the following in the CONFIG.SYS file:

The n parameter is what you say is "4096000". So, 14400 is maximum. If your number works, at all, it is only because it wraps to a smaller number, somewhere between 48 an 14400.

DISKCACHE is for FAT anyway, so it is probably not, at all, helpful to play with the default.

Utilities / Re: Backup solutions
« on: October 28, 2021, 08:05:31 pm »
ZIP does files greater than 2 GB (in and out), as long as you use the ZIP/UNZIP supplied by RPM/YUM (my biggest ZIP backup file is 27 GB). I did try RAR, years ago, and found that ZIP is more user friendly (and does get updated occasionally). It is important to use the "y" parameter, so it handles symlinks properly, and the "S" parameter for hidden, and system, files. No "special" handling is required for either boot drives (even when you are booted from them), or data drives. The same command does work properly. You can also split it into more than one file, if you want to (the -s <size> parameter). I use the RAMDISK as a temporary storage place, while it is ZIPing, if there is enough room (the -b <temp location> parameter). That eliminates a lot of seek time. It still needs to copy the final result to the final location, but it is faster overall (especially when using spinning disks, but even with a SSD, it is a little faster - NVME can be a little faster than a RAMDISK). I do multiple ZIPs, at the same time (up to 6 of them). Some planning is required, for that to work with minimal interference, but it is always faster than doing them individually.

To restore a drive, or just individual files, I use the Arca Noae Archive Tool (it is also called ZIPPY, I think). For a full drive restore, I format it first, but the tool offers various ways to do a restore, while keeping existing files. I have done restore many times (even for multiple drives, at the same time, and usually not for a failure), with no problems. You still need to manage your backups, and you will still have problems if bad files get into your backups (with no other, older, backup to recover them from), but that has nothing to do with using ZIP as a backup tool.

I gave up trying to manage incremental, and differential, backups. It is always a case of "get it done now", and trying to figure out what needs to be restored, from multiple files, takes a lot longer than just doing a full restore. Of course, I also have a nightly backup (scheduled by the Drag Text Schedule feature), using RSync, where I can recover files that were changed after the full ZIP backup was done. UNZIP can be done using the ArcaOS installer System Management tools, if you don't have another operational (or partly operational) boot system.

Hardware / Re: Display-port and HDMI
« on: October 28, 2021, 08:06:52 am »
Some years ago, I played with HDMI, and Display Port, on my Lenovo ThinkPad T510, and L530. I don't recall having any video problems using a TV as a screen. You do need to know how to switch, using the Fn key combination, and that probably needs to be done BEFORE starting to boot OS/2.

Utilities / Re: Backup solutions
« on: October 28, 2021, 07:55:15 am »
I ZIP whole drives, then RSYNC those ZIP files to other places (NAS, USB). A typical ZIP command is:
Code: [Select]
zip.exe -9yrS CDRIVE.ZIP C:\* -x@exclude.list 2>&1 | tee.exe CDRIVE.LOG
The file exclude.list contains a list of files that I don't need to back up. The ZIP file, and the log file, go into the current (Working) directory.

Storage / Re: JFS cache sizing, and system "speed-up"
« on: October 21, 2021, 07:16:45 pm »
OK, so I've had my VIRTUALADDRESSLIMIT=3072 setup in CONFIG.SYS for quite some time, and utilized a JFS cache of 256M. Seemed to work fine, no issues. Keep in mind the underlying disk is a SSD (Samsung 850Evo).

Personally, I always use VAL=2560, which seems to be a sweet spot, for my usage. At least two of my systems get very unstable if I use more than 2900 (and don't run very long, if I go to 3000). I do monitor the available lower, and upper, private, and shared, memory (using Above512, and a logging script). I find that the lowest I ever get with upper shared memory, is somewhere in the 800K range. Lower shared memory does get down to the 35K range (when I run VBox), but most of the time, it is above 100K. When it does get below 100K, I try to prevent further programs from running, because it sometimes does go to zero, with bad results (it seems that some programs never check to see if they got lower shared memory, before they try to use it, and then they can't recover). Of course, I do use HIGHMEM to set as much as possible to use high shared memory. I find that it is necessary to run HIGHMEM again, after some updates. It doesn't always find something that got changed, but sometimes it does.

For those who don't know, VAL has absolutely nothing to do with available real memory. I use VAL=2560 on my antique ThinkPad, which only has 256 Mb of real memory. What it seems to do, is increase the amount of upper shared memory, that is available.

I find it interesting, that private memory (upper, or lower) rarely changes value. Almost everything allocates shared memory, but that could be because Above512 is lying to me.

So off I went experimenting a little bit:

Been there, done some of it. I usually use JFS cache at 132K (on machines with more than 1 GB of real memory), which seems to be a bit of a sweet spot (64K, or 10% of real memory, is default). Using more tended to end up not helping much (it has been a few years since I tried that), and I found that the systems became somewhat less stable. Usually the problem was an unexplained hang, although I seem to be able to reproduce a similar problem by writing huge (20 GB on JFS) files to USB devices (JFS), while using the default cache size.

I never played with buffer sizes, and cache size doesn't seem to matter if it is using a spinning disk, or a SSD. A SSD, of course, has no seek time, or spin, delay, so it is faster.

I gave up using HPFS, years ago, because the HPFS cache goes into lower shared memory, which is already in very short supply. JFS does have other, significant, advantages over HPFS.

The apps are loading consistently faster, on multiple attempts.

A lot of that "improvement" depends on timing. If the program parts have not been flushed from the cache, the load time is faster (even when using a small cache). Using a larger cache does increase the probability that most of it will still be there, but it depends on what you do between starts.

Obviously not on the first attempt, but openning OO the 2nd and 3rd and 4th time is faster.

If you are using the AOO QuickStart feature, that may be what you are seeing. From what I see, QuickStart keeps parts of the program available in memory, after you close it, but it looks, to me, like the first program load still needs to load everything. QuickStart does work around DLL unload problems when using upper shared memory, without the kernel fixes that ArcaOS has.

FWIW, I have turned off all Lazy Write settings (FAT32, and JFS). That doesn't seem to affect system response much, but it does seem to make the system less likely to hang, especially when writing large files to USB.

Pages: [1] 2 3 ... 99