Author Topic: JFS file corruption  (Read 16115 times)

Greg Pringle

  • Full Member
  • ***
  • Posts: 148
  • Karma: +0/-0
    • View Profile
JFS file corruption
« on: October 27, 2019, 07:18:55 pm »
I use JFS for large volumes and need back ups. When doing this in the past I used XCOPY /S /E which worked well. Then when the disks became too big I use cp -r which worked with larger data sets. Then both started failing.
Then I wrote some Java to do the same thing.
First it creates a text file listing of every file and directory with full paths.
On a 2T drive this could be about a 150MB file.
Then I read the listing and create smaller files limited by the number of files and the data size.
This created 110 smaller listings.
These listings could be used to copy the files to any drive or subdirectory.
However when I went to copy the files I had a JFS sytem crash about 8 listings in.
After some review I determined the file being copied  was currupt on the disk but had passed chkdsk.
It turned out I had to remove the file from the listing and about 20 more after it.
This was done by ediiting the listing file and then restaring the copy.
This approach copied the files but not the extended attributes.

Then I took the copy on the new SSD drive and put it in a fast machine which had an SSD boot drive.
The copy failed again. The speed was too fast between the SSD volumes for JFS to sync up.
It turned out to be two reasons. First, there needed to be some throttling when writing 400MB files and second there was a java problem. The java RandomAccessFile does not actually write in real time like standard ioctl disk access. Most people rely on a simple close() to end the file write. This failes on very speedy systems. Instead a RandomAccessFile.getFD().sync() is required before the close().

To make my life harder the SSD had a fault at about half way where a write would cause a kernal panic. If I wrote past this part of the disk it worked fine. The disk was a brand new SanDisk 3D 1T drive.


ivan

  • Hero Member
  • *****
  • Posts: 1557
  • Karma: +17/-0
    • View Profile
Re: JFS file corruption
« Reply #1 on: October 27, 2019, 08:05:22 pm »
Hi Greg,

There you have found the reason why I never use a JFS partition larger than 500GB.  Having been bitten in the past by a corrupt 1 TB JFS partition I soon learned to keep the partitions down to a manageable size for easy backups.  A case of once bitten twice shy. 

The only place where I don't have multiple partitions is in my NAS boxes, those have RAID 1 disks formatted as either Ext3 or Ext4.

Greg Pringle

  • Full Member
  • ***
  • Posts: 148
  • Karma: +0/-0
    • View Profile
Re: JFS file corruption
« Reply #2 on: October 28, 2019, 01:36:39 pm »
Hi Ivan, that is good advice. Normally I do try and keep the partitions smaller. The problems I encountered were not a complete JFS failue but only a section of the disk. It looks like the file pointers became crossed for a section of files after a power outage.
In the past I have had 3 complete JFS corruption failures. Each one came from filling the JFS partition more than 90% and then continurally writing and deleting files with the near full partition. That will eventually kill the partition.
I have not tried the drive reorganization feature as a solution. Instead I don't do the > 90% activity or just copy the entire drive to a new one.
As to raid. I have used (now obsolete) the EzRaid devices which are raid 1 mirrors. These were nice devices because the disks were not changed by the controller and could be removed and used without the controller. Lately I have some StarTech PCI Express 2.0 SATA III 6Gbps RAID (rail 1) controllers on some drives. These mark the disks and make them only work with a similar controller. These use the Marvell - 88SE9230 controller which is also used by other manufacturers. The StarTech can be used for the boot drive. It has a built in setup utility accessed before the boot.
« Last Edit: October 28, 2019, 01:38:14 pm by Greg Pringle »

Paul Smedley

  • Hero Member
  • *****
  • Posts: 2092
  • Karma: +159/-0
    • View Profile
Re: JFS file corruption
« Reply #3 on: October 28, 2019, 10:01:21 pm »
Hi Greg,

I use JFS for large volumes and need back ups. When doing this in the past I used XCOPY /S /E which worked well. Then when the disks became too big I use cp -r which worked with larger data sets. Then both started failing.
Then I wrote some Java to do the same thing.

Have you considering using rsync to do this? rsync will only copy across the files that have changed - significantly reducing the duration/CPU impact of the copy.

My daily rsync of ~600GB of data on my NAS to an external drive runs in a couple of minutes.

Cheers,

Paul.

roberto

  • Hero Member
  • *****
  • Posts: 810
  • Karma: +3/-6
    • View Profile
Re: JFS file corruption
« Reply #4 on: October 29, 2019, 10:08:13 pm »
Maybe this could be the problem you had. With copy and with xcopy this is currently happening.
In fact, if someone can tell me how I can execute the copy command, I appreciate it.
I enclose a photo with the copy command in green, and the result later in pink.

I saw this same problem the other day when running on a windows connected to a test samba server with arcaos, when creating a new folder from windows I gave it two different names on the samba server.


Greg Pringle

  • Full Member
  • ***
  • Posts: 148
  • Karma: +0/-0
    • View Profile
Re: JFS file corruption
« Reply #5 on: October 30, 2019, 12:51:46 pm »
Maybe this could be the problem you had. With copy and with xcopy this is currently happening.

The problem here is not with copy or xcopy. It appears to be with the desktop.
First rename the second version were both match to a new name.
Then you need to change the desktop name of the confused one to match the "Real Name".
After that you can change the names again using the normal methods
« Last Edit: October 30, 2019, 12:54:57 pm by Greg Pringle »

Greg Pringle

  • Full Member
  • ***
  • Posts: 148
  • Karma: +0/-0
    • View Profile
Re: JFS file corruption
« Reply #6 on: October 30, 2019, 01:00:42 pm »

My daily rsync of ~600GB of data on my NAS to an external drive runs in a couple of minutes.


This is definitely something to try.
In addition I have the need to make archival disk copies of complete disks which can be stored or moved to another machine.
When there is corruption of a range of files I have used my copy software to bypass the corruption. Not sure what rsync would have done.

Ian Manners

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 464
  • Karma: +10/-0
  • I am the computer, it is me.
    • View Profile
    • ComKal Networks Australia
Re: JFS file corruption
« Reply #7 on: February 21, 2020, 06:58:42 am »
After much experimenting since the start of November since I moved to SSD's.

Turn JFS Lazywrite OFF if using an SSD.

There is minimal performance difference on an SSD, and it has been almost 3 months since I have had the odd weird file corruption.
The files that were corrupting on me were due to scripts that would copy, delete, process log files, sometimes the end of the files would be full of nul information as if I had deleted the file to be appended to another before the information could be written to the new file.
Cheers
Ian B Manners

Dave Yeo

  • Hero Member
  • *****
  • Posts: 4787
  • Karma: +99/-1
    • View Profile
Re: JFS file corruption
« Reply #8 on: February 21, 2020, 08:21:17 am »
Logs can wear out SSD's due to the way they rewrite data, write a couple of bytes to a file and the SSD moves a 4k block of data around.

Ian Manners

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 464
  • Karma: +10/-0
  • I am the computer, it is me.
    • View Profile
    • ComKal Networks Australia
Re: JFS file corruption
« Reply #9 on: February 21, 2020, 10:02:59 am »
Hi Dave,

Makes sense but I was seeing problems from day 1 on two Western Digital Blue 500G SSD's, and one Western Digital Blue 1T SSD.
I suspect my problem was due to the double caching that was happening on both my server and my OS/2 primary computer.

RAM drive now stores everything that I do not mind is lost on the server, or is transient, including log files from InJoy FW etc that I really don't need but look at occasionally anyway. What I've done since is to move all data manipulation to the RAM drive (about 500MB in size) for a nightly stats run. The server is really only used for os2site.com and my personal email now so the only logs that grow on the SSD are Apache (http) and Weasel (mail).  SYSLOG etc are all transient.

To overcome any possible problems I have been backing up more regularly but expect to copy the drive(s) to a new one occasionally but I don't intend to do that for at least 6 months, if ever, depending on what my results are going forward and if there is really a need to TRIM these drives.
The SSD's have been to me, an experiment in power consumption, noise, heating, and speed - though on the internet side there is no difference, on the network inside there is a big difference.

PS..
The server now only uses 43.183 MiB or RAM :-))
« Last Edit: February 22, 2020, 06:04:37 pm by Ian Manners »
Cheers
Ian B Manners

Andi B.

  • Hero Member
  • *****
  • Posts: 811
  • Karma: +11/-2
    • View Profile
Re: JFS file corruption
« Reply #10 on: February 21, 2020, 12:37:22 pm »
I'm using SSDs for my systems since many years. I've even \TEMP and \TMP on SSDs and these dirs are used a lot especially when building software. But especially my older SSDs are all from Samsung which were tested at that times as being very robust. Usually I've some spare space on SSDs which is not partitioned too. So far no single failure or problem like Ian reported.

roberto

  • Hero Member
  • *****
  • Posts: 810
  • Karma: +3/-6
    • View Profile
Re: JFS file corruption
« Reply #11 on: February 21, 2020, 03:51:10 pm »
My experience is more like what Andi says, SSD no problem.
If I remember years ago having had a problem, and what I did was make a backup with the RAR compressor. Look at the log and see the files not copied by errors.
Format and restore it and voila, no more mistakes. I would say it was 6 or 7 years ago.
That is if some files were lost, but it was not important either.

Andi B.

  • Hero Member
  • *****
  • Posts: 811
  • Karma: +11/-2
    • View Profile
Re: JFS file corruption
« Reply #12 on: February 21, 2020, 04:17:49 pm »
In the last 1-2 years I've bought 2 WD SSDs too. Think these are Sandisk internally as IIRC WD bought Sandisk. These WD drives are slower than my Samsung EVO 8xx drives. Unfortunately the price for Samsung SSDs is a bit high compared to WD. For my main machines where I use the SSDs a lot I still prefer Samsung EVO.

Neil Waldhauer

  • Hero Member
  • *****
  • Posts: 1024
  • Karma: +24/-0
    • View Profile
    • Blonde Guy
Re: JFS file corruption
« Reply #13 on: February 22, 2020, 05:03:58 am »
I'm sure you are all using SmartAHCI to track your SSD. You can compare your SSD values to the ones I have here for a hard disk.

I'm using 2 TB server class hard drive.

Code: [Select]
smartctl version 5.37 [i386-pc-ibmvac365] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     TOSHIBA HDWD120
Serial Number:    Z7D7ETBAS
Firmware Version: MX4OACF0
User Capacity:    2,000,398,934,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Fri Feb 21 07:57:51 2020   
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84) Offline data collection activity
was suspended by an interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (14439) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: (   1) minutes.
Extended self-test routine
recommended polling time: ( 241) minutes.
SCT capabilities:        (0x003d) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   140   140   054    Pre-fail  Offline      -       68
  3 Spin_Up_Time            0x0007   150   150   024    Pre-fail  Always       -       281 (Average 225)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       161
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   126   126   020    Pre-fail  Offline      -       32
  9 Power_On_Hours          0x0012   099   099   000    Old_age   Always       -       10568
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       161
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       164
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       164
194 Temperature_Celsius     0x0002   187   187   000    Old_age   Always       -       32 (Lifetime Min/Max 17/37)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      8385         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Expert consulting for ArcaOS, OS/2 and eComStation
http://www.blondeguy.com

Pete

  • Hero Member
  • *****
  • Posts: 1281
  • Karma: +9/-0
    • View Profile
Re: JFS file corruption
« Reply #14 on: February 22, 2020, 06:32:00 am »
Hi

The latest smartctl - http://hobbes.nmsu.edu/download/pub/os2/util/disk/smartmontools-6.6-r4424.zip - works with both AHCI and IDE controlled drives and is more capable than the smartahci I have installed (based on smartctl 5.37) with SSDs , there are less "unknown attributes" in the "ATTRIBUTE_NAME" column.


Regards

Pete