Maked as: Large
Hello,
here' s a rexx script for you all that you can use to determine if you've got duplicate files on your computer.
The script searches through your whole computer and calculate the CRC32 value and mark each files Extended Attribute (EA) with the crc32 value and the date it was last modified.
It also store information about each file if you set up a database with e.g.
UnixODBC Database Administrator, type of database shouldn't matter just as long as it work and have a basic structure described further down. If you have a database present, the code will try to extract information about the files that have the same CRC32-value (files that contain the same data).
Prerequisite:
rxCRC32 extend rexx to support calculation of CRC32-values.
rxODBC (see attached file if you log on) extend rexx to support ODBC connections similar to
rexxSQL that is included with
UnixODBC.
Restrictions:
rxODBC support multithreaded calls so that the code can do other things while the database server process a SQL command to retrieve information, while
UnixODBC doesn't seem to support multithreaded calls. You can't therefore set the last parameter in rxSQLCommand, rxSQLFetch to -2, -3, ... etc. as rxODBC then can't call
UnixODBC as it barf. The script has therefore been designed to do everything in sequence, just as it would have worked with
rexxSQL.
It's not possible to calculate values for large files (>2Gb) since rexx can't handle it. The code can be rewritten easily though if a solution should appear that require modification.
Parameter:
Drive to start to search through. You can therefore type X: to let it skip the first ones.
Info:
The output format of the file is CSV, with the database column names on the first row and data on row 2 and beyond, that look like.
"COMPUTER","DRIVE","PATH","NAME","CRC32","MODIFIED"
"....computer name...","...drive...","...path...","...file name...","...crc32...","...last modified..."
The code has been tested on 2 computers on my home network with the database engine PostgreSQL 8.0i by Lorne Sunley.
ODBC setup
1) Ensure that the connection has been labled "FileSystem" (without the quotes) in the column "DSN Name" where you add the connection (User DSN, System DSN or...)
2) No password may restrict access, you'll have to provide one in the connection settings if necessary (doesn't apply to text, dbf and SQLite databases as they don't rely on a database engine)
Database structure
Table name: file
Column names: computer, drive, path, name, crc32, modified
All columns set to type text (except the last that can have date time stamp set).
/* Set EA of file with information about CRC32-value */
/* File name: rxcrc32.cmd */
/* Parameters: drive */
/* Example: rxcrc32 E: */
database = 'FileSystem'
IF LENGTH( ARG(1) ) = 2 THEN
start_drive = ARG(1)
ELSE start_drive = 'C:'
c_name = VALUE( 'HOSTNAME',, 'OS2ENVIRONMENT' )
IF LENGTH( c_name ) = 0 THEN DO
c_name = VALUE( 'USER',, 'OS2ENVIRONMENT' )
IF LENGTH( c_name ) = 0 THEN
c_name = 'UNKNOWN'
END
out_file = database||'_'||c_name||'.csv'
start_time = TIME( 'S' )
CALL RxFuncAdd 'SysLoadFuncs', 'RexxUtil', 'SysLoadFuncs'
CALL SysLoadFuncs
CALL RxFuncAdd 'rxSQLLoadFuncs', 'rxODBC', 'rxSQLLoadFuncs'
CALL rxSQLLoadFuncs
/* Load External RxCRC32 Routines: */
CALL RxFuncAdd "RxCRC32Ver", "RXCRC32", "RXCRC32VER"
CALL RxFuncAdd "RxCRC32Init", "RXCRC32", "RXCRC32INIT"
CALL RxFuncAdd "RxCRC32Done", "RXCRC32", "RXCRC32DONE"
CALL RxFuncAdd "RxCRC32Update", "RXCRC32", "RXCRC32UPDATE"
DO i = 1 TO 2
CALL rxOut i, ''
END
DO i = 3 TO 5
CALL rxProgress i, 0
END
CALL SysCls
db_writes = 0
CALL SysSleep 1
drives = SysDriveMap( start_drive, 'LOCAL' )
DO i = 1 TO WORDS( drives )
CALL SysCls
CALL rxOut 1, "Searching for files on drive" SUBWORD( drives, i, 1 )
CALL rxProgress 5, i / WORDS( drives )
CALL rxProgress 4, 0
/* Find all files in subdirectories on drive */
CALL SysFileTree SUBWORD( drives, i, 1 )||'\*.*', 'file', 'SFO'
/* Connect to database with the name FileSystem */
db_support = rxSQLConnect( 'DUPLICATES', 'DSN='||database )
DO j = 1 TO file.0
retval = SysGetEA( file.j, 'LASTWRITE', 'w_time' )
f_time = SysGetFileDateTime( file.j, 'WRITE' )
f_drive = FILESPEC( 'D', file.j )
f_path = FILESPEC( 'P', file.j )
f_name = FILESPEC( 'N', file.j )
/* Check if the CRC32 value has been calculated before so we don't have to do it again */
/* Also determined by the modification stamp we've set */
IF retval = 0 THEN
IF LENGTH( w_time ) > 0 THEN
IF f_time = w_time THEN DO
retval = SysGetEA( file.j, 'CRC32', 'CRC' )
IF retval = 0 THEN
IF LENGTH( crc ) > 0 THEN DO
IF Add2DB() THEN
CALL rxOut 2, FORMAT( TIME( 'S' ) - start_time,, 0 )"s, CRC32:" crc "@" f_time "for" f_name
CALL rxProgress 4, j / file.0
/* Go to next file to shorten the processing time */
ITERATE j
END
END
crc = RxCRC32Init()
/* Calculate CRC-32 of File: */
BufferSize = 4096
f_size = STREAM( file.j, 'C', 'QUERY SIZE' )
IF f_size = 0 THEN ITERATE j
c_size = 0
o_time = TIME( 'S' )
post_time = o_time
pre_time = o_time
IF STREAM( file.j, 'C', 'OPEN READ' ) = 'READY:' THEN
DO UNTIL LENGTH( Buffer ) \= BufferSize /* Traverse File */
Buffer = CHARIN( file.j,, BufferSize ) /* Fill Buffer (there may be a faster way :-) */
c_size = c_size + LENGTH( Buffer )
CALL rxProgress 3, c_size / f_size
crc = RxCRC32Update( crc, Buffer ) /* Update CRC with Buffer */
IF TIME( 'S' ) - o_time < pre_time THEN DO
BufferSize = MIN( f_size, BufferSize + 1 )
pre_time = TIME( 'S' ) - o_time
END
ELSE IF TIME( 'S' ) - o_time > post_time THEN DO
BufferSize = MAX( f_size, BufferSize - 1 )
post_time = TIME( 'S' ) - o_time
END
o_time = TIME( 'S' )
END
CALL STREAM file.j, 'C', 'CLOSE'
CALL rxProgress 4, j / file.0
/* Complete Calculation: */
crc = RxCRC32Done( crc )
CALL SysPutEA file.j, "CRC32", crc
CALL SysPutEA file.j, "LASTWRITE", f_time
/* Report the result: */
IF Add2DB() THEN
CALL rxOut 2, FORMAT( TIME( 'S' ) - start_time,, 0 )"s, CRC32:" crc "@" f_time "for" f_name
END
CALL rxSQLDisconnect 'DUPLICATES'
END
CALL SysCls
db_support = rxSQLConnect( 'DUPLICATES', 'DSN='||database )
CALL Export2CSV
CALL rxSQLDisconnect 'DUPLICATES'
CALL rxSQLDropFuncs
CALL SysDropFuncs
Return 0
/* display_on_row, text */
rxOut: PROCEDURE EXPOSE __last_write. __width
row = ARG(1)
IF \DATATYPE( row, 'W' ) THEN row = 0
IF \DATATYPE( __last_write.row, 'N' ) THEN
__last_write.row = TIME( 'S' )
IF \DATATYPE( __width, 'W' ) THEN
PARSE VALUE SysTextScreenSize() WITH . __width
IF __last_write.row + 1 < TIME( 'S' ) THEN DO
IF DATATYPE( row, 'W' ) THEN
PARSE VALUE SysCurPos( row, 0 ) with prev_row prev_col
SAY LEFT( ARG(2), __width )
IF DATATYPE( row, 'W' ) THEN
CALL SysCurPos prev_row, prev_col
__last_write.row = TIME( 'S' )
END
Return 0
/* display_on_row, percentage */
rxProgress: PROCEDURE EXPOSE __prev_progress. __last_write. __width
row = ARG(1)
IF \DATATYPE( __width, 'W' ) THEN
PARSE VALUE SysTextScreenSize() WITH . __width
IF DATATYPE( ARG(2), 'N' ) THEN
DO
progress = MIN( TRUNC( ( __width - 4 ) * ARG(2) ), ( __width - 4 ) )
IF progress <> __prev_progress.row THEN DO
CALL rxOut row, LEFT( COPIES( '█', progress )||COPIES( '░', ( __width - 4 ) - progress ), ( __width - 4 ) )||RIGHT( TRUNC( 100 * ARG(2) )||'%', 4 )
__prev_progress.row = progress
END
END
Return 0
Add2DB:
dup = 0
sql_path = Insert_val( '\', f_path )
sql_name = Insert_val( "'", f_name, "\" )
IF db_support THEN DO
IF rxSQLCommand( "DUPLICATES", "SELECT crc32 FROM file WHERE computer = '"||c_name||"' AND drive = '"||f_drive||"' AND path = '"||sql_path||"' AND name = '"||sql_name||"'" ) THEN DO
IF DATATYPE( DUPLICATES.CRC32.0, 'W' ) THEN
DO k = 1 TO DUPLICATES.crc32.0
IF DUPLICATES.crc32.k = crc THEN dup = 1
END
IF dup <> 1 THEN
RETURN rxSQLCommand( "DUPLICATES", "INSERT INTO file ( computer,drive,path,name,crc32,modified ) VALUES ( '"||c_name||"','"||f_drive||"','"||sql_path||"','"||sql_name||"','"||crc||"','"||f_time||"' )" )
ELSE RETURN rxSQLCommand( "DUPLICATES", "UPDATE file SET crc32 = '"||crc||"', modified = '"||f_time||"' WHERE computer = '"||c_name||"' AND drive = '"||f_drive||"' AND path = '"||sql_path||"' AND name = '"||sql_name||"'" )
END
ELSE IF \rxSQLCommand( "DUPLICATES", "INSERT INTO file ( computer,drive,path,name,crc32,modified ) VALUES ( '"||c_name||"','"||f_drive||"','"||sql_path||"','"||sql_name||"','"||crc||"','"||f_time||"' )" ) THEN
IF rxSQLInfo( 'M' ) = 1 THEN CALL rxOut 6, rxMessage( 'M' )
ELSE RETURN 1
IF rxSQLInfo( 'E' ) = 1 THEN CALL rxOut 7, rxMessage( 'E' )
IF rxSQLInfo( 'S' ) = 1 THEN CALL rxOut 8, rxMessage( 'S' )
END
CALL rxSQLDisconnect
db_support = rxSQLConnect( 'DUPLICATES', 'DSN='||database )
RETURN db_support
Insert_Val: PROCEDURE
add_text = ARG(1)
modify_text = ARG(2)
IF LENGTH( ARG(3) ) > 0 THEN
insert_text = ARG(3)
ELSE insert_text = add_text
from = POS( add_text, modify_text )
DO WHILE from > 0
modify_text = INSERT( insert_text, modify_text, from - 1 )
from = POS( add_text, modify_text, from + LENGTH( insert_text ) + 1 )
END
RETURN modify_text
Export2CSV:
IF db_support THEN DO
IF rxSQLCommand( "DUPLICATES", "ANALYZE file" ) THEN
CALL rxOut 3, "Analysis done..."
row_count = 0
row_no = 1
IF STREAM( out_file, 'C', 'OPEN' ) = 'READY:' THEN DO
drives = SysDriveMap( , 'LOCAL' )
CALL rxOut 2, 'Quering database for duplicate files... Please wait!'
IF rxSQLCommand( "DUPLICATES", "SELECT file.computer AS computer, file.drive AS drive, file.path AS path, duplicate.name AS name, duplicate.crc32 AS crc32, file.modified AS modified FROM file, ( SELECT dup.crc32, dup.name FROM file AS dup WHERE dup.computer = '"||c_name||"' GROUP BY dup.crc32, dup.name HAVING ( COUNT( dup.crc32 ) > 1 ) ORDER BY dup.crc32 LIMIT 100 OFFSET 0 ) AS duplicate WHERE file.name = duplicate.name AND file.crc32 = duplicate.crc32 ORDER BY file.crc32" ) = 1 THEN
DO WHILE row_no > 0
row_count = row_count + 100
CALL rxOut 1, row_no '(' row_count ')'
CALL rxProgress 4, 100 / row_no
IF Write2CSV( ( row_no = 1 ) ) THEN LEAVE
IF rxSQLCommand( "DUPLICATES", "SELECT file.computer AS computer, file.drive AS drive, file.path AS path, duplicate.name AS name, duplicate.crc32 AS crc32, file.modified AS modified FROM file, ( SELECT dup.crc32, dup.name FROM file AS dup WHERE dup.computer = '"||c_name||"' GROUP BY dup.crc32, dup.name HAVING ( COUNT( dup.crc32 ) > 1 ) ORDER BY dup.crc32 LIMIT 100 OFFSET "||row_count||" ) AS duplicate WHERE file.name = duplicate.name AND file.crc32 = duplicate.crc32 ORDER BY file.crc32" ) <> 1 THEN
DO
IF rxSQLInfo( 'M' ) = 1 THEN CALL rxOut 6, rxMessage( 'M' )
IF rxSQLInfo( 'E' ) = 1 THEN CALL rxOut 7, rxMessage( 'E' )
IF rxSQLInfo( 'S' ) = 1 THEN CALL rxOut 8, rxMessage( 'S' )
LEAVE
END
END
ELSE DO
CALL rxOut 2, 'Quering database... Please wait!'
IF rxSQLCommand( "DUP", "SELECT crc32, name FROM file WHERE computer = '"||c_name||"' GROUP BY crc32, name HAVING ( COUNT( crc32 ) > 1 ) ORDER BY crc32" ) = 1 THEN
DO c = 1 TO DUP.crc32.0
CALL rxProgress 4, c / DUP.crc32.0
IF rxSQLCommand( "DUPLICATES", "SELECT DISTINCT * FROM file WHERE crc32 = '"||dup.crc32.c||"'" ) THEN
DO
IF Write2CSV( c = 1 ) THEN ITERATE d
END
ELSE DO
IF rxSQLInfo( 'M' ) = 1 THEN CALL rxOut 6, rxMessage( 'M' )
IF rxSQLInfo( 'E' ) = 1 THEN CALL rxOut 7, rxMessage( 'E' )
IF rxSQLInfo( 'S' ) = 1 THEN CALL rxOut 8, rxMessage( 'S' )
END
END
ELSE DO
CALL LINEOUT 'STDERR', 'No data could be retrived, try manually instead!'
IF rxSQLInfo( 'M' ) = 1 THEN CALL rxOut 6, rxMessage( 'M' )
IF rxSQLInfo( 'E' ) = 1 THEN CALL rxOut 7, rxMessage( 'E' )
IF rxSQLInfo( 'S' ) = 1 THEN CALL rxOut 8, rxMessage( 'S' )
END
END
END
CALL STREAM out_file, 'C', 'CLOSE'
END
RETURN 0
Write2CSV:
CALL rxOut 2, 'Writing duplicate entries to' out_file
col_no = DUPLICATES.0
col_name = DUPLICATES.1
row_no = VALUE( 'DUPLICATES.'col_name'.0' )
IF ARG(1) = 1 & row_no > 1 THEN DO
col_val = ''
DO col_num = 1 TO DUPLICATES.0
col_val = col_val||'"'||VALUE( 'DUPLICATES.'col_num )||'",'
END
CALL LINEOUT out_file, STRIP( col_val, 'T', ',' )
END
IF DATATYPE( row_no, 'W' ) THEN
DO
IF row_no > 1 THEN
DO row_num = 1 TO row_no
CALL rxProgress 3, row_num / row_no
row_val = ''
DO col_num = 1 TO DUPLICATES.0
col_name = DUPLICATES.col_num
row_val = row_val||'"'||STRIP( VALUE( 'DUPLICATES.'col_name'.'row_num ) )||'",'
END
CALL LINEOUT out_file, STRIP( row_val, 'T', ',' )
END
ELSE RETURN 1
END
ELSE CALL rxOut 6, 'No data saved:' row_no
DROP DUPLICATES.
RETURN 0
I wish you all a
Happy New Year
//Jan-Erik