• Welcome to OS2World OLD-STATIC-BACKUP Forum.
 

News:

This is an old OS2World backup forum for reference only. IT IS READ ONLY!!!

If you need help with OS/2 - eComStation visit http://www.os2world.com/forum

Main Menu

Script->Scanimage->GOCR->rexxqueue->Script

Started by jep, 2008.10.22, 15:32:24

Previous topic - Next topic

jep

Marked as: Normal
Hello,

I've just recently bought a USB scanner (Canon Canoscan LiDE 25) and tried it out with SANE and TAME. I later decided to try to write a script to scan documents and invoices, bills and interpret the images to text and present that info automatically.

To scan images you need SANE to try this out
Paul Smedely has updated SANE to 1.1.0 and the package also include TAME
http://www.smedley.info/os2ports/index.php?page=sane-for-os-2

(A version of TAME and SANE is also included with eComStation, you just need to select it during installation)

To interpret the image into text you also need GOCR (JOCR) for OS/2-eComStation
Franz Bakan compiled GOCR 0.45 the other day as it seem :o
http://www.fbakan.de/gocr-os2.htm

OBS!! *** VARNING!! *** OBS!!
This code is provided "AS IS" without any warranty of any kind that it may work and/or not cause damage to your hardware (e.g. scanner).

I've tested this script repeatedly on my computer, but the documentation for SANE mention that you should take care not to use the wrong settings that the scanner can't handle as it then may do something illegal and thus cause damage. Please, read the manual for SANE to check that the parameters used in this script is ok with your setup. You may also want to adjust the params, but any usage of the script that result in fault/damage is your own fault.

Pay special attention to the variable for scanner resolution ( cfg.scanarea.1.dpi, cfg.scanarea.2.dpi etc. ) and that your scanner can handle it.
OBS!! *** VARNING!! *** OBS!!


This script will teach you a bit how to use one application to do one task, then let another take care of the result and process it further and send the text info to the rexx queue so the script can retrieve the result.

The script has a variable (cfg.debug) set to 1, if you change it to 0 you'll enable the usage of pipes described above, while current setting cause it to send each result to a file that the next part has to read from file and process. You can view the output image and text file between each stage if you add statements that halt execution (PARSE PULL) or (CALL TRACE '?I') to the script at appropriate positions.

The example code below contain various paths for the script to take depending on what it can find during the first scan. Since it's almost the end of this month and I have some bills to pay, that's what I've used for reference. The code has been targeted toward how bills look like in Sweden, so it probably won't work for you without modification.

The first scan area is positioned at the bottom and use the form:
# OCR_identification_number # amount to pay check_number > reciver_acount#payment_service#

and it's important that the OCR engine can determine the number representing the payment_service as the rest of the scan areas are positioned differently depending on what it find there.

Note that the coordinate values has been set according to SANEs documentation and is by default in millimeters. (25.4 millimeters = 1 inch)
Please observe that the assumed paper size is A4 (210 x 297 mm) and the maximum numbers you may therefore use are
cfg.scanarea.1.l + cfg.scanarea.1.x  must be less or equal to 210
(Where "l" mean distance from left side of paper and "x" width of scan area )

cfg.scanarea.1.t + cfg.scanarea.1.y  must be less or equal to 297
(Where "t" mean distance from top of paper and "y" height of scan area )
etc.


/* Pipe output from ScanImage->GOCR->rexxqueue and into script for further processing */
'@echo off'

/* Clear session queue in case it contain "garbage" */
'@rxqueue SESSION /CLEAR'

/* Try to determine the scanner, send output to the app rexxqueue, that place it in the rexx queue */
'@scanimage -L | rxqueue SESSION /FIFO'
/* Read the first line from the rexx queue */
parse value LINEIN( 'QUEUE:' ) with 'device `'cfg.device"' is a" cfg.scanner.manufacturer cfg.scanner.name cfg.scanner.model cfg.scanner.type 'scanner'
'@rxqueue SESSION /CLEAR'

if length( cfg.device ) = 0 then /* Couldn't determine the settings we need */
  if ARG() = 0 then /* And no parameter on the command line? */
    return Usage() /* Tell the user what parameter he/she should provide on the command line */
  else
    cfg.device = ARG(1) /* e.g. plustek:usbcalls:2 */

/* Cause the function "GetTextWithOCR" to call each application separately and not use pipes */
cfg.debug = 1

/* Coordinate ranges for consecutive scans */
cfg.scanarea.1.l = 5
cfg.scanarea.1.t = 271
cfg.scanarea.1.x = 200
cfg.scanarea.1.y = 15
cfg.scanarea.1.dpi = 600 /* scan resolution, different values set below! */
cfg.scanarea.1.filter = XRANGE( 0, 9 )||"#>" /* Limit the characters it may determine */
cfg.scanarea.2.l = 105
cfg.scanarea.2.t = 208
cfg.scanarea.2.x = 35
cfg.scanarea.2.y = 5
cfg.scanarea.2.dpi = 800
cfg.scanarea.2.filter = XRANGE(A, Z)||"ÅÄÖ"||XRANGE(a, z)||"åäö"||XRANGE( 0, 9 )||"-"
cfg.scanarea.3.l = 105
cfg.scanarea.3.t = 217
cfg.scanarea.3.x = 55
cfg.scanarea.3.y = 10
cfg.scanarea.3.dpi = 800
cfg.scanarea.3.filter = XRANGE(A, Z)||"ÅÄÖ"||XRANGE(a, z)||"åäö"||XRANGE( 0, 9 )||"-"
cfg.scanarea.4.l = 111
cfg.scanarea.4.t = 255
cfg.scanarea.4.x = 90
cfg.scanarea.4.y = 6
cfg.scanarea.4.dpi = 800
cfg.scanarea.4.filter = XRANGE(A, Z)||"ÅÄÖ"||XRANGE(a, z)||"åäö"||XRANGE( 0, 9 )||"-"
cfg.scanarea.5.l = 10
cfg.scanarea.5.t = 227
cfg.scanarea.5.x = 180
cfg.scanarea.5.y = 25
cfg.scanarea.5.dpi = 800
cfg.scanarea.5.filter = XRANGE(A, Z)||"ÅÄÖ"||XRANGE(a, z)||"åäö"||XRANGE( 0, 9 )||"-"

retval = GetTextWithOCR(1)
PARSE VALUE retval WITH '#'OCRNumber'#'Amount'#'PaymentMethod'#'
PaymentMethod = space( PaymentMethod, 0 )

PARSE VALUE Amount WITH AmountMain AmountPart CheckNum '>' RecieverAccountNo
RecieverAccountNo = space( RecieverAccountNo, 0 )

service = ', account' strip( RecieverAccountNo ) 'the amount' strip( AmountMain ) strip( AmountPart ) 'identified by OCR No.' strip( OCRNumber )

if PaymentMethod = 14 then /* Identification number for services in Sweden, adjust to your preference if applicable */
do
  PARSE VALUE GetTextWithOCR(2) WITH AccountNo
  PARSE VALUE GetTextWithOCR(3) WITH Name
  retval = translate( GetTextWithOCR(5) )
  if pos( 'TILLHANDA', retval ) > 0 then
    PARSE VALUE retval WITH .'TILLHANDA' date_of_payment ./* Swedish */
  else
    PARSE VALUE retval WITH .'FÖRFALLODAG' date_of_payment ./* Swedish */
  say 'Postal service'||service
  say 'Reciever:' Name
  say 'No later than:' strip( date_of_payment )
end
else if PaymentMethod = 41 then
do
  PARSE VALUE GetTextWithOCR(4) WITH AccountNo Name
  retval = translate( GetTextWithOCR(5) )
  if pos( 'TILLHANDA', retval ) > 0 then
    PARSE VALUE retval WITH .'TILLHANDA' date_of_payment ./* Swedish */
  else
    PARSE VALUE retval WITH .'FÖRFALLODAG' date_of_payment ./* Swedish */
  say 'Bank service'||service
  say 'Reciever:' Name
  say 'No later than:' strip( date_of_payment )
end
else
  say "Unknown payment method, please check the scanner settings and your bill!"
Return 0

GetTextWithOCR: procedure expose cfg.
  counter = ARG(1) /* variable to point to the right coordinates */
  if cfg.debug then
  do
/* Send image to out.pnm */
    '@scanimage -d '||cfg.device||' --format=pnm --mode Gray -l '||cfg.scanarea.counter.l||' -t '||cfg.scanarea.counter.t||' -x '||cfg.scanarea.counter.x||' -y '||cfg.scanarea.counter.y||' --resolution '||cfg.scanarea.counter.dpi||' > out.pnm'
/* Let GOCR read the image and write out.txt that contain the text found */
    '@bin\gocr -C "'||cfg.scanarea.counter.filter||'" -i out.pnm -o out.txt'
/* Read the text file into a variable */
    filesize = STREAM( 'out.txt', 'C', 'QUERY SIZE' )
    if filesize > 0 then
      retval = CHARIN( 'out.txt', 1, filesize )
    else retval = ''
    call STREAM 'out.txt', 'C', 'CLOSE'
  end
  else
  do
  /* Scan the area of the image with the info we want to read */
  /* pipe the image data to GOCR to interpret it and pipe the output text to the current rexx queue */
    '@scanimage -d '||cfg.device||' --format=pnm --mode Gray -l '||cfg.scanarea.counter.l||' -t '||cfg.scanarea.counter.t||' -x '||cfg.scanarea.counter.x||' -y '||cfg.scanarea.counter.y||' --resolution '||cfg.scanare.counter.dpi||'| bin\gocr "'||cfg.scanarea.counter.filter||'" - | rxqueue SESSION /FIFO'
/* Read from queue */
    retval = LINEIN( 'QUEUE:' )
/* Clear queue just in case */
    '@rxqueue SESSION /CLEAR'
  end
Return retval /* Return the text found (if any) */

/* Show how to use the script */
Usage:
PARSE SOURCE . . fname
say FILESPEC( 'N', fname )||' name_of_device'
say
say 'Example: '||FILESPEC( 'N', fname )||' plustek:usbcalls:2'
Return 0

ddan

Jep, this intrigues me, but I'm not up to speed on the items here.

Would you field a couple of questions? ── Or does that make three, and THIS
four? ... Must. break. out. of. infinite. loop... Unnnh!

Anyhoo, 1st: any chance that SANE works with parallel port attached scanners?
Have a nice one, SMALL, which has a Win31 driver and worked wonderfully under
OS/2 2.1 GA WINOS2 ── at least ten times as fast on exact same hardware as in
DOS/Win31, one of the major reasons that began my "bias" ── however, modern
systems turn out to be TOO fast ── apparently because feeble Win31 doesn't
have a real time delay ── and the between sheets wait is almost non-existent,
much too fast for any reasonable use, and of course no way to change it.

I scanned in hundreds of pages with that little 256-gray level scanner, still
worked fine six months ago, and had a project of scanning in old novels that
are falling apart, but it's unusable with present systems, another LOST bit of
older and simpler technology that actually worked better at OCR than current
flat-bed 2400dpi 48-bit color scanner will, easier, and at lower cost.

2nd, ignoring the above, have an HP 5 or something SCSI scanner or a couple of
USB models; question is, how well does the OCR program work? I've tried one of
the USB under PCLinux, and its error rate can literally be in excess of the
number of scanned characters, better off simply typing it, where with the
above error rate was 1-2% on clear pages, with highly regular mistakes that
search and replace took care of. OCR for current project must deal with
yellowed, sometimes stained, small pages of a paperback, so has to be good.

RobertM

ddan, if you find the OCR capabilities of the SANE/TAME package insufficient (or jep indicates they are), you may wish to consider Tesseract, which I have had much success with. Tesseract seems very good (and very reliable - over 95% on my tests) when a page is scanned. Not quite as good on computer generated ones - unless they are rotated 1 degree (or a little more). Hence, scanned pages seem fine, because they are not perfectly scanned at 0% rotation on virtually any scanner. At 0% rotation (or a perfect scan), it seems to mistake a couple letter/number pairs (zero and O, one and lower case L). At one percent, I received no mistakes on multiple test scans.

-Robert


|
|
Kirk's 5 Year Mission Continues at:
Star Trek New Voyages
|
|


ddan

Okay, thanks, will look into that. But have the feeling I'm going to be digging out an antique, because the little Tamarack scanner really is that much easier than a flat-bed. Once going, you just drop sheets in. Of course, have to destroy the book binding to do so, but that's not a big objection with most of these crumbling antique paperbacks. Only thing better that I know of is a sheet feeder for a flat-bed, but that's out of my price range, and still requires destroying the binding.

jep

Quote from: ddan on 2008.10.25, 03:40:04
Jep, this intrigues me, but I'm not up to speed on the items here.

Would you field a couple of questions? ── Or does that make three, and THIS
four? ... Must. break. out. of. infinite. loop... Unnnh!
do forever
say ':D'
end
Quote from: ddan on 2008.10.25, 03:40:04
Anyhoo, 1st: any chance that SANE works with parallel port attached scanners?

Yes, but olny a limited number of them and not the one I have as it's originally a scsi scanner with only a parallell port. You have to go to the SANE site and look, or better you check the list at Paul Smedley's site.

The LiDE 25 just costed me about 99US$ (~66€) and worked right away, so it was an easy decision to make as it was listed on the SANE site.

Quote from: ddan on 2008.10.25, 03:40:04
Have a nice one, SMALL, which has a Win31 driver and worked wonderfully under
OS/2 2.1 GA WINOS2 ── at least ten times as fast on exact same hardware as in
DOS/Win31, one of the major reasons that began my "bias" ── however, modern
systems turn out to be TOO fast ── apparently because feeble Win31 doesn't
have a real time delay ── and the between sheets wait is almost non-existent,
much too fast for any reasonable use, and of course no way to change it.

Hmmm, perhaps one should add:
call SysDelay 10000
or
say 'Please replace sheet with the next one and press Enter. Any other key to end scan.'
retval = c2d( SysGetKey() )
if retval = ...
Quote from: ddan on 2008.10.25, 03:40:04
I scanned in hundreds of pages with that little 256-gray level scanner, still
worked fine six months ago, and had a project of scanning in old novels that
are falling apart, but it's unusable with present systems, another LOST bit of
older and simpler technology that actually worked better at OCR than current
flat-bed 2400dpi 48-bit color scanner will, easier, and at lower cost.
It's perhaps possible to tweak something to make it work in OS/2-eCS as well. Doesn't CFM TWAIN nor STi TWAIN work for you?

OK, but what's the output from that software? An Image? Then it's possible to use whatever OCR software that is available to try to extract the text. I've read that STi has got some OCR software that one can buy, but i found it quite expensive.

GOCR 0.43 that I used (earlier version than the one compiled the other day) didn't do a great job even though all text was printed with even spacing, aligned and with clear and distinct characters. I'll try 0.45 when I get the time.

Quote from: ddan on 2008.10.25, 03:40:04
2nd, ignoring the above, have an HP 5 or something SCSI scanner or a couple of
USB models; question is, how well does the OCR program work? I've tried one of

I believe it's not really up to whether you have a scsi, usb or parallell port scanner, it depend more on how much noice it recognize/add to the scanned image, the resolution used and how well the ocr software can cope with that.

Quote from: ddan on 2008.10.25, 03:40:04
the USB under PCLinux, and its error rate can literally be in excess of the
number of scanned characters, better off simply typing it, where with the
above error rate was 1-2% on clear pages, with highly regular mistakes that
search and replace took care of. OCR for current project must deal with
yellowed, sometimes stained, small pages of a paperback, so has to be good.

Yes, I noticed that, as the error rate for the 2:nd, 3:rd etc. scan was very high.

You can combine the script with gbm that has excellent rexx support and preprocess the image a bit futher to change problematic areas such as crop edges, reduce noice and colors etc.

GOCR 0.43 doesn't handle lines, differently aligned (vertical) text, tables etc.
One can however tweak the behavior somewhat with params to see how it tries to detect text and where it fail. I've managed to turn on output to image of detected text in the past, and one can understand that it has quite a way to go yet, but as always, it's getting better.

There are also other ocr software out there, some does a great job, and would be really nice to have in OS/2-eCS as well.

jep

Quote from: RobertM on 2008.10.25, 05:34:14
ddan, if you find the OCR capabilities of the SANE/TAME package insufficient (or jep indicates they are), you may wish to consider Tesseract, which I have had much success with. Tesseract seems very good (and very reliable - over 95% on my tests) when a page is scanned. Not quite as good on computer generated ones - unless they are rotated 1 degree (or a little more). Hence, scanned pages seem fine, because they are not perfectly scanned at 0% rotation on virtually any scanner. At 0% rotation (or a perfect scan), it seems to mistake a couple letter/number pairs (zero and O, one and lower case L). At one percent, I received no mistakes on multiple test scans.

-Robert


I also tried Tesseract, but not that extensively... but it didn't like the "perfect" scans I have, but the output may have been in Unicode, UTF-8 or something when I come to think of it ... because the text didn't make any sence.

Hmm, better look into it.  :)

ddan

I've now read through most of Bakan and Smedley: apparently no parallel port scanners are known to work under OS/2, and particularly not the model I have.

Problem with OCR of several hundred pages isn't scanning or recognition as such, it's the mechanics of doing it. The little Tamarack starts when a sheet is inserted, rolls them through, spits them out when done, actually hanging on just a bit so it doesn't drop, and has a delay so you can grab that sheet and run its other side through for continuous text output. The Win31 software with it has settings for multiple sheets, when no more are inserted, it automatically goes through OCR, drops that into notepad where I titled the section by page numbers for later editing. Problem under W4 with an 1800 Duron is that between-sheets delay is too short, just barely long enough for me to reverse it, not long enough to align the sheet straight and trip the switch reliably to continue scanning before starts OCR on what should be ten or so pages. Without the least exaggeration, that system is ten times faster than any flat-bed scanner without a sheet feeder can be, and was what made scanning in several books practical. In OS/2, but not with "real" Win31, I could also do some editing while it was doing OCR. Its recognition rate was phenomenal even with a mix of French accent marks and italics, much better than any I later got with scanners having supposedly better resolution.

So, that project definitely looks as though I need to dig out an old, slow, W3 box.

Thanks for the tips, though.

jep

Hello ddan,

is it possible to run that ocr app on its own, e.g. start it separately and select the image by hand?

ddan

It's still the mechanics that are the drawback: having to place a sheet, drop the lid GENTLY so that the sheet isn't dislodged, manually start a scan, remove the sheet, usually having to pry it up with nails, flip it or insert another, are operations that take TOO much of MY time, while I'll be able to stand a slow CPU grinding away by itself through the actual OCR on batches of ten pages. A flat-bed scanner always requires fiddling with alignment, especially with the old, wrinkled, torn pages that I most frequently scan.

Don't see any way around the mechanics. If you haven't used that type of sheet scanner, you may not see my point.

jep

If you can run them separately, then it may be possible to do something about your problem.

Can the image scanner (Win31) software scan an image using command line parameters?
Run in seamless mode?
Can the ocr software run in seamless mode as well, if you point it to the right image?

Here's some kind of pseudo code to explain what I mean.

set the number_of_images to none
do forever
  scan one "image" and then exit the win31 app
  one more number_of_images
  pause here while you feed the next sheet/page etc.
  if not key is enter leave
end
find the number_of_images scanned in the file system
set the counter, of images looked at, to none
while counter is less than number_of_images
  ocr next image, output text to file
  add the page number, chapter etc.
end


Have you asked them if they can give you some of the source code found in the Win31 driver/software instead? Would be great to "borrow" knowledge. :-D

ddan

Nope, command line is out for this, complete Win31 app.

It's no major trauma to set up another system just for this, has advantages, even, so that's what I'll do. Thanks.