Marked as: Advanced
Hello,
I've been working an a GUI tool so tesseract can handle more languages, but now I need some help from you to complete it. It's written in/for DrDialog, though the actual rexx code reside in a plain text file (eFTBE.rex), just as one write for command line mode and should therefore not cause you troubles to read, just open it with EPM¹.
(http://www.xn--lrka-loa.com/rxImgSze/eTFBE.png)
Please log on to this forum to download the zip-file with the code and to see the image I've prepared how it look and behave in its current state. To try it out and see what each part does, download DrDialog from hobbes (http://hobbes.nmsu.edu/h-search.php?key=drdialog&pushbutton=Search) for free with additional fixes to look at the graphical stuff.
You may also look at Douglas Clark's excellent articles about DrDialog at OS/2 eZine (http://www.os2ezine.com/20010216/drdialog.html) on how to install it, what each part of it does and how to use it.
What I want you to do is to add another function, select one below and try, see more info about it at the Tesseract training page (http://code.google.com/p/tesseract-ocr/wiki/TrainingTesseract).
- Fix current limitations and problems
- Generate training images
- Make Box Files (*Done, may need patch/fix)
- Bootstrapping a new character set
- Run Tesseract for Training
- Clustering
- Compute the Character Set
- Dictionary Data
- The last file (intrinsic ambiguity between characters)
¹) You can turn on syntax highlighting as well in EPM so you can read the code easier.
a) Open EPM
b) Right click next to the buttons on a free area
c) Select "Buffet..." from the menu
d) Click on the narrow button to the right to scroll to the right
e) Look for a button with the text KwdHilit, drag it to the toolbar below
f) Click on the button OK to close the window
g) A message tell you to save the changes to the toolbar or you'll loose it the next time you open EPM.
h) Therefore, right click next to the buttons again and select "Save as..." this time
i) Type "Highlight" (without the quotes) in input field that appear in a small window.
j) Make this toolbar the one to always use by selecting "Options->Preferences->Settings..." from the menu
k) Click on the last tab (Toolbar) in the window that appear
l) Select the toolbar you just saved from the dropdown menu
m) Click on the button "Make current" and ensure that "Display toolbar" is checked.
n) Click on the tab labeled "Colors", as wee need to make an adjustment.
o) Select "Text" in the list and background on the radio buttons below.
p) Select "Light_grey" in the list to the right for color selection to ensure that known Rexxutil functions are viewable too.
q) Close and Select "Options->Save options" from the menu.
r) You now have another button, click on it to get syntax highlighting.
Note 1: You have to open a file first before you can use syntax highlighting.
Note 2: Syntax higlighting only work on known file types (.c, .cpp, .rex, .cmd and several more)
Note 3: Rexxutil functions appear in white, so they disappear if we don't change the background color to light grey.
//Jan-Erik
Hi Jep!
Great idea. I'm working with tesseract for OCR on a WIKI-SOURCE project. Therefore I need the old-german language-support (fractur). But it has some problems with "A" and other characters. If your program could solve these problems it would be great.
I build the WPI and got therefor from Paul the updated 2.03-binaries. There were some problems on compiling the training-programs. Hope this is no problem for your project. Maybe new GCC can fix the porblems?
I don not know, if I find time for test it - but I can write a WPI for it if it's ready.
Here is something JEP may be interested in also.
http://www.os2world.com/cdwriting/drdialog/files/drdialog-controls-0_1_5.zip
This is not found on Hobbes.
David
Hello Batchheizer,
Glad to hear that you're working on tesseract, perhaps we could get some inside info or rexx support dll?!
Are you sure you tried an image saved in Black and White only, not gray scale? I made that mistake the first 20 times to take an image that looked all right to me that contained anti aliased text. I was ashamed for my mistake when I found out about what I did wrong.
Anyway, I've used eTFBE to test a text I wrote for a paper magazine that we distribute to members (not OS/2-eCS related tough) in the region where I live here in Sweden. I may add that Swedish contain the additional letters åäö (ÅÄÖ) over the English ones, something tesseract doesn't recognize unless one teach it to first. That were eTFBE come in, as one should use it to create the support configuration for each language that tell it that e.g. the "dirt" over "A", "O" etc. should be considered as well.
eTFBE can do a few more tricks to help you as user.
Type space to make the Editor add a space in the view, nothing will be added to the box file though, but you can save it as text (ascii or utf8) and then reuse it in eTFBE.
If you store the original text in a text file and open it in the editor, it will re flow and analyze the text automatically (until it find a problem). The unicode support contain a function that can disassemble characters for comparison. It is very useful in the import/open project and text routines as it then can adjust characters that tesseract interpret as A even though they're Ä (remember the "dirt" above), but it will stop and report on problems like "rn" that tesseract may interpret as the single character "m". Note that eTFBE does in no way interpret the image itself, but rather (re)use knowledge presented in a text file that should represent the truth. ("baseline"/answer).
eTFBE won't be affected by problems in tesseract and the supporting executables, unless they provide strange output. Another thing is that it hasn't progressed that long that it use so many of those, but it will hopefully do soon, with help from you, the reader.
When you start eTFBE it will tell you what support functions it can't load/find, click on each link to go to each developers page and download each function.
Download Tesseract from: hobbes (http://hobbes.nmsu.edu/h-search.php?key=tesseract&pushbutton=Search)
Download DrDlgCtrl from: Chris Wolgemuth's page (http://www.os2world.com/cdwriting/drdialog/drdialogmain.htm)
Download REXXULS from: Alex Taylor's page (http://users.socis.ca/~ataylo00/os2/toolkits/uls/)
Download GBM from: Heiko Nitzsche's page (http://heikon.home.tlink.de/)
Download rxImgSze from: Jan-Erik Lärka's page (http://www.xn--lrka-loa.com/rxImgSze/)
//Jan-Erik
Hello DavidG,
Yes, it's currently used for the image viewer and tooltip part in the project. I've got an issue with it though that has to be fixed, e.g. that RexxFuncQuery doesn't think that DRCtrlVersion has been loaded so it show the download link/page all of the time instead of version info.
Additional info:
To create a function for the second point in the list "Generate training images" we also need a rexx support .dll that can take text and output it as a .tiff-image in B&W.
//Jan-Erik
Hi Jep!
Of cause I run only B/W-Tiffs with tesseract.
I tried to run your rexx-script - but I have no time to learn about Dr.Dialog and so on. If you build an executable I could test it. Write me a message for this. I hope you find hints to get this done.
Is the purpose of the project to build and improve language-packs? If so, is the training software on google-code better for this purpose? Can other platforms read and use the new/improved packs (created with your software)?
Hej jep,
Quote from: jep on 2009.01.18, 18:38:13
To create a function for the second point in the list "Generate training images" we also need a rexx support .dll that can take text and output it as a .tiff-image in B&W.
not sure but I guess gbm can do that...(?)
And if not... how about ghostscript in the background?
And, ehhh... sorry but regarding DrDialog, if someone wants to learn about it, there's also a quite extensive series of articles about it on the VOICE newsletter (both English and also German).
Cheers,
Thomas
Hello Batchheizer,
Quote from: Batchheizer on 2009.01.18, 22:35:51
Hi Jep!
Of cause I run only B/W-Tiffs with tesseract.
I tried to run your rexx-script - but I have no time to learn about Dr.Dialog and so on. If you build an executable I could test it. Write me a message for this. I hope you find hints to get this done.
If you've got DrDialog installed, then you just have to open it and run it. It'll run just as any executable except for the debug window. Then you cn drag & drop it on the Rex2Exe Icon and it'll give you what you want.
At the moment it output text files (ascii or utf8-encoded) and the box file mentioned in the list as *Done.
You can teach it/improve the .box file with it.
Text to image: Not yet, I want you readers to code a section for that.
Quote
Is the purpose of the project to build and improve language-packs? If so, is the training software on google-code better for this purpose? Can other platforms read and use the new/improved packs (created with your software)?
Yes, it is intended to create language packs.
Do you mean the training software included with tesseract (text mode apps)?
If so, that's the ones it'll use.
If not, you mean the GUI software other people has made in some compiled code that one can get from google?
Well, its meant to be the equivalent for OS/2-eCS, but not do things forced on us by how other OSes does it. If the google-code (GUI) is better? Probably at the moment, if it can do more of the things mentioned on the traning tesseract page, yes. But to what I've read, they're not that far ahead.
The purpose is to create files that follow the standard and not produce any specific to this software.
//Jan-Erik
Hello warpcafe,
Quote from: warpcafe on 2009.01.19, 00:25:05
not sure but I guess gbm can do that...(?)
And if not... how about ghostscript in the background?
Heiko may have the answer if GBM can be used, but I don't think so in its current form, it's aimed at other things. I know there are libraries that can do it, but perhaps not to TIFF directly.
Text -> Ghostscript -> TIFF
Sounds interesting, but can you or someone else give a hint how to do it? I don't feel I have the knowledge.
Quote
And, ehhh... sorry but regarding DrDialog, if someone wants to learn about it, there's also a quite extensive series of articles about it on the VOICE newsletter (both English and also German).
Ohhh, yes, quite right, your series cover alot of what one can do with DrDialog. I waited and read through them all when they came out, great reading as well.
Links to each of the OS2VOICE Newsletters:
DrDialog, or: How I learned to stop worrying and love REXX - Part 1 (http://www.os2voice.org/VNL/past_issues/VNL0902H/vnewsf6.htm)
DrDialog, or: How I learned to stop worrying and love REXX - Part 2 (http://www.os2voice.org/VNL/past_issues/VNL1002H/vnewsf2.htm)
DrDialog, or: How I learned to stop worrying and love REXX - Part 3 (http://www.os2voice.org/VNL/past_issues/VNL1102H/vnewsf2.htm)
DrDialog, or: How I learned to stop worrying and love REXX - Part 4 (http://www.os2voice.org/VNL/past_issues/VNL1202H/vnewsf2.htm)
DrDialog, or: How I learned to stop worrying and love REXX - Part 5 (http://www.os2voice.org/VNL/past_issues/VNL0103H/vnewsf2.htm)
DrDialog, or: How I learned to stop worrying and love REXX - Part 6 (http://www.os2voice.org/VNL/past_issues/VNL0303H/vnewsf2.htm)
DrDialog, or: How I learned to stop worrying and love REXX - Part 7 (http://www.os2voice.org/VNL/past_issues/VNL0403H/vnewsf2.htm)
DrDialog, or: How I learned to stop worrying and love REXX - Part 8 (http://www.os2voice.org/VNL/past_issues/VNL0503H/vnewsf2.htm)
DrDialog, or: How I learned to stop worrying and love REXX - Part 9 (http://www.os2voice.org/VNL/past_issues/VNL0703H/vnewsf2.htm)
DrDialog, or: How I learned to stop worrying and love REXX - Part 10 (http://www.os2voice.org/VNL/past_issues/VNL0903H/vnewsf2.htm)
DrDialog, or: How I learned to stop worrying and love REXX - Part 11 (http://www.os2voice.org/VNL/past_issues/VNL0104H/vnewsf2.htm)
DrDialog, or: How I learned to stop worrying and love REXX - Part 12 (http://www.os2voice.org/VNL/past_issues/VNL0404H/vnewsf2.htm)
DrDialog, or: How I learned to stop worrying and love REXX - Part 13 (http://www.os2voice.org/VNL/past_issues/VNL0604H/vnewsf2.htm)
DrDialog, or: How I learned to stop worrying and love REXX - Part 14 (http://www.os2voice.org/VNL/past_issues/VNL0605H/vnewsf5.htm)
One thing I thnik they didn't cover, not related either, but useful non the less, is how to extract the images built in to drrexx.exe, alter them to something modern looking and use rc.exe to put them back into drrexx.exe. That way one doesn't have to use extra image dll's but just call the images inside the executable. Something like the example on how to tweak the looks of OS2Chess. I've tried it, and the result looked... awful... because the images I used, didn't look that great. ;-)
//Jan-Erik
Hi Jan-Erik
I was playing around and made a few bitmaps for DrDialog. If you like, I would try to make more when I have time. I have attached them below.
David
Hello David,
I like 'em, they look real good, great work David! :)
Would be nice if we could get a unified look on all buttons in as many apps as possible. May we use your work?
One button I have a concern about is the group box though, it look like it display a group of people on an image/picture on a box (just as "group box" sounds like), though one might want to let it portray something like a "named container with a border" or something closer to what the developer will see/get as he/she push the button. The same goes for many development tools, I guess. End user apps are different.
Can you disassemble drrexx.exe, drdialog.exe some time and suggest more improvements just as these?
Perhaps create some equivalent/replacement to drdialog.dll as well?
[hr width="70%"]
I'll look into the issue where TFBE doesn't load the image when one open an existing project.
//Jan-Erik
A small instruction how to use the current version of the Tesseract Font Box Editor
You need some additional libraries to do various things, e.g. convert images, calculate sizes and color depth, convert between different codepages and allow you to view then image. See the About screen for links, click on each link to visit those pages and download the libraries.
(http://www.xn--lrka-loa.com/rxImgSze/Deutsch.jpg)
1. Create a black and white tiff image with some text you want to test
2. Open eTFBE
3. Select and open the image
a) Menu: File -> Open -> B&W TIFF...
b) Select: Image
4. Create a box file
a) Menu: OCR -> Box
5. Save your work as a Project
a) Menu: File -> Save As -> Project
b) State a name of the Project, e.g. "Deutsch"
6. Edit each character in the Editor (or Selection window)
a) Keyboard: a, b, c ...
* It change the selected character, jump to the next and wait for the next key stroke
b) Keyboard: " "
* (space) add a space to make the text more readable, but doesn't save to the box-file
c) Keyboard: ½
* (half sign) split a character in 2
d) Keyboard: Arrow
* (Up/Down/Left/Right) to move between characters and rows
e) Keyboard: Alt + number
* ONLY WORK PARTIALLY, PLEASE ADVICE FIX
7. Remember to adjust the selection for each character that you join and split in the Selection window
a) Click the button OK to store changes for that character
8. Save your work
a) Menu: File -> Save
9. Close Project to work on something else
a) Menu: File -> Close
10. Open (another) Project
a) Menu: File -> Open -> Project
11. Select folder with the name of the Project that contain the project files
12. The image will load, the last saved state will be read, restored and presented, with window sizes restored.
________
Need 4 speed, or just in a hurry?
eTFBE can analyse the text and adjust it, report ambiguities and let you type only letters that doesn't look right.
5.1) Create the text file (ASCII or UTF-8) with the original text and layout (newline for each row on the image).
5.2) Open the prepared text file
a) Menu: File -> Open -> ASCII Text File...
or
b) Menu: File -> Open -> UTF-8 Text File...
7.1) If it complain about a problematic character, adjust it as in 6) and 7), then repeat step 5), 5.1), 5.2) etc. until you're done.
_________
Please help me make it better, have a look at it and try to figure out if things aren't working, or behave strange. Let's hear about it in the forum. There are some things that I can't do much about though (load the image viewer the second time one start it), but I appreciate if you have suggestions and/or code snippets to improve it.
Hi Jep,
Thanks for this new version. I'll try it in the next days. All above URLs in your postings are invalid - put I got the files. Except one: How can I get "Download rxImgSze from: Jan-Erik Lärka's page"?
Batchheizer
Hello Batchheizer,
Ohhps, sorry about that, thank you for notifying me of the problem. SMF apparently add extra http:// and doesn't like quote signs around links. :-(
I've now corrected all the links in this thread so that they work (Thomas articles about DrDialog at OS/2Voice and the links to required libraries from various sources).
The link to my own server (rxImgSze from: Jan-Erik Lärka's page (http://www.xn--lrka-loa.com/rxImgSze/)) should also work now :-)
You can also find rxImgSze in another topic here in the rexx forum regarding Resizing of Images and require you to have GBM installed to work.
//Jan-Erik
Hi all,
Quote from: jep on 2009.01.22, 09:19:18
Can you disassemble drrexx.exe, drdialog.exe some time and suggest more improvements just as these?
Perhaps create some equivalent/replacement to drdialog.dll as well?
Goodness! If someone is able to do that, can he/she please-please have a look at the famous "Warp4 style notebook" topic and let us know how to patch /reassemble DrDialog such that it is able to use the "warp" (more contemporary) style for notebook controls? :)
Cheers,
Thomas
Quote from: warpcafe on 2009.02.09, 11:18:53
Hi all,
----->8=======
=======8<-----
Goodness! If someone is able to do that, can he/she please-please have a look at the famous "Warp4 style notebook" topic and let us know how to patch /reassemble DrDialog such that it is able to use the "warp" (more contemporary) style for notebook controls? :)
Cheers,
Thomas
While it's easy to change all the built in images (and e.g. rearrange the debug window) inside the executables, it's more difficult to touch the code to set the right bit to make it turn on the Warp4 notebook style on in DrDialog. I tried as some discussion suggested and was able to get the Warp4 notebook look, but the tabs didn't work right (Very narrow, label not visible or floating around), nor did the tabs respond to input. :-<
Unfortunately, I guess that the notebook creation must reside far down, inside the code.
(http://www.xn--lrka-loa.com/rxImgSze/DrRexxAlt.jpg)
//Jan-Erik
Hi,
I've now updated eTFBE some more, as a matter of fact it seem to be able to produce all files required, except the one to create a dictionary! You just have to move the files to the right place in a subfolder to tesseract and rename them!
(The files created haven't been tested, but seem to be ok and should therefore work)
Have enabled output to ".tr", and the various file formats AND added code to save the ambiguities to file.
You'll find most options under the OCR menu when you load a project except the one to save Ambiguities, it remain under File -> Save as -> Ambiguities.
(http://www.xn--lrka-loa.com/rxImgSze/OutputFiles.png)
What remain to be done (with your help):
Allow one to select target language and use it in the project
Create image from text
Fix various isseus in the GUI
Handle output from some of the tesseract apps.
Add code to convert a text file with words to a dictionary
etc.
//Jan-Erik
DrDialog can create executables for you! (out of the DrDialog .res file)
Just drag & drop the .res file in the .zip-file to the program object REStoEXE in the DrDialog folder on you desktop and you're ready to go.
Hi jep,
Quote from: jep on 2009.02.19, 07:50:52
What remain to be done (with your help):
...
Create image from text
...
I haven't followed in detail... are you talking about creating a graphics image file (e.g. ".BMP") from a plain-text file (e.g. ".TXT") ? Or to put it that way: "Make a picture of a text page"? In that case -and with regards to DrDialog or rexx in general- I would recommend to use the rexxgd package. IIRC there are functions to create graphics (empty in your case) and insert text into them. All by calling rexx DLL procedures. ;)
Cheers,
Thomas
Hi warpcafe,
Quote from: warpcafe on 2009.02.19, 12:06:08
Hi jep,
I haven't followed in detail... are you talking about creating a graphics image file (e.g. ".BMP") from a plain-text file (e.g. ".TXT") ?
Yes, TXT (or UTF-8) -> BMP ( -> TIF, see below )
or use the editor to write the text down...
or take text from the clipboard...
and create the image from that
Quote from: warpcafe on 2009.02.19, 12:06:08
Or to put it that way: "Make a picture of a text page"?
Yes, exactly, why not use text we already have to create tons of training images!
Note: eTFBE can use the original text file for the training procedure as it depend on either human input or help from a text file.
Quote from: warpcafe on 2009.02.19, 12:06:08
In that case -and with regards to DrDialog or rexx in general- I would recommend to use the rexxgd package.
Yes, rexxgd (http://rexxgd.sourceforge.net/index.html) by Mark Hessling...
Quote from: warpcafe on 2009.02.19, 12:06:08
IIRC there are functions to create graphics (empty in your case) and insert text into them. All by calling rexx DLL procedures. ;)
Cheers,
Thomas
Can you show us how, Thomas?!
Just take into account the "fontmetrics"¹ calculations for each row.
Anyone else may try as well! Get rexxgd (http://sourceforge.net/projects/rexxgd)!
You can then convert it to TIF with rxImgSze/GBM combo with
call rxImageSize img_bmp_extension, img_tif_extensionafterward
I'll look at the dictionary in the mean time... I probably have to find a way to convert all rows to UTF-8 before feeding it to tesseract.
//Jan-Erik
_____________
¹) Length of the text for the specified font on the drawing surface.
Hi Jep,
thanks for your work on the Font Box Editor. But I did not get it started (neither from DrDialog nor the created Exe). I do not know the reasons (and will not discuss this issue). I would like to test the program and help to improve it. PLEASE distribute it as a package (WPI or ZIP) that does not need any other special software and runs out of the box.
Hi,
I'll package the files and post the zip tonight (CET) if not sooner.
//Jan-Erik
Hi,
please see attached -zip-file. It contain the files you need, except Tesseract and GBM that you should have on you computer already anyway.
Note!
eTFBE doesn't run a second time unless you restart your computer due to that it can't load the image control and the main window doesn't appear. I don't have a clue why, but someone else may know what cause the problem and how to solve it. If you know how to solve it, please take the time to reply here and tell me/us what to do to fix it.
Info:
1) Use the menu option "Close" to close the image and the editor before you open another image you plan to use tesseract with.
2) All images you open will be moved to the project you create at the beginning. (See previous instruction).
3) Save your work once in a while, especially if you've edited text and plan to open another file.
4) eTFBE remember and load the last image and box file you worked with whenver you open a project.
5) The function to Cluster files, list and use all .tr-files it can find in the project folder, the same goes for .box-files to create the character set.
//Jan-Erik
Hi Jep!
It runs! Loading takes a long time, but it seems to work. There is lot of harddisk-access on loading the program. Why? Does it search for something?
I will make some tests next week. Thanks for your work.
Hi Batchheizer,
Yes, it's a strategic choice I've made so you wouldn't have to configure it by hand. It tries to find what it need on it's own and report to you what may be missing or doesn't seem to work.
That's what I consider user friendly, the first time you run it, it searches for all files needed such as tesseract on your computer and that may take a moment. It store that info into the Extended Attributes (EA) of eTFBE.cfg (it doesn't write into the file) and start the application. (See more info at the bottom)
The next time you run eTFBE, it first tries to read the EA from eTFBE.cfg.
If it can find the info and the check that file exist succeeds then it doesn't have to search further (as the first time) and can continue with the next procedure right away. That way it'll load and run faster though be able to handle changes in your system.
So, if you move tesseract or delete eTFBE.cfg it'll take longer to start again as it search for files. Also note that information about position and sizes of the windows are stored in eTFBE.cfg (The project folder instead if you load a project). Please see the file eTFBE.REX and the startup routine (RxInit) for more information how I've designed it.
//Jan-Erik
========
Supporting functions that load right, appear with version info in the About box
Rename a .dll and try how eTFBE behave.
If it can't find tesseract (or any other part) a message box should appear that tell you to get the the missing function.
The About box should then appear and show you where to get the missing function/dll
It should even show you the latest known location where to get it and if you click the link your browser will load the page so you can read more about it and download what's missing.
Hello,
I've begun to write the code for word2dawg to convert a wordlist to a suitable format.
I've decided to add support for OpenOffice.org dictionaries since it support many languages and they're hopefully quite complete. It's quite easy to extract the info needed, clean the data (remove extra info from each word) and convert it to UTF8. The problem is that word2dawg doesn't run, not even with just a few words in the file, something that cause alot of problems. I've just downloaded the code and can see that it must be the initial loading (allocation, "emalloc") that doesn't handle it right.
Let's hope that Paul Smedley or someone else can come up with a solution to what might be the problem.
I'll add support for plain text and UTF8 files as well, but word2dawg has to be fixed before we can expect it to produce the right files.
//Jan-Erik
Quote from: jep on 2009.02.26, 13:48:21
Let's hope that Paul Smedley or someone else can come up with a solution to what might be the problem.
I sent you a new compiled version of word2dawg attached in a mail.
This version worked here (I couldn't check the output, but at least it doesn't crash anymore and it produces a file)
Did you miss my mail?
best regards
Franz
Hello Franz,
Quote from: fbakan on 2009.02.26, 19:52:26
I sent you a new compiled version of word2dawg attached in a mail.
This version worked here (I couldn't check the output, but at least it doesn't crash anymore and it produces a file)
Did you miss my mail?
best regards
Franz
Thank you for you help Franz, I really appreciate it. :)
I'll lookout for it in my inbox whenever it may arrive!
//Jan-Erik
One step futher, one step closer to completing this topic
Hi all,
I'm now making progress with the wordlist2dawg app that Franz wrote.
I need your help though to handle a couple of things, but you don't have to write any code, just come up with ideas.
Language Codes
I've got a file with the language codes that they want us to use for to represent each language, but what is the most efficient way to:
a) Load the text
b) Present the loooOOOoong list
c) Make it easy for the user to select the right code without scrolling toooOOOo much
Example: ISO-639-2 Codes (http://en.wikipedia.org/wiki/List_of_ISO_639-2_codes) to represent each langauge
The info below may require understanding for how rexx behave.
CodePages present within the spelling dictionaries
One ".aff" (inside the .oxt-zip-file) may contain a row like "SET IS0-8859-15", while the other contain "SET ISO-8859-2" and a third something completley different, but all has to be converted to UTF-8 to work with tesseract. It's "easy" to get that info from there, but then the problem begin.
What's the most efficient way to find the codepage to use for rxuls that each dictionary as each may contain different encodings?
I have a list with matching codepages, but how do one load it (a modified/adapted version to rexx) and get the right info/match from it with as little effort as possible?
Alex Taylors list of Codepages in OS/2-eCS (http://users.socis.ca/~ataylo00/os2/toolkits/uls/codepages.html)
//Jan-Erik
Hi Jan-Erik,
Quote from: jep on 2009.03.06, 14:11:46
..., but what is the most efficient way to:
a) Load the text
b) Present the loooOOOoong list
c) Make it easy for the user to select the right code without scrolling toooOOOo much
Well it depends - you want to go for alpha-3 or alpha-2?
With alpha-2 the list will be considerably smaller. Another way would be to select in two steps like country first, then language but that would need a list of combinations allowed (e.g. Switzerland -> ch/de/it...) also a deductive preselection based upon the "lang" setting could ease the selection for the user.
A loooong list is not a problem if you can incremetally "type" in it to set the cursor. OS/2 hasn't got that I'm afraid. ;) It also depends if that is a one-time, on-demand or regular (for each file) process... perhaps use a "wizard"-thing that does the trick in two or more steps. That's better than a single spot which is cluttered with controls and listboxes.
Quote from: jep on 2009.03.06, 14:11:46
I have a list with matching codepages, but how do one load it (a modified/adapted version to rexx) and get the right info/match from it with as little effort as possible?
I'm context-heavy enough I guess. Can you give an example with files (or records)? Perhaps then I can give a suggestion based upon what functional approach is best/fast. We can alos think about transforming the information into something more machine-suited to find better/faster algorithms...
You know with rexx... the string manipulation functions are awesome (also when combined with stems) and one can do nifty tricks with that as long as we know what we would need to achieve. ;)
Cheers,
Thomas
Hi,
Quote from: warpcafe on 2009.03.06, 17:29:10
========8<---------
--------->8========
Well it depends - you want to go for alpha-3 or alpha-2?
The tesseract documentation says one should use Alpha-3 so all will know what we mean then.
A code for Alpha-2 may also not be available for all languages.
Example of what Alpha-3 and Alpha-2 mean below:
| Alpha-3 | Alpha-2 | Language name |
| abk | ab | Abkhazian |
| ace | | Acehnese, Achinese |
| ach | | Acoli |
| ada | | Adangme |
| ady | | Adyghe; Adygei |
| aar | aa | Afar |
| etc. | | |
Quote from: warpcafe on 2009.03.06, 17:29:10
With alpha-2 the list will be considerably smaller. Another way would be to select in two steps like country first, then language but that would need a list of combinations allowed (e.g. Switzerland -> ch/de/it...) also a deductive preselection based upon the "lang" setting could ease the selection for the user.
Yes, but even a 2 step one doesn't quite reduce the number of rows enough that has to be handled.
What do you mean with "a deductive preselection based upon the "lang" setting"?
Do you mean something like that it should extract the codepage set on the computer and assume that it may be the first choice?
Yes, I think that it's very likely to eliminate a lot of unnecessary loading procedures, very good thinking!
Hmm, one may even split it up into several files so that it only has to load the file that correspond to that info in the first run and load another if required.
Quote from: warpcafe on 2009.03.06, 17:29:10
A loooong list is not a problem if you can incremetally "type" in it to set the cursor. OS/2 hasn't got that I'm afraid. ;) It also depends if that is a one-time, on-demand or regular (for each file) process... perhaps use a "wizard"-thing that does the trick in two or more steps. That's better than a single spot which is cluttered with controls and listboxes.
Yes, seen a few compiled applications that try that approach in OS/2-eCS. Hmm, would be interesting if someone could come up with a smart and lightweight approach. It depend on how fast one can load, preload or fake the next part and present it to the user.
It would interesting if we could try something else before we give in and use a good and safe wizard-approach though.
Quote from: warpcafe on 2009.03.06, 17:29:10
========8<---------
--------->8========
I'm context-heavy enough I guess. Can you give an example with files (or records)? Perhaps then I can give a suggestion based upon what functional approach is best/fast. We can alos think about transforming the information into something more machine-suited to find better/faster algorithms...
It's easy to extract the text that define what encoding to use as mentioned by the SET statement mentioned earlier, from the dictionary .oxt-file, such as ISO-8859-1, Latin 1, Latin 2, Latin 3, Latin 4 etc.
The function that has to use that information in rxuls does however not like it in that form, so one has to convert it to something else that represent the codepage in plain numeric form. 639, 1004, ... 850 etc. That info is available and can be placed in a file that the app read in some way, but there are a lot of possible encoding-pairs to search through.
What I need to do is to take e.g the text "Latin 4" and find the matching needle "1257" (or the OS/2 equivalvence) in a haystack (Alex Taylors list of Codepages in OS/2-eCS (http://users.socis.ca/~ataylo00/os2/toolkits/uls/codepages.html)), I guess. :P
Coding geek talk:
The SET-statement is present within the first 1 to 20 variables in a stem that one get with the unzip procedureThe time to spend on that is neglectable while searching in a text file for a match that return a value, take many times longer.
One can preload things, make it look fast (progressbar to obscure that it take time) or just fake things.
The thing is that it has to be selected only once for a project, so it shouldn't be a huge thing, but the user should be helped by it to select/set the correct three-letter code and not find it to be in the way.
Coding geek talk:
If one could have a variable for each key-value-pair that would return the right answer right away?
"Latin 4" could perhaps be transformed into the variable latin_4 that would contain the value 1257 (that b.t.w. is Polish) that could be used later on?
or
something so one doesn't have to preload or search through the huge list and many codepagesQuote from: warpcafe on 2009.03.06, 17:29:10
You know with rexx... the string manipulation functions are awesome (also when combined with stems) and one can do nifty tricks with that as long as we know what we would need to achieve. ;)
Cheers,
Thomas
Yes, that's what so awesome with rexx!
Thank you for your suggestions Thomas!
The attached example contain some code to extract words from an .oxt file, but assume that it's encoded with the codepage 1004. It may be different for your language and is mentioned within the first lines of the *.aff-file also present within the .oxt-file. The code assume that you have the file "ooo_swedish_dict_1.29.oxt" present in D: if you don't provide the path and file name as a parameter to the script. You also need rxuls form Alex Taylors page to test the code.
//Jan-Erik
HI,
here's a script that extract the info about encoding inside the dictionary.
just type the name of the script and the path to the dictionary and it will tell you the encoding, on what row it was found inside and name of the package. Ensure you don't place .zip-files in the same directory, unless they're spelling dictionaries from OOorg you want to check. See below.
if you download all packages (yes there's an option for that as well) it'll scan them all as the're named e.g. af_Za.zip to zu_ZA.zip
//Jan-Erik
Sample output:
Encoding:UTF-8 Name:cop_EG Row:19
Encoding:UTF-8 Name:csb_CSB Row:1
Encoding:ISO8859-2 Name:cs_CZ Row:1
Encoding:ISO8859-14 Name:cy_GB Row:79
Encoding:ISO8859-1 Name:da_DK Row:1
Encoding:ISO8859-1 Name:de_AT Row:1
Encoding:ISO8859-1 Name:de_CH Row:1
Encoding:ISO8859-1 Name:de_DE_comb Row:1
Encoding:ISO8859-1 Name:de_DE_neu Row:1
Encoding:ISO8859-1 Name:de_DE Row:1
Encoding:ISO8859-1 Name:de_frami_neu Row:1
Encoding:ISO8859-7 Name:el_GR Row:31
Encoding:ISO8859-1 Name:en_AU Row:7
Encoding:ISO8859-1 Name:en_CA Row:1
Encoding:ISO8859-1 Name:en_GB-oed Row:7
Encoding:ISO8859-1 Name:en_GB Row:7
Encoding:ISO8859-1 Name:en_NZ Row:7
Encoding:ISO8859-1 Name:en_US Row:1
Encoding:ISO8859-1 Name:en_ZA Row:7
Encoding:ISO8859-3 Name:eo_l3 Row:1
Encoding:ISO8859-1 Name:es_ES Row:1
Encoding:ISO8859-1 Name:es_MX Row:1
Encoding:ISO8859-1 Name:et_EE Row:1
Encoding:ISO8859-1 Name:eu-ES Row:9
Note! You need unzip32.dll from Info-Zip to use it, but it should be included in your system...
/* OpenOffice.org v3 Dictionaries -> Dictionary Encoding */
call RxFuncAdd 'SysLoadFuncs', 'RexxUtil', 'SysLoadFuncs'
call SysLoadFuncs
call RxFuncAdd 'UZLoadFuncs', 'UNZIP32', 'UZLoadFuncs'
call UZLoadFuncs
parse arg retval
fname = 'D:\OOorg_Spelling_pack\'
if retval = '' then retval = fname
cfg.file = retval
call RxFuncAdd 'ULSLoadFuncs', 'rxuls', 'ULSLoadFuncs'
call ULSLoadFuncs
curr_time = TIME( 'R' )
CALL SysFileTree retval||'*.oxt', 'fnames', 'FO'
IF fnames.0 = 0 THEN
CALL SysFileTree retval||'*.zip', 'fnames', 'FO'
DO a = 1 to fnames.0
CALL UZFileTree fnames.a, 'file','*.aff'
DO i = 1 to file.0
PARSE VALUE FILESPEC( 'N', file.i ) WITH file.i.name'.'.
lng. = 0
retval = UZUnZipToVar( fnames.a, file.i, 'lng' )
DO j = 1 TO lng.0
PARSE VALUE lng.j WITH 'SET' rest
rest = STRIP( rest )
IF LENGTH( rest ) > 0 THEN DO
say 'Encoding:'||rest||' Name:'||file.i.name||' Row:'||j
LEAVE j
END
END
DROP lng.
END
DROP file.
END
CALL ULSDropFuncs
call UZDropFuncs
exit
Hello,
I've now found a convenient solution on how to load and match codepages in the dictionary. The attached file contain both the executable version, the sources with the new modifications (plus some additional adjustments to fix some bugs and shortcomings in the code). This version require that you have unzip32.dll installed as well.
It's now possible to select an OOorg dictionary and it'll read the info from it and (hopefully) decode it right before it launch wordlist2dawg to create a word list. Note that the included wordlist2dawg is a special build by Franz Bakan that's not included in the packages you can find elsewhere. It require the included gcc433.dll to work, doesn't crash as other versions have done in the past.
eTFBE can also create the word list for frequently used words, but use a rather naive implementation, but it's hopefully ok for most situations.
Notice the changed behaviour:
The file eFTBE.cfg now contain information about codepages and languages and is required to be present if you want to create word lists. eFBE.cfg is actually a renamed .zip-file created from the folder eFTBE that you also find in the package. There's no reference to the files a to z yet, but may come whenever language selection will be implemented. When you replace your current version you'll loose the startup settings saved in the EA (Extended Attributes) of the current eFTBE.cfg for Window size and position. It does however not affect your project folders or the Window setting saved there.
//Jan-Erik
Hello,
I've now looked at rexx/gd for OS/2 to create images with text, but it seem like it doesn't support the required function to create images with TTF fonts in OS/2-eCS.
I therefore need your help to come up with a solution for it. It would be very welcome if another solution excist or if we can get the functionality through GBM by Heiko Nitche, or if I have to write something with your help and add it to rxImgSze. I've written some code in C/C++ before, but require your help to handle image manipulation and fonts to get futher.
//Jan-Erik
Hi jep,
now that's what I would call a coincidence!
I was just about writing you concerning ImageMagick: Given the fact that Paul Smedley just posted about his current port today, I went to check out what ImageMagick can do. And it looks quite promising, see
http://www.imagemagick.org/Usage/text (http://www.imagemagick.org/Usage/text)
and scroll down a little below to see multiline text.
There is no rexx "library" for it, but as along as everything can be done via commandline (which seems to be the case from what I saw) we should be on the safe side and able to control it from rexx too.
Hope this helps
(if it does - be sure to send a note to Paul... :) )
Thomas
Hello Thomas,
Quote from: warpcafe on 2009.03.24, 10:06:15
Hi jep,
now that's what I would call a coincidence!
I was just about writing you concerning ImageMagick: Given the fact that Paul Smedley just posted about his current port today, I went to check out what ImageMagick can do. And it looks quite promising, see
http://www.imagemagick.org/Usage/text (http://www.imagemagick.org/Usage/text)
Neat, exactly what we need, just wonder if one can merge some code and get tiff output right away...
Quote from: warpcafe on 2009.03.24, 10:06:15
and scroll down a little below to see multiline text.
There is no rexx "library" for it, but as along as everything can be done via commandline (which seems to be the case from what I saw) we should be on the safe side and able to control it from rexx too.
Hope this helps
(if it does - be sure to send a note to Paul... :) )
Thomas
Hmm, I've noticed that it's quite easy to rewrite textmode apps, so I think I'll rewrite tesseracct as a rexx dll, so why not ImageMagic as well. ;D
The only problem is that it may take time to set everything up right, and make it work (if at all possible) with OpenWatcom.
//Jan-Erik