• Welcome to OS2World OLD-STATIC-BACKUP Forum.
 

News:

This is an old OS2World backup forum for reference only. IT IS READ ONLY!!!

If you need help with OS/2 - eComStation visit http://www.os2world.com/forum

Main Menu

split text into words (database)

Started by jep, 2008.06.17, 16:03:55

Previous topic - Next topic

jep

Hello,

Familiar with databases and how they work?
I'd like to get some help from you to write a function that can split plain text into words, preferably on the server in its own scripting language to remove the burdon from client apps.

The "easy part" of it is that one can use
tempvariable := split_part( txt, ' ', i );
in PostgreSQL (PL/pgSQL) but the trickier part is that I want it to check each separately defined delimiter in the database as seen futher below and return things in the right order to keep words clean from those extra characters.

Example Input:
This is a test! Please, ensure that "the code" can sparate each word as accurate as possible.
Example Output (indicated with a # after each new record to be):
This# #is# #a# #test#!# #Please#,# #ensure# #that# #"#the# #code#"# #can# #separate# #each# #word# #as# #accurate# #as# #possible#.#

To add more problems... some words may contain characters from the table below that do belong there... so it's not always correct just to chop a word in two at a '-character. or split "e.g." i 4 pieces.

//Jan-Erik

Table to look into

Text_IDText
1
2,
3.
4?
5;
6:
7-
8+
9/
10°
11¹
12²
13³
14@
15(
16)
17%
18&
19<
20>
21!
22"
23'

ddan

Ran into this when making a tool for checking my word usage and to make a dictionary. Below, if you can figure it out after the online editor mangled it somewhat, is the help screen of my (Turbo Pascal 5.5) program. Doesn't seem possible to exclude all symbols all the time - at least not in English - so I just provided command line switches for various types of text. And I think you missed listing the really annoying underline: "_", which programmers like to use in variable names. I didn't handle "@" but nowadays it should clearly be counted as part of a word if in middle of alphas as in web addresses. Good luck handling all the special cases...

   Exclude  : /a = apostrophes      /h = hyphens   <───[As part of word]──┐');
   Include  : /',apost,' = all apostrophes  /- = all hyphens  /u = underlines<,
   Output   : /f = word frequency   /#[-]nnn = truncate above/below frequency');
   Translate: /t = translate foreign characters into a-z');   
- By default, leading or trailing apostrophes and hyphens are stripped;');
     others are included in word.  Exclude all with /a or /h switches.');
     Include lead/trail apostrophes or hyphens with /',apost,' or /- switches.');
- Underlines are NOT included in word by default: add with /u switch.');
- Translates foreign characters as below (ligatures are expanded to two):');
Ä Å â ä à å á ª Æ æ Ç ç  É é ê ë è ƒ ï î ì í £ ô ö ò Ö ó º Ñ ñ × Ü ü û ù ú ÿ');
          a       │ae       c  │    e    │f│   i   │l│     o     │ n │p│    u    │ y ');
- ß used for German s-set is translated to one "s"; not entirely satisfactory.');
- Output is all lowercase, 7-bit ASCII, one word per line (plus #, with /f).');