Hello,
Familiar with databases and how they work?
I'd like to get some help from you to write a function that can split plain text into words, preferably on the server in its own scripting language to remove the burdon from client apps.
The "easy part" of it is that one can use
tempvariable := split_part( txt, ' ', i );
in PostgreSQL (PL/pgSQL) but the trickier part is that I want it to check each separately defined delimiter in the database as seen futher below and return things in the right order to keep words clean from those extra characters.
Example Input:
This is a test! Please, ensure that "the code" can sparate each word as accurate as possible.
Example Output (indicated with a # after each new record to be):
This# #is# #a# #test#!# #Please#,# #ensure# #that# #"#the# #code#"# #can# #separate# #each# #word# #as# #accurate# #as# #possible#.#
To add more problems... some words may contain characters from the table below that do belong there... so it's not always correct just to chop a word in two at a '-character. or split "e.g." i 4 pieces.
//Jan-Erik
Table to look into
Text_ID | Text |
1 | |
2 | , |
3 | . |
4 | ? |
5 | ; |
6 | : |
7 | - |
8 | + |
9 | / |
10 | ° |
11 | ¹ |
12 | ² |
13 | ³ |
14 | @ |
15 | ( |
16 | ) |
17 | % |
18 | & |
19 | < |
20 | > |
21 | ! |
22 | " |
23 | ' |
Ran into this when making a tool for checking my word usage and to make a dictionary. Below, if you can figure it out after the online editor mangled it somewhat, is the help screen of my (Turbo Pascal 5.5) program. Doesn't seem possible to exclude all symbols all the time - at least not in English - so I just provided command line switches for various types of text. And I think you missed listing the really annoying underline: "_", which programmers like to use in variable names. I didn't handle "@" but nowadays it should clearly be counted as part of a word if in middle of alphas as in web addresses. Good luck handling all the special cases...
Exclude : /a = apostrophes /h = hyphens <───[As part of word]──┐');
Include : /',apost,' = all apostrophes /- = all hyphens /u = underlines<,
Output : /f = word frequency /#[-]nnn = truncate above/below frequency');
Translate: /t = translate foreign characters into a-z');
- By default, leading or trailing apostrophes and hyphens are stripped;');
others are included in word. Exclude all with /a or /h switches.');
Include lead/trail apostrophes or hyphens with /',apost,' or /- switches.');
- Underlines are NOT included in word by default: add with /u switch.');
- Translates foreign characters as below (ligatures are expanded to two):');
Ä Å â ä à å á ª Æ æ Ç ç É é ê ë è ƒ ï î ì í £ ô ö ò Ö ó º Ñ ñ × Ü ü û ù ú ÿ');
a │ae c │ e │f│ i │l│ o │ n │p│ u │ y ');
- ß used for German s-set is translated to one "s"; not entirely satisfactory.');
- Output is all lowercase, 7-bit ASCII, one word per line (plus #, with /f).');