Screen Reader/2 - Programmed Access to the GUI: Difference between revisions
(Created page with " By Jim Thatcher Interaction Technology Mathematical Sciences Department IBM Research Yorktown Heights, NY 19598 ==Abstract== In this paper I will describe the Sc...") |
m (→BIBLIOGRAPHY) |
||
(7 intermediate revisions by 2 users not shown) | |||
Line 4: | Line 4: | ||
[[IBM]] Research | [[IBM]] Research | ||
Yorktown Heights, NY 19598 | Yorktown Heights, NY 19598 | ||
{{IBM-Reprint}} | |||
==Abstract== | ==Abstract== | ||
In this paper I will describe the Screen | In this paper I will describe the Screen Reader/2 personality, that is the 'look and feel' of Screen Reader/2. As I do that I hope to give the reader an introduction to the Profile Access Language (PAL), which is one of the most important components of Screen Reader/2. This PAL introduction will not be formal, in the sense of careful syntax and semantics; I want to get across the idea of PAL so that one might want to program in the Profile Access Language to access the Graphical User Interface (GUI) of OS/2 and of Windows applications running under OS/2. | ||
Reader/2 | |||
of Screen Reader/2. | |||
the | |||
Language (PAL), which is one of the most important | |||
components | |||
introduction | |||
careful syntax and semantics; I want to get across | |||
the idea of PAL so that one might want to | |||
in | |||
Graphical User Interface | |||
Windows applications running under OS/2. | |||
Screen | ==1.0 INTRODUCTION== | ||
IBM Screen Reader/2 is the only access system that gives blind computer users access to OS/2, DOS and Windows applications (running seamlessly). It is the only screen reader to have a fully programmable interface. Screen Reader/2 has grown over nine years in response to the demands of users in IBM and outside who needed the function for their work, education and home environments. | |||
the | |||
==2.0 BASIC SCREEN READER/2== | |||
Screen Reader/2 for OS/2 is a follow-on to the DOS based product that was released in January of 1988 and upgraded two times since. Two things about IBM Screen Reader distinguish it from all other screen access software; the separate 18 key keypad and its total programmability. On the latter count, the power of the programming language, PAL, is a significant distinguishing feature. | |||
Screen | ===2.1 THE SCREEN READER/2 PACKAGE=== | ||
keypad | Screen Reader/2 for OS/2 consists of an 18 key keypad, a keypad cable, some audio cassettes, some hard copy documents, and two diskettes with on line documentation and software. The keypad is attached through the mouse port. If such is not available, there is an adapter card for the keypad that can be used instead. | ||
documents, | |||
software. | |||
such is not available, there is | |||
keypad that can be used instead. | |||
The software can be grouped into roughly five sections. | The software can be grouped into roughly five sections. | ||
#Screen Reader executables and dynamic link libraries. | |||
#Configuration files and utilities. | |||
#Installation utilities. | |||
#The PAL compiler and profiles. | |||
#On line documentation. | |||
The keypad is used as an input device to Screen Reader/2 to request information from the display or to change settings in the speech environment. There are literally hundreds of states that effect how Screen Reader/2 responds. | |||
request information from the display or to | |||
in | |||
states that effect how Screen Reader/2 responds. | |||
Screen Reader/2 responds or speaks as the result of a user request through the keypad or because of some autospeak. In both cases the nature of the response is determined by the current Screen Reader/2 state and the PAL profile that is active. The important point here is that everything that Screen Reader/2 does is a consequence of some profile code written in the Profile Access Language. | |||
the Screen Reader/2 | |||
===2.2 THE ROLE OF PROFILES=== | |||
Profiles both define the basic screen reading environment, the Screen Reader/2 personality, and they are also used as add-ons to tailor that basic environment to specific applications when that is desirable or even necessary. | |||
applications easier to | This aspect of Screen Reader is often misunderstood. The application specific profiles are developed for making applications easier to use. Everything that is done in application specific profiles could be done without them (probably), but with more effort (probably a lot more effort). | ||
application | |||
(probably), but | |||
effort). | |||
The | The simplest example of the desirability of an application specific profile is when the application has a status area in which, say, error messages are displayed. Using the keypad and standard core requests, the Screen Reader/2 user can read that area to find out about any messages without a special profile. With an application specific profile, however, that area can be monitored and automatically read. | ||
specific profile is when the application has a | In addition, a simple application specific key sequence could be defined to read the status area. | ||
in | |||
keypad and standard core requests, the Screen Reader/2 | |||
can | |||
special profile. | |||
however, | |||
In addition, a | |||
could be defined to read the status area. | |||
There | There are about 30 applications specifically supported by the Screen Reader/2 product. These profiles are supplied for OS/2, DOS, and Windows applications. | ||
the Screen Reader/2 product. | |||
for OS/2, DOS, and Windows applications. | |||
Whether | Whether considering application profiles or the base Screen Reader profile, it is crucial also to understand that Screen Reader's behavior is completely determined by the current active profile or profiles. | ||
Reader profile, it is crucial also to understand that Screen | |||
Reader's behavior is completely determined | |||
active profile or profiles. | |||
===2.3 | ===2.3 SESSIONS=== | ||
There is one base or core profile, called srd2.kpd. This profile textually includes the sub-profiles for the edit facility, mouse movement, icons, scanning, etc. These are all conceptually and syntactically part of the base profile. | |||
The way this base profile works is that it is loaded and active when Screen Reader/2 is started. This base profile and any environment settings that might be determined by it, define what we call default Screen Reader/2 session. | |||
profile | |||
Whenever a new profile is added to (we say pushed on) the base profile, i.e., the default session, that action creates a new Screen Reader session associated with the foreground process. You can save settings in that session, you can push another profiles for that session. When the process is killed the session goes away. | |||
To illustrate this session concept, just imagine that there are two sessions, the default session and a new session created by pushing a profile for, say, the System Editor. Then you could be in spell format (where everything is spelled) in the System Editor session, and text format (normal reading) in the default session. Then any time you switched from the default session to the system editor session, you would switch from text to spell format and conversely. | |||
a | |||
===2.4 THE NATURE OF PROFILES=== | |||
A simple example will help to illustrate the use of PAL and the idea that any Screen Reader/2 response can occur as a result of a keypad entry or some event that triggers an | |||
===2.4 | |||
A | |||
the idea that any Screen Reader/2 response can | |||
result | |||
autospeak. | autospeak. | ||
EXAMPLE 1 | EXAMPLE 1 | ||
include 'srd2.inc' | include 'srd2.inc' | ||
Var TimeWatch: AutoHandle; | Var TimeWatch: AutoHandle; | ||
{1} Msg ( 'it is ', DateTime( DT_HOUR ), ' o clock' ); | {1} Msg ( 'it is ', DateTime( DT_HOUR ), ' o clock' ); | ||
AutoSpeak TimeWatch | AutoSpeak TimeWatch | ||
Watch DateTime( | Watch DateTime( DT_HOUR ) Do | ||
Msg ( 'it is ', trigger , ' o clock' ); | Msg ( 'it is ', trigger , ' o clock' ); | ||
The file srd2.inc has definitions of constants like DT_HOUR. The third line of PAL code above gives meaning to the 1 key, {1}, of the screen Reader Keypad. When this fragment is active and the 1 key is depressed, the message 'it is 9 o clock' is spoken. This line of PAL Code is executed when the 1 key is pressed. | |||
The file srd2.inc has definitions of constants like DT_HOUR. | |||
The third line of PAL code above gives meaning to the 1 key, | |||
{1}, | |||
active and the 1 key is depressed, the message 'it | |||
clock' | |||
the 1 key is pressed. | |||
On the other hand, the autospeak with handle TimeWatch, watches the value returned by the function DateTime(DT_HOUR) which, the reader might have guessed, is the current hour. When that changes, then the body of the autospeak is executed. That body is also a message statement (Msg sends its arguments to the attached serial device). The value of the reserved variable, trigger, is the value that triggered the autospeak, in this case the current hour. So at the moment the hour changes to 9, the message 'it is 9 o clock' is announced, just like pressing key 1. | |||
===2.5 BASIC REVIEW REQUESTS=== | |||
Screen Reader has many standard review requests programmed into its base profile (os2core.kpd, included in srd2.kpd mentioned above). Each of these requests is tempered by the Mode and Format setting. | |||
the | |||
current | There are two reading 'modes,' called cursor and pointer. In cursor mode all reading requests are relative to the application cursor; in pointer mode all requests are relative to Screen Reader/2's own 'pointer' (called the review cursor by some screen reading packages). Because other screen readers do it differently, it is important to emphasize that there are many reading requests and all of them work in either pointer or cursor mode. The request is the same sequence on the Screen Reader keypad, like {2} for current line. Depending on the current mode, that will be the cursor line, or the line containing the Screen Reader/2 pointer. | ||
the cursor line, or the line containing the Screen | |||
pointer. | |||
There | There are five reading formats called text, pronounce, spell, phonetic, and ASCII. In text format Screen Reader tries to read as if reading a book. In pronounce format, punctuation is also announced. In spell format, all words are spelled and in phonetic format everything is spelled using the international phonetic alphabet. In ASCII format you hear the ASCII value of each character. | ||
spell, phonetic, and ASCII. | |||
tries | |||
punctuation is also announced. | |||
are | |||
using the international phonetic alphabet. In | |||
you hear the ASCII value of each character. | |||
Again, | Again, I want to emphasize that all the basic requests are available and the result of each depends on the current format setting. There are shortcuts, however. For example, it is unlikely you really want ASCII format, so a key sequence is defined to give the ASCII value of the current character, and none to switch to ASCII format. Similarly, | ||
available and the result of | another sequence is defined to spell the current word without changing the active format. | ||
format | |||
it is unlikely you | |||
sequence | |||
character, and none to switch to | |||
another | |||
Here is a list of some basic review requests. | Here is a list of some basic review requests. | ||
* Read the entire window or read from current position to the end. | |||
* Read the current, previous, next sentence, line, word, or character. | |||
the end. | * Move pointer to cursor, to top left, to bottom left, or to right edge. | ||
* Spell the current, previous or next word. | |||
* ASCII value of current character. | |||
or character. | * Current character in Phonetic alphabet. | ||
* Search for a string, search again, or search from bottom. | |||
This list could be expanded, but such detail becomes tedious. There is a separate, though related, list which is important in portraying the Screen Reader/2 personality. As I mentioned above there are many components of the Screen Reader/2 state that can be set by the user, and in fact, can be different in different Screen Reader/2 sessions. The ways these state components are set is not so important as is the following illustrative (certainly not exhaustive) list. | |||
to right edge. | * Mode cursor or pointer. | ||
* Format: text, pronounce, spell or phonetic. | |||
* Ignore capitalization or not. | |||
* Announce spaces or not. | |||
* Treat entire screen as single line (wrap) or not. | |||
* Announce line numbers while reading or not. | |||
* Use the dictionary or not. | |||
bottom. | |||
This | |||
tedious. | |||
important | |||
I mentioned above there are many components | |||
Reader/2 state that can be set by the user, and in fact, can | |||
be different in different Screen Reader/2 sessions. The ways | |||
these state components are set is not so important as is the | |||
following illustrative (certainly not exhaustive) list. | |||
The following are GUI specific. | The following are GUI specific. | ||
* Hear drawing noise or not. | * Hear drawing noise or not. | ||
* Include icons or not. | * Include icons or not. | ||
Having just listed a sampling of Screen Reader/2 core requests, now I want to give a sample how some of these appear in PAL. Remember that we are not detailing the syntax and semantics of PAL. I believe that the examples will, for the most part, be self explanatory. The following examples are exactly as they appear in the profile, os2core.kpd, shipped with Screen Reader/2. | |||
Having | |||
requests, now I want to give a | |||
appear in PAL. Remember that we are not detailing the syntax | |||
and | |||
the most part, be self explanatory. | |||
are | |||
shipped with Screen Reader/2. | |||
EXAMPLE 2 | EXAMPLE 2 | ||
Line 278: | Line 114: | ||
{*C} 'Ascii value of current character' | {*C} 'Ascii value of current character' | ||
Save( Format); Set( Format, Ascii); Say( Char); | Save( Format); Set( Format, Ascii); Say( Char); | ||
A key definition in PAL is a key or key sequence in braces, followed by an optional comment used for help. After that is a sequence of PAL commands. | |||
Only four Screen Reader/2 commands are used in the key definitions in example above. The Save command saves one or more components of the Screen Reader/2 state. Any saved component is restored automatically at the end of the command sequence. | |||
Only four Screen Reader/2 | |||
definitions | |||
more components of the | |||
component | |||
command sequence. | |||
The Set command sets | The Set command sets a component of the state. In the | ||
statement | statement sequence for {0A}, Wrap is set to on which means | ||
the entire view is treated as one line. | the entire view is treated as one line. The format is set | ||
to ASCII in the command sequence for {*C}. | to ASCII in the command sequence for {*C}. | ||
The | The Get command positions a local pointer to a position | ||
specified by its | specified by its arguments. When a command sequence is | ||
started, | started, the local pointer is assigned the cursor position | ||
if in cursor mode, and the pointer position | if in cursor mode, and the pointer position if in pointer | ||
mode. | mode. The local pointer's coordinates are always given in | ||
two read-only variables, row and col. The local pointer gets | two read-only variables, row and col. The local pointer gets | ||
updated throughout a command sequence and is assigned to the | updated throughout a command sequence and is assigned to the | ||
Screen Reader/2 pointer at the end of | Screen Reader/2 pointer at the end of the sequence if in | ||
pointer mode. | pointer mode. | ||
The Say command is used to send text from the display to the | The Say command is used to send text from the display to the | ||
synthesizer | synthesizer (or other device). Depending on its argument, | ||
Say works on a line (or whole screen if wrap is on), word or | Say works on a line (or whole screen if wrap is on), word or | ||
character. | character. | ||
===2.6 | ===2.6 EDITING=== | ||
The Screen Reader/2 edit facility provides for | The Screen Reader/2 edit facility provides for feedback as | ||
the user moves the cursor and types. The edit facility is in | the user moves the cursor and types. The edit facility is in | ||
fact | fact another profile, and a rather complex profile at that. | ||
I don't want to explore the design | I don't want to explore the design of that profile here. | ||
(Neither do I want to discuss the question of whether or not | (Neither do I want to discuss the question of whether or not | ||
the edit facility functions should or should not be built in | the edit facility functions should or should not be built in | ||
at a lower level in Screen Reader/2.) | at a lower level in Screen Reader/2.) | ||
Instead | Instead I will mention the 5 major feedback options in the | ||
edit facility. | edit facility. | ||
====Line Browse. ==== | |||
With Line browse, every time the cursor | |||
moves one position to a new character, that character is | moves one position to a new character, that character is | ||
spoken; | spoken; when the cursor moves horizontally a word at a | ||
time, the word is spoken, and when | time, the word is spoken, and when the cursor changes | ||
row | row or the line changes as in a scroll, the new line is | ||
spoken. | spoken. | ||
====Flush.==== | |||
response cuts of (stops) the previous response, | With flush turned on, each new edit facility response cuts of (stops) the previous response, | ||
The | The Default edit settings are line browse and flush. In | ||
addition there are three 'echo' settings. | addition there are three 'echo' settings. | ||
====Character echo.==== | |||
Echo each character as typed. | |||
====Word echo.==== | |||
Echo changed words as the space is typed at | |||
the end of the word. | the end of the word. | ||
====Line echo.==== | |||
Echo changed lines when move to new line. | |||
There | There other options in the edit facility that can make life | ||
easier | easier under some circumstances, including a margin | ||
indicator and a column browser. | indicator and a column browser. | ||
==3.0 | ==3.0 NAVIGATING AROUND THE GRAPHICAL USER INTERFACE== | ||
Almost all OS/2 and Windows applications respond to keyboard | Almost all OS/2 and Windows applications respond to keyboard | ||
input | input as a substitute for mouse input. As a simple example, | ||
the window list (the list of running programs) is | the window list (the list of running programs) is displayed | ||
using | using the right mouse button on the desktop, or using the | ||
keyboard combination Ctrl+Esc. | keyboard combination Ctrl+Esc. One can highlight an item in | ||
a list by using the mouse and single click, or by moving | a list by using the mouse and single click, or by moving to | ||
the | the item with the arrow keys. The item is invoked (or | ||
opened) | opened) using a double click with the mouse or by | ||
highlighting the item and pressing enter. | highlighting the item and pressing enter. | ||
In | In general, a blind user navigates around the GUI with the | ||
keyboard. During that navigation, Screen | keyboard. During that navigation, Screen Reader/2 provides | ||
constant feedback. | constant feedback. | ||
===3.1 | ===3.1 THE CONCEPT OF VIEW=== | ||
Under the default settings, Screen Reader/2 always restricts reading to what it determines to be the main window and all text and icons contained in that main window. Because the word 'window' is over used, we call that rectangle the Screen Reader/2 view. As the user switches between applications, or between main windows, Screen Reader/2 automatically adjusts the view to the current main window. | |||
Under the default settings, Screen Reader/2 always restricts | |||
reading | |||
text and icons contained in that main | |||
word | |||
Screen | |||
applications, | |||
automatically adjusts the view to the current main window. | |||
Besides adjusting the view; | Besides adjusting the view; Screen Reader/2 automatically announces the window title when the main window or application changes. | ||
announces | |||
application changes. | |||
Once the view has been established, the view is treated very | Once the view has been established, the view is treated very much as if it were a text window. All the text in that view is sorted by baseline into a certain number (view.rows) of distinct rows and each row is assumed to have the same number (view.cols) of 'columns' namely the maximum character length of the rows. | ||
much as if it were a text window. All the text in that | |||
is | |||
distinct rows and each row | |||
number (view.cols) of 'columns' namely the maximum character | |||
length of the rows. | |||
We | We believe that this simplifies access to the graphical view, though it is not always an accurate reflection of what is displayed. Our experience has been, with many users of | ||
view, though it is not always an accurate reflection of what | Screen Reader/2, that this filtering of the displayed data is a help for review and that no significant information is lost. | ||
is displayed. | |||
Screen | |||
is a help for review and that no significant information | |||
lost. | |||
The | The characters in the first 'column' cannot be expected to be seen to be lined up vertically as they are in a text screen. There is no implication about vertical positioning even of different rows. One 'row' might be huge and take up most of the view, with the remaining rows crammed in a small font at the bottom of the view, like, say, footnotes. One block of text could have a baseline just above a block to its right. These would be considered distinct rows. | ||
be seen to be lined up vertically as | |||
screen. | |||
even of different rows. | |||
most of the view, with the remaining rows crammed in a small | |||
font at the bottom of the view, like, say, footnotes. | |||
block | |||
its right. | |||
Two blocks of text could be the same font, | Two blocks of text could be the same font, size and style and have the same baseline and still be on different 'rows.' This happens when the text occurs in separate windows. And the reason we put text in different windows into distinct rows is because, almost certainly, one wants to be read the text of those windows separately. | ||
and have the same baseline and still be on different 'rows.' | |||
This | |||
the reason we put text in different | |||
rows | |||
text of those windows separately | |||
All the information that is ignored by this row-column access of the view is available to the user. This includes: | |||
* The pixel position of each character and its size (char.left, char.bottom, char.right, char.top). | |||
* The font, pitch, style and color of each character. (char.font, char.pitch, char.style, char.color). | |||
* The pixel size of the view and its position on the desktop. (view.top, view.left, view.bottom, view.right). | |||
* The identity of the window containing the text, the window handle By calculations in PAL (we have not found reason to use these in basic profiles). | |||
* Whether one row or column truly lines up below another. | * Whether one row or column truly lines up below another. | ||
* The amount of white space between rows. | * The amount of white space between rows. | ||
* | * The amount of white space at the beginning and end of rows. | ||
The reason the view is set as it is by Screen Reader/2 is that conventional review requests would be utterly confusing without it. Successive lines or blocks of text might come from different main windows and make no sense at all. As with most function and default operation of Screen Reader/2, the user can have it both ways. With standard requests, the user can make the whole display (or desktop) the view, or only the contents of a single push button. | |||
Views are determined by the windows. A view always includes all child windows contained within a specified window. Thus the view determined by the desktop window includes everything displayed; the view determined by a push button is just the button text or icon. Pushbuttons are windows in the technical sense (except in the case of some Windows applications written by MicroSoft). | |||
Views are | Views are not the only way of restricting reading. Viewports are rectangular regions within a view, to which reading can be constrained. | ||
===3.2 USING CONTROLS WITHIN A VIEW=== | |||
A dialog is one kind of main window that does not fit well | |||
into the row and column format described above. Dialogs | |||
A | typically consist of several windows (in the technical | ||
into the row and column format | sense, again) which allow the user to enter data and make | ||
typically | choices. The individual windows (buttons, entry fields, | ||
sense, again) which allow the user to enter | static text fields, check boxes) are called controls. The | ||
choices. | user makes choices with the buttons, entry fields and check | ||
static text fields, check boxes) are | boxes; the prompts or headings are provided with by the | ||
user | |||
boxes; the prompts or headings | |||
'static text.' | 'static text.' | ||
A | A sample will help. In the OS/2 Enhanced editor, a search | ||
dialog appears when you want | dialog appears when you want to search for and possibly | ||
replace | replace text in a file you are editing. At the top of the | ||
dialog are two entry fields. The first has | dialog are two entry fields. The first has the prompt (or | ||
name) | name) 'search' and the second has the prompt 'replace.' You | ||
type in the string you want to search for in the first entry | type in the string you want to search for in the first entry | ||
field and press enter. | field and press enter. | ||
These two entry fields and their prompts | These two entry fields and their prompts actually comprise | ||
four | four windows in this view, and four rows as well. As you | ||
move to them, however, with the standard | move to them, however, with the standard tab-key movement, | ||
you | you hear 'search entry blank' and 'replace entry blank' | ||
respectively. | respectively. When the data is put in the entry field, then | ||
the editing functions of Screen Reader/2 can be used. | the editing functions of Screen Reader/2 can be used. | ||
Because of the feedback from Screen Reader/2 | Because of the feedback from Screen Reader/2 to the user, | ||
there | there is little interest in what rows and columns are | ||
involved, or the fact that the sighted user might | involved, or the fact that the sighted user might say that | ||
the prompt and the entry field data appear to be on the same | the prompt and the entry field data appear to be on the same | ||
'row.' | 'row.' | ||
Continuing | Continuing with this example, below the entry fields is a | ||
box (looks like a window and is called a group box) | box (looks like a window and is called a group box) with a | ||
heading 'Options.' That box contains five check box controls | heading 'Options.' That box contains five check box controls | ||
which | which consist of small squares (about one-quarter inch) | ||
which may or may not have check marks in them. Each has text | which may or may not have check marks in them. Each has text | ||
immediately to the right of the small | immediately to the right of the small square. These are | ||
used to select options such as 'ignore case,' or 'change all | used to select options such as 'ignore case,' or 'change all | ||
occurrences.' | occurrences.' | ||
As | As the user tabs into this area the and arrows around the | ||
check boxes, Screen Reader/2 responds with | check boxes, Screen Reader/2 responds with | ||
'Group box options check box ignore case checked' | 'Group box options check box ignore case checked' | ||
or | or | ||
'Group box options check | 'Group box options check box change all occurrences not | ||
checked' | checked' | ||
It | It is standard in the GUI to toggle the check state of a | ||
check box with the space bar. Hitting the space bar when | check box with the space bar. Hitting the space bar when on | ||
the ignore case check box, results in the announcement | the ignore case check box, results in the announcement | ||
'Group box options check box ignore case not checked' | 'Group box options check box ignore case not checked' | ||
Line 507: | Line 279: | ||
Finally, at the bottom of this dialog is a group of six push | Finally, at the bottom of this dialog is a group of six push | ||
buttons, | buttons, things like 'Find,' 'Change,' and 'Cancel.' When | ||
the user tabs to this group of buttons, | the user tabs to this group of buttons, and arrows around | ||
there, | there, the information, is presented in a form similar to | ||
that for check boxes: | that for check boxes: | ||
'push button find' | 'push button find' | ||
'push button cancel' | 'push button cancel' | ||
===3.3 | ===3.3 MOUSE ACTIONS AND OTHER TRICKS IN SCREEN READER/2.=== | ||
There | There is no doubt that not using a mouse with the graphical | ||
user interface is a disadvantage. | user interface is a disadvantage. It is quicker to do some | ||
things with a mouse than with the keyboard equivalents (when | things with a mouse than with the keyboard equivalents (when | ||
those | those keyboard equivalents are known). The worst part of | ||
this is probably the fact that help panels are written | this is probably the fact that help panels are written with | ||
the | the mouse user in mind. They often do not include the the | ||
keyboard alternatives. | keyboard alternatives. | ||
Screen Reader/2 has done some things to make this | Screen Reader/2 has done some things to make this situation | ||
better. | better. One of the most important features is our switch | ||
list. | list. | ||
A mouse user brings a window to the foreground by moving the | A mouse user brings a window to the foreground by moving the | ||
mouse to any part of that window and clicking with the | mouse to any part of that window and clicking with the left | ||
mouse | mouse button. If that window is totally obscured, either | ||
other windows must be moved, resized, minimized, or | other windows must be moved, resized, minimized, or closed; | ||
or | or the mouse user can use the keyboard equivalent if they | ||
know what it is. | know what it is. | ||
From the keyboard, Ctrl+Esc | From the keyboard, Ctrl+Esc brings up the list of open | ||
programs. | programs. Then the Screen Reader/2 user can scan that list | ||
one at a time, or pres the first letter of the title, | one at a time, or pres the first letter of the title, until | ||
the | the item is found and then open the folder or start the | ||
program by pressing Enter. This is | program by pressing Enter. This is a significantly longer | ||
activity than moving the mouse and clicking. | activity than moving the mouse and clicking. | ||
The | The Screen Reader/2 user can identify up to 15 windows (in | ||
the default case) and when they want to | the default case) and when they want to switch to one of | ||
their | their favorite windows (whether obscured or not) they just | ||
enter the number of the window preceded by the simple | enter the number of the window preceded by the simple chord | ||
89 | 89 on the keypad. Movement between a collection of standard | ||
windows becomes easier than the same task for the | windows becomes easier than the same task for the (sighted) | ||
mouse user. | mouse user. | ||
The | The OS/2 2.1 desktop covers the entire display. On it are | ||
icons for programs and program | icons for programs and program folders and other objects | ||
like | like printers. To open (start) one of these objects the | ||
mouse user must find it, and | mouse user must find it, and double click. Other windows | ||
that | that are open are very likely to obscure one or more of | ||
these icons, and the only option here for the mouse user | these icons, and the only option here for the mouse user is | ||
to | to close, move, resize, or minimize some of the open | ||
windows. This is often a problem for the sighted | windows. This is often a problem for the sighted user, but | ||
it would be a disaster for the blind user. | it would be a disaster for the blind user. | ||
Screen | Screen Reader/2 solves this problem with an alternative | ||
desktop which is a normal folder object and can | desktop which is a normal folder object and can be brought | ||
to the foreground just like other folders. In particular the | to the foreground just like other folders. In particular the | ||
Line 569: | Line 341: | ||
As the last subject in this section, mouse actions (left and | As the last subject in this section, mouse actions (left and | ||
right | right button, click and double click) can be preformed with | ||
Screen Reader/2 at any time. | Screen Reader/2 at any time. | ||
Think of the Search Dialog for the Enhanced Editor described | Think of the Search Dialog for the Enhanced Editor described | ||
in the previous section. As we discussed there, | in the previous section. As we discussed there, the normal | ||
use | use of that dialog is to tab or arrow key to the required | ||
spot, enter data, or change options with the space bar, | spot, enter data, or change options with the space bar, and | ||
then | then press enter to carry out the action. An alternative is | ||
available for any application. | available for any application. | ||
Using a couple of basic Screen Reader/2 | Using a couple of basic Screen Reader/2 requests from the | ||
keypad, | keypad, the blind user can perform a mouse single click | ||
(chord 23 then 4) or double click (chord 23 | (chord 23 then 4) or double click (chord 23 then 5). So | ||
when the Screen Reader/2 user reviews the contents of dialog | when the Screen Reader/2 user reviews the contents of dialog | ||
and | and is reading, for example, the 'ignore case" checkbox, a | ||
single click (using the keypad) will change the check | single click (using the keypad) will change the check state | ||
of the check box from checked to non-checked or conversely. | of the check box from checked to non-checked or conversely. | ||
==4.0 | ==4.0 SOME TECHNICAL CONSTRUCTS IN PAL== | ||
In | In this last section I will describe a couple of the unique | ||
and | and powerful features of the Profile Access Language. | ||
Continuing | Continuing in the spirit set in the first section, I will | ||
not | not stress the detailed syntax of PAL, but instead | ||
illustrate the ideas with PAL code and descriptive text. | illustrate the ideas with PAL code and descriptive text. | ||
===4.1 | ===4.1 AUTOSPEAKS AND PROCEDURES=== | ||
In | In the first section, EXAMPLE 1 showed how an autospeak | ||
could be defined to announce the time when the hour changed. | could be defined to announce the time when the hour changed. | ||
The | The watched expression in that case was a function | ||
(DateTime) which returns the current hour. | (DateTime) which returns the current hour. | ||
This | This idea, watching a function or procedure, is key to a | ||
very powerful concept for Screen Reader/2; but some | very powerful concept for Screen Reader/2; but some history | ||
first. | first. | ||
Ten | Ten years ago, when Screen Reader was a research project | ||
called PC SAID, we did | called PC SAID, we did have programmable autospeaks, but | ||
only | only to watch specified screen positions, and somewhat | ||
later, numeric expressions. In its early days, this was | later, numeric expressions. In its early days, this was far | ||
ahead | ahead of anything else that was available or planned in the | ||
screen access area. | screen access area. | ||
Lets review how the autospeak construct | Lets review how the autospeak construct works. The syntax | ||
(roughly) is: | (roughly) is: | ||
Line 622: | Line 394: | ||
do <Command Sequence> | do <Command Sequence> | ||
The | The autohandle is a variable needed to query the state of the autospeak, and to turn it on or off. It is not important for understanding this discussion. The expression is just that, any expression in PAL. So you can watch a part of the display as in the original PC SAID autospeaks by watching, for example, | ||
the autospeak, and to | |||
important for understanding this discussion. | |||
is just that, any expression in PAL. So you can watch a part | |||
of | |||
watching, for example, | |||
Display( 23, 67, 4) | Display( 23, 67, 4) | ||
to watch row 23, column 67 for 4 characters. | to watch row 23, column 67 for 4 characters. | ||
Then Command Sequence is any PAL sequence of commands. | Then Command Sequence is any PAL sequence of commands. | ||
The way this works | The way this works is that the expression is evaluated | ||
periodically, | periodically, say every tenth of a second. If that value | ||
changes (and stays changed for | changes (and stays changed for some specified time -- a | ||
detail | detail that won't bother us) then the autospeak body (the | ||
Command Sequence) is executed. | Command Sequence) is executed. As indicated in the time | ||
example | example in Section 2.4. there is a convenient system | ||
variable, | variable, trigger, which contains the value of the | ||
expression that triggered the autospeak. | expression that triggered the autospeak. | ||
All | All of this is pretty much the same as has been available | ||
since the days of PC SAID; the syntax has | since the days of PC SAID; the syntax has been cleaned up | ||
and there are some conveniences. | and there are some conveniences. But what is truly new here | ||
is | is the fact that PAL for Screen Reader/2 has user-defined | ||
procedures that return values, and, of course, these can | procedures that return values, and, of course, these can be | ||
parts | parts of expressions. This is a major functional change, | ||
significantly increasing the power of the autospeak feature. | significantly increasing the power of the autospeak feature. | ||
Why is this powerful? Because | Why is this powerful? Because in principle it makes the | ||
autospeak | autospeak construct universal, that is, one could, in | ||
principle, design and code an autospeak to announce whatever | principle, design and code an autospeak to announce whatever | ||
was desired based on the information | was desired based on the information that Screen Reader/2 | ||
knows to be on the display. | knows to be on the display. | ||
There are dozens of examples of procedures which are watched | There are dozens of examples of procedures which are watched | ||
by | by autospeaks in the profiles shipped with Screen Reader/2. | ||
These are generally used to search for something, either | These are generally used to search for something, either in | ||
the | the given view, looking for some special indication, or | ||
sometimes in another view. | sometimes in another view. | ||
There | There are important simplifications implied by watched | ||
procedures | procedures also. It is good practice in profile development | ||
to write a procedure to obtain information from the | to write a procedure to obtain information from the display | ||
(from | (from the view) and to use that procedure in a key sequence | ||
(for review) as well. | (for review) as well. | ||
A prototypical use of this idea is checking for a | A prototypical use of this idea is checking for a highlight | ||
bar in a text based application running under the OS/2. This | bar in a text based application running under the OS/2. This | ||
is | is a requirement for working with some DOS applications | ||
running under OS/2, but | running under OS/2, but generally not for the OS/2 GUI | ||
applications. | applications. | ||
Think | Think first of a single row (say row 3) across the top of | ||
the | the display, with menu items on that row which are | ||
highlighted | highlighted by changing the background color when accessed | ||
with F10 (usually) followed by arrow keys. | with F10 (usually) followed by arrow keys. | ||
In this case the procedure (called | In this case the procedure (called action) to be watched checks row three and returns the column of the highlight if there is one and returns zero otherwise. | ||
checks | |||
there is one and returns zero otherwise. | |||
EXAMPLE 3 | EXAMPLE 3 | ||
Line 698: | Line 461: | ||
return place; | return place; | ||
The key command in the procedure above is | The key command in the procedure above is the Get command for finding colors (attributes). This is, in fact the most complex command in PAL. The first argument is a reserved word; you want to find the same or different attributes | ||
for | (sameattr, or diffattr, respectively). The second is argument is the attribute to be compared using the third argument as a mask. The forth argument tells whether to go forward to backward. | ||
complex command in PAL. The first | |||
word; | |||
(sameattr, | |||
argument is the attribute to be | |||
argument | |||
forward to backward. | |||
In our case we are to find the first occurrence of | In our case we are to find the first occurrence of a color which is different from the current attribute (char.color) in the second byte (the background), and go in the forward (+) direction. | ||
which | |||
in the second byte (the background), and go in | |||
(+) direction. | |||
The | The return code (RC) will be set to zero if the search was successful, and non zero otherwise. Therefore the procedure returns the column where the different background was found it it was found, and zero otherwise. | ||
successful, and non zero otherwise. | |||
returns the column where the different background was | |||
it it was found, and zero otherwise. | |||
Now | Now if we watch this procedure with an autospeak it looks like this. | ||
like this. | |||
EXAMPLE 4 | EXAMPLE 4 | ||
Line 729: | Line 479: | ||
endif; | endif; | ||
This | This autospeak results in the procedure action being evaluated periodically, and if action finds a highlighted item, the return value is non-zero and the body of the autospeak will announce the highlighted item because say (filed) speaks from the current position to the end of field, where a field is a contiguous sequence of character positions with the same attributes (color). | ||
evaluated | |||
item, the return value is | |||
autospeak | |||
filed) speaks from the current position to the end of field, | |||
where | |||
positions with the same attributes (color). | |||
The | The application on which I am testing these examples uses the the same highlight for lists of elements inside boxes. When an action item is selected (with enter) the action item is no longer highlighted, and some other item is. | ||
the the same highlight for lists of elements | |||
When an action item is selected (with enter) the action item | |||
is no longer highlighted, and some other item is. | |||
With | With an simple change in the procedure, action, all these highlight items can be announced, For this the wrap switch needs to be used (see EXAMPLE 3) to search the whole view for a highlight. Then only minor modifications are needed in the autospeak as well. The following presents the completed example. | ||
highlight items can be announced, For this the | |||
needs | |||
for a highlight. Then only minor modifications are needed in | |||
the autospeak as well. | |||
example. | |||
EXAMPLE 4 | EXAMPLE 4 | ||
Line 772: | Line 508: | ||
endif; | endif; | ||
===4.2 | ===4.2 AUTOSPEAKS AND EVENTS=== | ||
As I indicated in the preceeding section, the expressions that autospeaks can watch provide a great deal of power over what was available for Screen Reader/DOS. There is another addition for ScreenReader/2. Autospeaks can watch events. | |||
As | Unlike expressions, which must be evaluated periodically, events are generated by Screen Reader/2, or even another process, and those events trigger the execution of the body | ||
that autospeaks can watch provide a great deal of power over | |||
what was available for Screen Reader/DOS. There | |||
addition | |||
Unlike expressions, which must | |||
events | |||
process, and those events trigger the execution of the | |||
of the autospeak. | of the autospeak. | ||
Here | Here is an extremely simple example of an autospeak on EventCursorChange that does a lot of what the edit facility (Section 2.6) does. It certainly does not do all that the edit facility does. | ||
EventCursorChange that does a lot of what the edit | |||
(Section | |||
edit facility does. | |||
EXAMPLE 5 | EXAMPLE 5 | ||
Line 795: | Line 522: | ||
autospeak Edit | autospeak Edit | ||
watch EventCursorChange Do | watch EventCursorChange Do | ||
if r <> crow then | if r <> crow then -- row changed | ||
get ( crow, 1 ); Say ( line ); | get ( crow, 1 ); Say ( line ); | ||
elseif if abs( c - ccol) = 1 then | elseif if abs( c - ccol) = 1 then -- one col change | ||
get ( crow, ccol ); | get ( crow, ccol ); Say ( char ); | ||
else Say ( word ); | else Say ( word ); | ||
endif; | endif; | ||
r := crow; c := ccol; | r := crow; c := ccol; | ||
The idea of this edit autospeak is to say the | The idea of this edit autospeak is to say the line if the cursor line changes; the character if only one column changes, and the word where the cursor lands if the change | ||
cursor | is more than one column. The variables r and c are used to save the previous values of the cursor row and column which are contained in the read-only variables, crow, and ccol, | ||
changes, and the word where the cursor lands if | |||
is | |||
save the previous values of the cursor row and column | |||
are | |||
respectively. | respectively. | ||
The | The following is a list of events available for programmingaccess to OS/2 and its GUI. | ||
EventCursorChange. | EventCursorChange. A cursor being created, deleted, or moved. | ||
moved. | |||
EventDeviceIn. | EventDeviceIn. Input from the synthesizer or brialle device. | ||
device. | |||
EventExternal. | EventExternal. Another process can generate this event. Sample illustrative code is included with ScreenReader/2. | ||
Sample | |||
ScreenReader/2. | |||
EventIndexIn. | EventIndexIn. An index marker returned to Screen Reader/2 causes this event. | ||
Reader/2 causes this event. | |||
EventKeyPressed. | EventKeyPressed. Key pressed on the keyboard. | ||
EventMouseMove. | EventMouseMove. This event occurs when the mouse moves. | ||
EventPointerChange. | EventPointerChange. Whenever the Screen Reader/2 pointer changes, this event occurs. | ||
pointer changes, this event occurs. | |||
EventProcessChange. | EventProcessChange. Whenever the current process changes, this event occurs. | ||
changes, this event occurs. | |||
EventSelectorChange. | EventSelectorChange. A selector being created, deleted, or moved causes this event. | ||
or moved causes this event. | |||
EventShiftChange. | EventShiftChange. A shift key being pressed causes this event. | ||
event. | |||
EventScroll. | EventScroll. When window scrolling is detected. | ||
EventSpinButton. | EventSpinButton. Whenever a spin button control is used to change a value. | ||
to change a value. | |||
EventViewChange. | EventViewChange. This event occurs whenever Screen Reader/2 changes the view. | ||
Reader/2 changes the view. | |||
==ACKNOWLEDGEMENT== | ==ACKNOWLEDGEMENT== | ||
Screen Reader/2 is the result of the efforts of many people. The author has had a project in the Mathematical Science Department of IBM Research for almost 10 years and several people have worked on that project. The Special Needs Organization of the IBM PC Company is the organization responsible for making a research project into a product. | |||
Screen Reader/2 is the result of the efforts of many people. | They took PC SAID and made IBM Screen Reader. Many people have been involved there as well, developers, planners, and writers. Fran Hayden is one of those writers and she has been invaluable in making this paper more readable. Finally there are users. Those individuals both inside and outside IBM provided both the direction and motivation for the Screen Reader/2 effort. | ||
The | |||
Department of IBM Research for almost 10 years | |||
people | |||
Organization of the | |||
responsible | |||
They took PC SAID and made IBM Screen Reader. | |||
have | |||
writers. Fran Hayden is one of those | |||
been invaluable in making this paper more readable. | |||
there | |||
IBM provided both | |||
Screen Reader/2 effort. | |||
==BIBLIOGRAPHY== | ==BIBLIOGRAPHY== | ||
#Adams, F.R., Crepy, H, Jameson, D.H. and Thatcher, J.W., "IBM Products for Persons with Disabilities," IEEE GlobeCom '89. November, 1989. (1990) 496-502. | |||
#Adams, F.R. and McTyre, J.H., "Screen Reader: An Audio Access System," ICAART Conference Proceedings, Montreal June, 1988. | |||
#Boyd, L.H., Boyd, W.L., and Vanderheiden, G.C., "The Graphical User Interface: Crisis, Danger and Opportunity," Journal of Visual Impairment and Blindness 84 (1990) 496-502. | |||
#Emerson, M.R., Jameson, D.H., Pike, G.G., Schwerdtfeger, R.S., and Thatcher, J.W., "Screen Reader/PM - GUI Access by blind computer users," Proceedings, World Congress on Technology, December, 1991. | |||
#"IBM Independence Series Screen Reader, Reference," The IBM Corporation, Amonk, New York, 1989. | |||
#Jameson, D.H., "PC SAID: Computer Access for the Vsually Disabled," M.Sc. Thesis, Trinity College, Dublin,Ireland, 1986. | |||
#Mynatt, E.D., and Weber, G., "NonVisual Presentation of Graphical User Interfaces: Contrasting Two Approaches", Proceedings, CHI '94., April, 1994. | |||
#Schwerdtfeger, R.S., "Making the GUI Talk", BYTE, 118-128 (1991). | |||
#Thatcher, J.W., "How to make OS/2 Talk and Why. Access to OS/2 with Screen Reader/2," To Appear, Proceedings OS/2 World Conference, July, 1994, Santa Clara, CA. | |||
#Thatcher, J.W., "Screen Reader/2 - Access to OS/2 and the Graphical User Interface" Submitted, ASSETS '94 (Sponsonsered by SIGCAPH - Computers And the Physically Handicapped), October, 1994. | |||
[[Category:Screen Reader/2]] | |||
Revision as of 16:58, 24 October 2017
By Jim Thatcher Interaction Technology Mathematical Sciences Department IBM Research Yorktown Heights, NY 19598
Reprint Courtesy of International Business Machines Corporation, © International Business Machines Corporation
Abstract
In this paper I will describe the Screen Reader/2 personality, that is the 'look and feel' of Screen Reader/2. As I do that I hope to give the reader an introduction to the Profile Access Language (PAL), which is one of the most important components of Screen Reader/2. This PAL introduction will not be formal, in the sense of careful syntax and semantics; I want to get across the idea of PAL so that one might want to program in the Profile Access Language to access the Graphical User Interface (GUI) of OS/2 and of Windows applications running under OS/2.
1.0 INTRODUCTION
IBM Screen Reader/2 is the only access system that gives blind computer users access to OS/2, DOS and Windows applications (running seamlessly). It is the only screen reader to have a fully programmable interface. Screen Reader/2 has grown over nine years in response to the demands of users in IBM and outside who needed the function for their work, education and home environments.
2.0 BASIC SCREEN READER/2
Screen Reader/2 for OS/2 is a follow-on to the DOS based product that was released in January of 1988 and upgraded two times since. Two things about IBM Screen Reader distinguish it from all other screen access software; the separate 18 key keypad and its total programmability. On the latter count, the power of the programming language, PAL, is a significant distinguishing feature.
2.1 THE SCREEN READER/2 PACKAGE
Screen Reader/2 for OS/2 consists of an 18 key keypad, a keypad cable, some audio cassettes, some hard copy documents, and two diskettes with on line documentation and software. The keypad is attached through the mouse port. If such is not available, there is an adapter card for the keypad that can be used instead.
The software can be grouped into roughly five sections.
- Screen Reader executables and dynamic link libraries.
- Configuration files and utilities.
- Installation utilities.
- The PAL compiler and profiles.
- On line documentation.
The keypad is used as an input device to Screen Reader/2 to request information from the display or to change settings in the speech environment. There are literally hundreds of states that effect how Screen Reader/2 responds.
Screen Reader/2 responds or speaks as the result of a user request through the keypad or because of some autospeak. In both cases the nature of the response is determined by the current Screen Reader/2 state and the PAL profile that is active. The important point here is that everything that Screen Reader/2 does is a consequence of some profile code written in the Profile Access Language.
2.2 THE ROLE OF PROFILES
Profiles both define the basic screen reading environment, the Screen Reader/2 personality, and they are also used as add-ons to tailor that basic environment to specific applications when that is desirable or even necessary.
This aspect of Screen Reader is often misunderstood. The application specific profiles are developed for making applications easier to use. Everything that is done in application specific profiles could be done without them (probably), but with more effort (probably a lot more effort).
The simplest example of the desirability of an application specific profile is when the application has a status area in which, say, error messages are displayed. Using the keypad and standard core requests, the Screen Reader/2 user can read that area to find out about any messages without a special profile. With an application specific profile, however, that area can be monitored and automatically read. In addition, a simple application specific key sequence could be defined to read the status area.
There are about 30 applications specifically supported by the Screen Reader/2 product. These profiles are supplied for OS/2, DOS, and Windows applications.
Whether considering application profiles or the base Screen Reader profile, it is crucial also to understand that Screen Reader's behavior is completely determined by the current active profile or profiles.
2.3 SESSIONS
There is one base or core profile, called srd2.kpd. This profile textually includes the sub-profiles for the edit facility, mouse movement, icons, scanning, etc. These are all conceptually and syntactically part of the base profile.
The way this base profile works is that it is loaded and active when Screen Reader/2 is started. This base profile and any environment settings that might be determined by it, define what we call default Screen Reader/2 session.
Whenever a new profile is added to (we say pushed on) the base profile, i.e., the default session, that action creates a new Screen Reader session associated with the foreground process. You can save settings in that session, you can push another profiles for that session. When the process is killed the session goes away.
To illustrate this session concept, just imagine that there are two sessions, the default session and a new session created by pushing a profile for, say, the System Editor. Then you could be in spell format (where everything is spelled) in the System Editor session, and text format (normal reading) in the default session. Then any time you switched from the default session to the system editor session, you would switch from text to spell format and conversely.
2.4 THE NATURE OF PROFILES
A simple example will help to illustrate the use of PAL and the idea that any Screen Reader/2 response can occur as a result of a keypad entry or some event that triggers an autospeak.
EXAMPLE 1
include 'srd2.inc' Var TimeWatch: AutoHandle; {1} Msg ( 'it is ', DateTime( DT_HOUR ), ' o clock' ); AutoSpeak TimeWatch Watch DateTime( DT_HOUR ) Do Msg ( 'it is ', trigger , ' o clock' );
The file srd2.inc has definitions of constants like DT_HOUR. The third line of PAL code above gives meaning to the 1 key, {1}, of the screen Reader Keypad. When this fragment is active and the 1 key is depressed, the message 'it is 9 o clock' is spoken. This line of PAL Code is executed when the 1 key is pressed.
On the other hand, the autospeak with handle TimeWatch, watches the value returned by the function DateTime(DT_HOUR) which, the reader might have guessed, is the current hour. When that changes, then the body of the autospeak is executed. That body is also a message statement (Msg sends its arguments to the attached serial device). The value of the reserved variable, trigger, is the value that triggered the autospeak, in this case the current hour. So at the moment the hour changes to 9, the message 'it is 9 o clock' is announced, just like pressing key 1.
2.5 BASIC REVIEW REQUESTS
Screen Reader has many standard review requests programmed into its base profile (os2core.kpd, included in srd2.kpd mentioned above). Each of these requests is tempered by the Mode and Format setting.
There are two reading 'modes,' called cursor and pointer. In cursor mode all reading requests are relative to the application cursor; in pointer mode all requests are relative to Screen Reader/2's own 'pointer' (called the review cursor by some screen reading packages). Because other screen readers do it differently, it is important to emphasize that there are many reading requests and all of them work in either pointer or cursor mode. The request is the same sequence on the Screen Reader keypad, like {2} for current line. Depending on the current mode, that will be the cursor line, or the line containing the Screen Reader/2 pointer.
There are five reading formats called text, pronounce, spell, phonetic, and ASCII. In text format Screen Reader tries to read as if reading a book. In pronounce format, punctuation is also announced. In spell format, all words are spelled and in phonetic format everything is spelled using the international phonetic alphabet. In ASCII format you hear the ASCII value of each character.
Again, I want to emphasize that all the basic requests are available and the result of each depends on the current format setting. There are shortcuts, however. For example, it is unlikely you really want ASCII format, so a key sequence is defined to give the ASCII value of the current character, and none to switch to ASCII format. Similarly, another sequence is defined to spell the current word without changing the active format.
Here is a list of some basic review requests.
- Read the entire window or read from current position to the end.
- Read the current, previous, next sentence, line, word, or character.
- Move pointer to cursor, to top left, to bottom left, or to right edge.
- Spell the current, previous or next word.
- ASCII value of current character.
- Current character in Phonetic alphabet.
- Search for a string, search again, or search from bottom.
This list could be expanded, but such detail becomes tedious. There is a separate, though related, list which is important in portraying the Screen Reader/2 personality. As I mentioned above there are many components of the Screen Reader/2 state that can be set by the user, and in fact, can be different in different Screen Reader/2 sessions. The ways these state components are set is not so important as is the following illustrative (certainly not exhaustive) list.
- Mode cursor or pointer.
- Format: text, pronounce, spell or phonetic.
- Ignore capitalization or not.
- Announce spaces or not.
- Treat entire screen as single line (wrap) or not.
- Announce line numbers while reading or not.
- Use the dictionary or not.
The following are GUI specific.
- Hear drawing noise or not.
- Include icons or not.
Having just listed a sampling of Screen Reader/2 core requests, now I want to give a sample how some of these appear in PAL. Remember that we are not detailing the syntax and semantics of PAL. I believe that the examples will, for the most part, be self explanatory. The following examples are exactly as they appear in the profile, os2core.kpd, shipped with Screen Reader/2.
EXAMPLE 2
{0A} 'read entire screen' Save( Wrap); Set( Wrap, on); Get(1, 1); Say(line); {1} 'previous line' Get( row-1, 1); Say( Line); {6} 'next word' Get( Nextword); Say( Word); {*C} 'Ascii value of current character' Save( Format); Set( Format, Ascii); Say( Char);
A key definition in PAL is a key or key sequence in braces, followed by an optional comment used for help. After that is a sequence of PAL commands.
Only four Screen Reader/2 commands are used in the key definitions in example above. The Save command saves one or more components of the Screen Reader/2 state. Any saved component is restored automatically at the end of the command sequence.
The Set command sets a component of the state. In the statement sequence for {0A}, Wrap is set to on which means the entire view is treated as one line. The format is set to ASCII in the command sequence for {*C}.
The Get command positions a local pointer to a position specified by its arguments. When a command sequence is started, the local pointer is assigned the cursor position if in cursor mode, and the pointer position if in pointer mode. The local pointer's coordinates are always given in two read-only variables, row and col. The local pointer gets updated throughout a command sequence and is assigned to the
Screen Reader/2 pointer at the end of the sequence if in pointer mode.
The Say command is used to send text from the display to the synthesizer (or other device). Depending on its argument, Say works on a line (or whole screen if wrap is on), word or character.
2.6 EDITING
The Screen Reader/2 edit facility provides for feedback as the user moves the cursor and types. The edit facility is in fact another profile, and a rather complex profile at that. I don't want to explore the design of that profile here. (Neither do I want to discuss the question of whether or not the edit facility functions should or should not be built in at a lower level in Screen Reader/2.)
Instead I will mention the 5 major feedback options in the edit facility.
Line Browse.
With Line browse, every time the cursor moves one position to a new character, that character is spoken; when the cursor moves horizontally a word at a time, the word is spoken, and when the cursor changes row or the line changes as in a scroll, the new line is spoken.
Flush.
With flush turned on, each new edit facility response cuts of (stops) the previous response,
The Default edit settings are line browse and flush. In addition there are three 'echo' settings.
Character echo.
Echo each character as typed.
Word echo.
Echo changed words as the space is typed at the end of the word.
Line echo.
Echo changed lines when move to new line.
There other options in the edit facility that can make life easier under some circumstances, including a margin indicator and a column browser.
3.0 NAVIGATING AROUND THE GRAPHICAL USER INTERFACE
Almost all OS/2 and Windows applications respond to keyboard input as a substitute for mouse input. As a simple example, the window list (the list of running programs) is displayed using the right mouse button on the desktop, or using the keyboard combination Ctrl+Esc. One can highlight an item in a list by using the mouse and single click, or by moving to the item with the arrow keys. The item is invoked (or opened) using a double click with the mouse or by highlighting the item and pressing enter.
In general, a blind user navigates around the GUI with the keyboard. During that navigation, Screen Reader/2 provides constant feedback.
3.1 THE CONCEPT OF VIEW
Under the default settings, Screen Reader/2 always restricts reading to what it determines to be the main window and all text and icons contained in that main window. Because the word 'window' is over used, we call that rectangle the Screen Reader/2 view. As the user switches between applications, or between main windows, Screen Reader/2 automatically adjusts the view to the current main window.
Besides adjusting the view; Screen Reader/2 automatically announces the window title when the main window or application changes.
Once the view has been established, the view is treated very much as if it were a text window. All the text in that view is sorted by baseline into a certain number (view.rows) of distinct rows and each row is assumed to have the same number (view.cols) of 'columns' namely the maximum character length of the rows.
We believe that this simplifies access to the graphical view, though it is not always an accurate reflection of what is displayed. Our experience has been, with many users of Screen Reader/2, that this filtering of the displayed data is a help for review and that no significant information is lost.
The characters in the first 'column' cannot be expected to be seen to be lined up vertically as they are in a text screen. There is no implication about vertical positioning even of different rows. One 'row' might be huge and take up most of the view, with the remaining rows crammed in a small font at the bottom of the view, like, say, footnotes. One block of text could have a baseline just above a block to its right. These would be considered distinct rows.
Two blocks of text could be the same font, size and style and have the same baseline and still be on different 'rows.' This happens when the text occurs in separate windows. And the reason we put text in different windows into distinct rows is because, almost certainly, one wants to be read the text of those windows separately.
All the information that is ignored by this row-column access of the view is available to the user. This includes:
- The pixel position of each character and its size (char.left, char.bottom, char.right, char.top).
- The font, pitch, style and color of each character. (char.font, char.pitch, char.style, char.color).
- The pixel size of the view and its position on the desktop. (view.top, view.left, view.bottom, view.right).
- The identity of the window containing the text, the window handle By calculations in PAL (we have not found reason to use these in basic profiles).
- Whether one row or column truly lines up below another.
- The amount of white space between rows.
- The amount of white space at the beginning and end of rows.
The reason the view is set as it is by Screen Reader/2 is that conventional review requests would be utterly confusing without it. Successive lines or blocks of text might come from different main windows and make no sense at all. As with most function and default operation of Screen Reader/2, the user can have it both ways. With standard requests, the user can make the whole display (or desktop) the view, or only the contents of a single push button.
Views are determined by the windows. A view always includes all child windows contained within a specified window. Thus the view determined by the desktop window includes everything displayed; the view determined by a push button is just the button text or icon. Pushbuttons are windows in the technical sense (except in the case of some Windows applications written by MicroSoft).
Views are not the only way of restricting reading. Viewports are rectangular regions within a view, to which reading can be constrained.
3.2 USING CONTROLS WITHIN A VIEW
A dialog is one kind of main window that does not fit well into the row and column format described above. Dialogs typically consist of several windows (in the technical sense, again) which allow the user to enter data and make choices. The individual windows (buttons, entry fields, static text fields, check boxes) are called controls. The user makes choices with the buttons, entry fields and check boxes; the prompts or headings are provided with by the 'static text.'
A sample will help. In the OS/2 Enhanced editor, a search dialog appears when you want to search for and possibly replace text in a file you are editing. At the top of the dialog are two entry fields. The first has the prompt (or name) 'search' and the second has the prompt 'replace.' You type in the string you want to search for in the first entry field and press enter.
These two entry fields and their prompts actually comprise four windows in this view, and four rows as well. As you move to them, however, with the standard tab-key movement, you hear 'search entry blank' and 'replace entry blank' respectively. When the data is put in the entry field, then the editing functions of Screen Reader/2 can be used.
Because of the feedback from Screen Reader/2 to the user, there is little interest in what rows and columns are involved, or the fact that the sighted user might say that the prompt and the entry field data appear to be on the same 'row.'
Continuing with this example, below the entry fields is a box (looks like a window and is called a group box) with a heading 'Options.' That box contains five check box controls which consist of small squares (about one-quarter inch) which may or may not have check marks in them. Each has text immediately to the right of the small square. These are used to select options such as 'ignore case,' or 'change all occurrences.'
As the user tabs into this area the and arrows around the check boxes, Screen Reader/2 responds with 'Group box options check box ignore case checked' or 'Group box options check box change all occurrences not checked' It is standard in the GUI to toggle the check state of a check box with the space bar. Hitting the space bar when on the ignore case check box, results in the announcement 'Group box options check box ignore case not checked' from Screen Reader/2.
Finally, at the bottom of this dialog is a group of six push buttons, things like 'Find,' 'Change,' and 'Cancel.' When
the user tabs to this group of buttons, and arrows around there, the information, is presented in a form similar to that for check boxes: 'push button find' 'push button cancel'
3.3 MOUSE ACTIONS AND OTHER TRICKS IN SCREEN READER/2.
There is no doubt that not using a mouse with the graphical user interface is a disadvantage. It is quicker to do some things with a mouse than with the keyboard equivalents (when those keyboard equivalents are known). The worst part of this is probably the fact that help panels are written with the mouse user in mind. They often do not include the the keyboard alternatives.
Screen Reader/2 has done some things to make this situation better. One of the most important features is our switch list.
A mouse user brings a window to the foreground by moving the mouse to any part of that window and clicking with the left mouse button. If that window is totally obscured, either other windows must be moved, resized, minimized, or closed; or the mouse user can use the keyboard equivalent if they know what it is.
From the keyboard, Ctrl+Esc brings up the list of open programs. Then the Screen Reader/2 user can scan that list one at a time, or pres the first letter of the title, until the item is found and then open the folder or start the program by pressing Enter. This is a significantly longer activity than moving the mouse and clicking.
The Screen Reader/2 user can identify up to 15 windows (in the default case) and when they want to switch to one of their favorite windows (whether obscured or not) they just enter the number of the window preceded by the simple chord 89 on the keypad. Movement between a collection of standard windows becomes easier than the same task for the (sighted) mouse user.
The OS/2 2.1 desktop covers the entire display. On it are icons for programs and program folders and other objects like printers. To open (start) one of these objects the mouse user must find it, and double click. Other windows that are open are very likely to obscure one or more of these icons, and the only option here for the mouse user is to close, move, resize, or minimize some of the open windows. This is often a problem for the sighted user, but it would be a disaster for the blind user.
Screen Reader/2 solves this problem with an alternative desktop which is a normal folder object and can be brought
to the foreground just like other folders. In particular the Screen Reader/2 switch list can be used, to make the desktop the current view.
As the last subject in this section, mouse actions (left and right button, click and double click) can be preformed with Screen Reader/2 at any time.
Think of the Search Dialog for the Enhanced Editor described in the previous section. As we discussed there, the normal use of that dialog is to tab or arrow key to the required spot, enter data, or change options with the space bar, and then press enter to carry out the action. An alternative is available for any application.
Using a couple of basic Screen Reader/2 requests from the keypad, the blind user can perform a mouse single click (chord 23 then 4) or double click (chord 23 then 5). So when the Screen Reader/2 user reviews the contents of dialog and is reading, for example, the 'ignore case" checkbox, a single click (using the keypad) will change the check state of the check box from checked to non-checked or conversely.
4.0 SOME TECHNICAL CONSTRUCTS IN PAL
In this last section I will describe a couple of the unique and powerful features of the Profile Access Language. Continuing in the spirit set in the first section, I will not stress the detailed syntax of PAL, but instead illustrate the ideas with PAL code and descriptive text.
4.1 AUTOSPEAKS AND PROCEDURES
In the first section, EXAMPLE 1 showed how an autospeak could be defined to announce the time when the hour changed. The watched expression in that case was a function (DateTime) which returns the current hour.
This idea, watching a function or procedure, is key to a very powerful concept for Screen Reader/2; but some history first.
Ten years ago, when Screen Reader was a research project called PC SAID, we did have programmable autospeaks, but only to watch specified screen positions, and somewhat later, numeric expressions. In its early days, this was far ahead of anything else that was available or planned in the screen access area.
Lets review how the autospeak construct works. The syntax (roughly) is:
AUTOSPEAK SYNTAX AutoSpeak <autohandle> watch <expression> do <Command Sequence>
The autohandle is a variable needed to query the state of the autospeak, and to turn it on or off. It is not important for understanding this discussion. The expression is just that, any expression in PAL. So you can watch a part of the display as in the original PC SAID autospeaks by watching, for example,
Display( 23, 67, 4)
to watch row 23, column 67 for 4 characters.
Then Command Sequence is any PAL sequence of commands.
The way this works is that the expression is evaluated periodically, say every tenth of a second. If that value changes (and stays changed for some specified time -- a detail that won't bother us) then the autospeak body (the
Command Sequence) is executed. As indicated in the time example in Section 2.4. there is a convenient system variable, trigger, which contains the value of the expression that triggered the autospeak.
All of this is pretty much the same as has been available since the days of PC SAID; the syntax has been cleaned up and there are some conveniences. But what is truly new here is the fact that PAL for Screen Reader/2 has user-defined procedures that return values, and, of course, these can be parts of expressions. This is a major functional change, significantly increasing the power of the autospeak feature.
Why is this powerful? Because in principle it makes the autospeak construct universal, that is, one could, in principle, design and code an autospeak to announce whatever was desired based on the information that Screen Reader/2 knows to be on the display.
There are dozens of examples of procedures which are watched by autospeaks in the profiles shipped with Screen Reader/2. These are generally used to search for something, either in the given view, looking for some special indication, or sometimes in another view.
There are important simplifications implied by watched procedures also. It is good practice in profile development to write a procedure to obtain information from the display (from the view) and to use that procedure in a key sequence (for review) as well.
A prototypical use of this idea is checking for a highlight bar in a text based application running under the OS/2. This is a requirement for working with some DOS applications running under OS/2, but generally not for the OS/2 GUI applications.
Think first of a single row (say row 3) across the top of the display, with menu items on that row which are highlighted by changing the background color when accessed with F10 (usually) followed by arrow keys.
In this case the procedure (called action) to be watched checks row three and returns the column of the highlight if there is one and returns zero otherwise.
EXAMPLE 3
proc action returning integer is var place : integer; get ( 3, 1 ); Get( diffattr, (char.color bitand &hff00), &hff00, +); if rc = 0 then place := col; else place := 0; endif; return place;
The key command in the procedure above is the Get command for finding colors (attributes). This is, in fact the most complex command in PAL. The first argument is a reserved word; you want to find the same or different attributes (sameattr, or diffattr, respectively). The second is argument is the attribute to be compared using the third argument as a mask. The forth argument tells whether to go forward to backward.
In our case we are to find the first occurrence of a color which is different from the current attribute (char.color) in the second byte (the background), and go in the forward (+) direction.
The return code (RC) will be set to zero if the search was successful, and non zero otherwise. Therefore the procedure returns the column where the different background was found it it was found, and zero otherwise.
Now if we watch this procedure with an autospeak it looks like this.
EXAMPLE 4
var a: autohandle; autospeak a watch action do if trigger <> 0 then get( 3, trigger); say( field); endif;
This autospeak results in the procedure action being evaluated periodically, and if action finds a highlighted item, the return value is non-zero and the body of the autospeak will announce the highlighted item because say (filed) speaks from the current position to the end of field, where a field is a contiguous sequence of character positions with the same attributes (color).
The application on which I am testing these examples uses the the same highlight for lists of elements inside boxes. When an action item is selected (with enter) the action item is no longer highlighted, and some other item is.
With an simple change in the procedure, action, all these highlight items can be announced, For this the wrap switch needs to be used (see EXAMPLE 3) to search the whole view for a highlight. Then only minor modifications are needed in the autospeak as well. The following presents the completed example.
EXAMPLE 4
var r, c: integer; proc action returning integer is var place : integer; Save ( wrap ); set ( wrap, on ); get ( 3, 1 ); Get ( diffattr, ( char.color bitand &hff00), &hff00, +); if rc = 0 Then c := col; r:=row; place := row*25+col; else place := 0; endif; return place; var a: autohandle; autospeak a watch action do if trigger <> 0 then get ( r, c ); say ( field ); endif;
4.2 AUTOSPEAKS AND EVENTS
As I indicated in the preceeding section, the expressions that autospeaks can watch provide a great deal of power over what was available for Screen Reader/DOS. There is another addition for ScreenReader/2. Autospeaks can watch events. Unlike expressions, which must be evaluated periodically, events are generated by Screen Reader/2, or even another process, and those events trigger the execution of the body of the autospeak.
Here is an extremely simple example of an autospeak on EventCursorChange that does a lot of what the edit facility (Section 2.6) does. It certainly does not do all that the edit facility does.
EXAMPLE 5
var Edit : autohandle; var r, c : integer; autospeak Edit watch EventCursorChange Do if r <> crow then -- row changed get ( crow, 1 ); Say ( line ); elseif if abs( c - ccol) = 1 then -- one col change get ( crow, ccol ); Say ( char ); else Say ( word ); endif; r := crow; c := ccol;
The idea of this edit autospeak is to say the line if the cursor line changes; the character if only one column changes, and the word where the cursor lands if the change is more than one column. The variables r and c are used to save the previous values of the cursor row and column which are contained in the read-only variables, crow, and ccol, respectively.
The following is a list of events available for programmingaccess to OS/2 and its GUI.
EventCursorChange. A cursor being created, deleted, or moved.
EventDeviceIn. Input from the synthesizer or brialle device.
EventExternal. Another process can generate this event. Sample illustrative code is included with ScreenReader/2.
EventIndexIn. An index marker returned to Screen Reader/2 causes this event.
EventKeyPressed. Key pressed on the keyboard.
EventMouseMove. This event occurs when the mouse moves.
EventPointerChange. Whenever the Screen Reader/2 pointer changes, this event occurs.
EventProcessChange. Whenever the current process changes, this event occurs.
EventSelectorChange. A selector being created, deleted, or moved causes this event.
EventShiftChange. A shift key being pressed causes this event.
EventScroll. When window scrolling is detected.
EventSpinButton. Whenever a spin button control is used to change a value.
EventViewChange. This event occurs whenever Screen Reader/2 changes the view.
ACKNOWLEDGEMENT
Screen Reader/2 is the result of the efforts of many people. The author has had a project in the Mathematical Science Department of IBM Research for almost 10 years and several people have worked on that project. The Special Needs Organization of the IBM PC Company is the organization responsible for making a research project into a product. They took PC SAID and made IBM Screen Reader. Many people have been involved there as well, developers, planners, and writers. Fran Hayden is one of those writers and she has been invaluable in making this paper more readable. Finally there are users. Those individuals both inside and outside IBM provided both the direction and motivation for the Screen Reader/2 effort.
BIBLIOGRAPHY
- Adams, F.R., Crepy, H, Jameson, D.H. and Thatcher, J.W., "IBM Products for Persons with Disabilities," IEEE GlobeCom '89. November, 1989. (1990) 496-502.
- Adams, F.R. and McTyre, J.H., "Screen Reader: An Audio Access System," ICAART Conference Proceedings, Montreal June, 1988.
- Boyd, L.H., Boyd, W.L., and Vanderheiden, G.C., "The Graphical User Interface: Crisis, Danger and Opportunity," Journal of Visual Impairment and Blindness 84 (1990) 496-502.
- Emerson, M.R., Jameson, D.H., Pike, G.G., Schwerdtfeger, R.S., and Thatcher, J.W., "Screen Reader/PM - GUI Access by blind computer users," Proceedings, World Congress on Technology, December, 1991.
- "IBM Independence Series Screen Reader, Reference," The IBM Corporation, Amonk, New York, 1989.
- Jameson, D.H., "PC SAID: Computer Access for the Vsually Disabled," M.Sc. Thesis, Trinity College, Dublin,Ireland, 1986.
- Mynatt, E.D., and Weber, G., "NonVisual Presentation of Graphical User Interfaces: Contrasting Two Approaches", Proceedings, CHI '94., April, 1994.
- Schwerdtfeger, R.S., "Making the GUI Talk", BYTE, 118-128 (1991).
- Thatcher, J.W., "How to make OS/2 Talk and Why. Access to OS/2 with Screen Reader/2," To Appear, Proceedings OS/2 World Conference, July, 1994, Santa Clara, CA.
- Thatcher, J.W., "Screen Reader/2 - Access to OS/2 and the Graphical User Interface" Submitted, ASSETS '94 (Sponsonsered by SIGCAPH - Computers And the Physically Handicapped), October, 1994.