Making Java Applets More Accessible:

AWT Additions -Recommendations

1. What object properties need to be exposed?
2. How to expose object information to the operating system and make sure it is passed along to the screen reader Off Screen Model (OSM)?
3. When is the AWT not using system tools and how does this affect screen reader behavior and recognition of objects?

1. There should be an object type or class which allows the grouping of multiple graphic objects.

2. Objects that are drawn without system calls need to expose (at least) type, state, name and description.

3. Expose the digital state of an analog object.

4. Expose the descriptions of applet objects and actions.

5. Allow browser preferences to control Applet displays.

6. Open issue - Allow screen updating to be suspended for user analysis and response.

7. Open issue - Focus/Blur problems with Applets and parts of Applets.

8. Open issue - Navigating applets.

Multiple calls to drawString() (e.g. curve, line...) or getImage() should be able to be grouped by the developer as a graphic object. The changes caused by the AWT could be collected/reflected in the client OS, such as the Microsoft Windows Group Box window class or a special Java windows class which screen readers would know to look for. When viewing the balloon applet with a screen reader (Outspoken on Netscape in Win95) text is properly recognized but everything else is a generic image. In the Thermometer above the pressure gauge, the temperture text and different parts of the thermometer are recognized as separate objects by the screen reader. The relationships between the parts of the thermometer is totally obscure to the screen reader. Identifying the group of objects in the the thermometer as a single unit , i.e. thermometer, would expose the nature, state and descriptive attributes to the user. This assumes this information is first exposed to the operating system and provided by the developer. The screen reader would encounter one composite graphic object instead of all the parts of the graphic object. The ability to encounter all of the subparts of a composite image object should be retained. There could be multiple levels of hierarchy. An example of this is a toolbar made of individual images. The user could encounter a grouped image called 'toolbar', and if they wished, encounter each individual image in the toolbar.

This assumes the system can accept this information, the developer provides it, and the Java AWT can associate these attributes with an object.

The off screen model (OSM) created by screen readers records calls to the system library. Included in this model is a dictionary of the expected calls and how to interpret them. Thus, an object placed on the screen as a result of a library call is identified. Other information, such as the name of a button, may be retrievable from the screen reader OSM. The quality or nature of this model may vary between operating systems. Objects not present in the OSM dictionary (e.g. the Java component Scrollbar- not to be confused with the textarea scrollbar) and created by other means (e.g. draw an image) than a call to the system library are not identifiable by the screen reader. Screen readers know to look for the creation of bitmaps. If the creation of each graphic object involved an in-place BitBlt and the generation of a bitmap, the screen reader could recognize this bitmap and the attributes associated with this bitmap (e.g. state, name, and description).

The system calls which are associated with the Button and Checkbox components (and the state of the Checkbox) allow them to be recognized correctly by the screen reader. The Scrollbar component (not the scrollbar associated with a textarea) is recognized simply as a graphic. If an image is made to act as a button, information should be provided so the screen reader recognizes the image as a button rather than as a simple graphic. The information which identifies a Button object as a Button to the Screen Reader could be appended to the Graphic object so it is also identified as a Button. Object attributes (type, state, name, description) need to be generically passed to the operating system and incorporated into the OSM. Correctly identifying objects should be extended to custom Java AWT components, and to components accessible to Java Applets such as ActiveX or OpenDoc components.

Possibly the java notion of interface, could be used (extends Graphics implements Button) to make a non-button object to appear as a button to the screen reader. Though the graphic could access Button methods, the graphic was created by the graphic creation method, so should still appear as an image to the screen reader. Other information about the Graphic acting as a button could be presented in descriptor attributes or name.

Text presented as a Label is not identified as a Label, but simply read as text. This might be a problem depending on the function of the Label, to act as simple text or to act as a Label.

4. Expose the descriptions of applet objects and actions.

This assumes the system can accept this information, the developer provides its, and the Java AWT can associate these attributes with an object.

A descriptive attribute, like ALT TEXT for images in HTML, would be associated with an applet and the applet's objects. Accessing the NAME of an object (button, textfield, image) could act as a simplified description. The description could be textual (and then translated to speech), providing alternate means of communicating information. User preferences or actions would access the descriptive attribute. The descriptive attribute could be presented in dialog boxes or accessed via menus. The descriptor could replace the object, like ALT TEXT in HTML/Netscape, while still letting the object function. Descriptive attribute presentation could be determined by user choice or browser preferences. Descriptive attributes could be associated with

1. the applets on a html page.
2. the applet layout and function.
3. the objects in an applet (i.e. images, buttons, audio clips, URL links, etc.)
4. the relationships between objects in an applet, functional and spatial.
5. the occurrence of events and actions (i.e. color or text changes, sound events, etc.)

As the user moves about the physical layout and encounters components, descriptions of components would be available. The user/browser flags could determine the number of descriptive elements presented, (name of component, function of component, spatial relation of component to other components, etc.). The component would still be usable when the description mode is present. The description text would immediately show up in a manner accessible to screen readers or in other modes of presentation (eg. audio). These descriptions could be available to all users. A text windows which appear when a component is encountered need to remain after the screen reader focus is removed from the component and shifted to the text window.

Descriptors could describe what panels are present in a frame or window, describe the components (Button, TextField, TextArea etc.) present in a panel, as well as the text written on buttons and Labels present and what component they are spatially associated with. The relation of Labels to other components may need to be defined by the programmer, but could be generated by a layout description method. Canvases containing draw text or objects need also be described. Some simple descriptions (canvas contains "text", three circles, etc.) might be generated automatically. (Also see the "layout speed list".)

---

Graphics.
An image alt text or a descriptor is needed to describe the image object, its position/action/movement in relation to other objects and how to do an event triggering action. The If an image is acting as text or as a button, its contents, properties would be described. The description could be provided by the programmer, or if the relationships are simple, the layout relationships could be used to create a unique sentence describing the situation. If a complex object is created using a method or class (i.e. class car method draw car), a descriptor tag creation method could be associated with the method/class. The descriptor tag could adjust to the instance of the method expression (red car, green car, car with no bumper). Information would be lost to some users if an object is created that is unique and unpredictable, where an appropriate descriptor could not be provided in advance.

Image descriptor attributes would be part of the image file/descriptor class, accessible when ever an image is present. Descriptors should also be available for background tiled images. Layering of images and interaction of images could be confusing in textual/aural descriptions. To reduce confusion, a simplified alt view might be optionally presented. The simplified alt view could consist only of those images which communicate essential information. Background and purely decorative images would not be described, unless requested. The descriptor field of an image would be accessible whether the image is present or not (image presence could be needed for applet function). Could an image be "present" for the function of the applet, yet not shown be on the screen?

---

Audio Samples and Clips.
Audio clips descriptor attributes could present the words in the audio clip as text or provide textual desscriptions of the audio sample. The description should be accessible without having to "play" the audio clip.

URL Links.
A url link can be associated with any location or object and an action regarding that object or location. There is no standard indicator of url links in Java Applets as there is in present HTML documents (highlighted or underlined text or images). A descriptive speed list of the url links could be an easy way to find and access url links (AppletContext class showDocument(url)). The list of URL links could be presented to a screen reader like the browser Lynx present URL links in HTML. The object, location or action which activates the url link and the target of the url link could be described and accessed in the listing. The presence of the url link could also be announced/vocalized as encountered in moving about the applet. Since Java Applets are dynamic, the active status of the url links could change.

---

the relationships between objects in an applet, functional and spatial

The locational information present visually with the Label component in relation to other components, or a series of Button bounded on a Panel are lost to the screen reader, but could be included as a descriptive attribute of the Label.

The actions associated with Buttons, Scrollbars and other components could be described.

---

the occurrence of events and actions

Occurrence of Visual Events.
Occurrence of background visual events need to be cued by sound (non-verbal or verbal) or descriptive text (screen reader text/verbalize). If the page text is rewritten the user could be notified by a dialog box. The user could be alerted by a aural or a textual/screen reader cue to the temporal occurrence of a Dialog box, FileDialog box, and Exceptions. If information is presented in an analog manner, it should also be presented in a digital (accessible to screen readers) manner and the user notified when the information changes. The Ballon Applet contains a pressure gauge which changes over time. The analog information of the gauge is also presented as a TextField, but this pressure Textfield is unlabeled and has to be reread to detect change. Increasing pressure could be cued by a sound of rising pitch.

Methods are needed to verbalize a component (scrollbar, choice), provide the name, provide a description and provide the state or value in an alternate media (scrollbar.getValue()). Analog information could be presented digitally. The CheckBox state is accessible to the screen reader, but the value of a scrollbar is not accessible.

---

Color Communicating Information.
Color alone should not be used to communicate essential information. If color is the communication "event", a descriptor, created by the programmer, accessible to other medium should be available. The color isn't the object, the color's communication of information/ event is the object which would have the associated descriptive alert/tag.

---

Text Characteristics Communicating Information.
Text size, color and quality can communicate information. If the font, font size, color of text, and changes in these qualities communicates information, the information should be accessible to the user by alternate means. The existing conditions could be given in the applet/component descriptive attribute. The changing of these qualities could be indicated by a pop up alert or verbalization.

---

Sound.
Occurance of sound events need to be cued/described by nonaural methods (e.g. visual or textual). Sound alone should not be used to communicate essential information. If sound is the communication "event", a descriptor field accessible to other medium should be available. An alternate or added to form of the audio file/movie could be available to the user, providing a description of the sound or in line text (accessible to screen readers).

User preferences defined in the browser should be accessible to the applets. Preference could include alternate communication media, color, text characteristics, no image, descriptive attribute display, and screen update control.

---

Color.
The user should be able to adjust the color scheme to reduce display complexity or increase contrast. A high contrast default , light and dark colors which aren't perceived as hues of the same color, will allow things to be distinguishable even if the user is color blind. The color scheme choice could be made (by the user or by browser preferences for all applets) before starting the applet by the param sent to the applet, or adjusted by the user in the applet.[g.setColor(color.red)]

---

Text Size.
Text size/quality can impact accessibility by affecting readability and by communicating information. To make text more readable, the user/preferences should be able to adjust text size and access the system default settings. Presently browser text font/size preferences do not affect applets. The text of components [Label, CheckBox, Button, Dialog Box, Menu, Choice, TextArea or TextField] and text of the graphics drawString() have methods for altering text and providing information to make the altered font work in the format [setFont(), FontMetrics class, setMaxAscent(), setMaxDescent(), stringWidth()]. It would be nice if text characteristics could be specified by the users browser preferences.

Increasing the text size may create formatting problems. Thresholds of font size changes could be defined where layout wouldn't have problems. Increasing the font size as the applet is loaded may be easier on formatting problems than changing the font size after the applet is loaded. If the layout has to change to accommodate a font size change, increasing the window size (added scroll bars?) might help. Possibly, the preferences font size of the users browser could derive screen dimensions and access resize(), to change the applet's window size as the applet loads or as the user changes the font/font size. The text drawing starting point could be related to the window size. The proportional increase in window size could be used to define a proportional new text drawing starting point and proportional increase in text font size.

---

Alternate Media.
In the future, nonaccessible information could be presented in alternate format, i.e. video could be presented with captions or text could be presented audibly.

As a screen reader works its way down a page, a constantly updated text stream may not be relevant to material read earlier on the page. Suspending screen updates until the user reinitiates changes might be done using threads. All the threads, present in different applets which are changing the screen have to be suspended in a coordinated manner. The screen reader access could be provided by a thread built into the applet/browser, or if the screen reader was written in Java it could have a thread which would interact with the threads of the applet . The applet threads could be preempted by the screen reader access thread, so the applet does not have to do anything. The screen reader access thread could become active in a number of ways.

1. The thread could call an operation that will not return to its caller until input and output operations are complete (blocking on input/output). The operation could be the screen reader finishing the page.

2. Wait() could be used in synchronized threads. Synchronization of the threads causing screen changes with a thread for screen reader access could allow the screen reader thread to interact with the monitor and block the screen updating threads. The other threads would call wait() after a screen update, and the screen reader thread would call notify() or notifyAll() when the screen was analyzed.

3. Synchronization of threads takes time and would be difficult across applets. Yield() would allow the evaluation of the priority of threads, and the screen reader access having a high priority thread could become and stay active. The priority of threads could be altered by user/browser choice or defined in the applet so the screen reader thread has a high priority.

4. Suspend() could be called in the thread, and resume() could be called by the screen reader thread. The threads which affect screen update could be coordinated by being members of a threadgroup, allowing all the members of the threadgroup to suspend()/resume() at once.

5. Knowing the number of active threads, activeCount(), in a screen update thread group could determine if the screen reader access thread would keep control. The screen reader access thread could be a daemon, setDaemon(), only active when other threads are active. Can the group of active threads dictating a daemon's active state be defined?

6. Sleep() could be used if the time of sleep could be adjusted.

7. Maybe, interrupt() could be called outside the program to allow the screen reader access thread to become active, (more drastic).

8. Although handling exceptions is time consuming, maybe the user/browser/screen reader could throw an exception which is predefined exception designed into the applet to be handled by freezing the screen update. When the screen had been analyzed the user could get rid of the exception.

A screen reader recognizes an applet and the Choice component in applets as separate windows. To get into an applet and to leave applets and Choice components often requires a mouse event, rather than a screen reader navigation key stroke. The manner of creation of the applet frame and Choice component should make them act more like part of the browser window. In the applet, being able to use keystrokes to move between objects and to move into and out of applets would be helpful.