Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Windows 11 modern keyboard: recognize IME candidates window and items #14509

Closed
josephsl opened this issue Jan 4, 2023 · 1 comment · Fixed by #14836
Closed

Windows 11 modern keyboard: recognize IME candidates window and items #14509

josephsl opened this issue Jan 4, 2023 · 1 comment · Fixed by #14836
Labels
p4 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority triaged Has been triaged, issue is waiting for implementation.
Milestone
2023.2

Comments

@josephsl
Copy link
Collaborator

josephsl commented Jan 4, 2023

Hi,

This stems from work done in Windows App Essentials add-on:

Background:

NVDA can recognize modern IME's introduced in Windows 10, specificlaly with Version 2004 (May 2020 Update). Among other things, it allows NVDA to handle IME's as in IAccessible implementation from the past. Windows 11 does come with modern keyboard based IME interface but both gain focus and element selected events are fired when IME candidates are selected.

Is your feature request related to a problem? Please describe.

At the moment Windows 11 IME interface and candidates are not recognized.

Describe the solution you'd like

Recognize Windows 11 modern IME interface and items. This requires reistering new UI Automation Id's in overlay class chooser in emoji panel app module and refining element selected event to descend one more level to handle IME candidates in Windows 11.

Describe alternatives you've considered

No IME UI recognition.

Additional context

Fix is included in Windows App Essentials ad-on. Allowing IME candidates to be recognized allows folks to add further refinements.
@seanbudd seanbudd added p4 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority triaged Has been triaged, issue is waiting for implementation. labels Jan 10, 2023
josephsl added a commit to josephsl/nvda that referenced this issue Apr 16, 2023
@josephsl
appModules/modern keyboard: recognize Windows 11 IME candidate items.…
558b945 
… Re nvaccess#14509.

Windows 11 IME candiate's parent window has AutomationId of 'TEMPLATE_PART_CandidatePanel' and items underneath it wil be reocnigzed as IME candidate items.
josephsl added a commit to josephsl/nvda that referenced this issue Apr 16, 2023
@josephsl
Modern keyboard/Windows 11 IME: recognize IME UI in Windows 11. Re nv…
1ba3df3 
…access#14509.

If UI Automation Id is 'TEMPLATE_PART_CandidatePanel' or 'IME_Prediction_Window' and role is either a list or a pop-up menu, reocgnize this as Windows 11 IME user interface.
@josephsl josephsl mentioned this issue Apr 16, 2023
Merged
6 tasks
seanbudd pushed a commit that referenced this issue Apr 18, 2023
@josephsl
Windows 11/input experience window: recognize IME interface (#14836)
36a5eae 
Closes #14509

Summary of the issue:
NVDA does not recognize Windows 11 IME interface, or if it does, announces candidate items multiple times.

Description of user facing changes
NVDA will recognize and announce Windows 11 IME candidate items used when entering text in languages such as Chinese, Japanese, Korean, and other languages through use of input method editors (IME).

Description of development approach
Three changes to modern keyboard app module:

Recognize IME candidate UI and candidate items found in Windows 11.
Unlike Windows 10, Windows 11 adds an extra element to IME candidate window, therefore descend one more level when locating IME candidate items in app module version of UIA element selected event handler.
Veto UIA element selected event for IME candidate item in Windows 11 until a solution that allows focus and element selected events to cooperate when announcing IME candidate items.
For the third change, feedback from people speaking and typing Chinese, Japanese, and Korean would be helpful (this PR was based on Korean nput but could be expanded to cover Chinese, Japanese, and perhaps other languages in the future, either part of this pull request (unlikely) or in future PR's).

Commits:

* appModules/modern keyboard: recognize Windows 11 IME candidate items. Re #14509.

Windows 11 IME candiate's parent window has AutomationId of 'TEMPLATE_PART_CandidatePanel' and items underneath it wil be reocnigzed as IME candidate items.

* Modern keyboard/Windows 11 IME: recognize IME UI in Windows 11. Re #14509.

If UI Automation Id is 'TEMPLATE_PART_CandidatePanel' or 'IME_Prediction_Window' and role is either a list or a pop-up menu, reocgnize this as Windows 11 IME user interface.

* IME candidate item: do not allow UIA elemnet selected event in Windows 11.

Windows 11 raises focus event when IME candidate receives system focus, and without turning off element selected event, it will result in NVDA announcing candidate items twice. Therefore return early if IME candidate item raises element selected event while running Windows 11.

* IME/element selected: descend one more level when handling IME candidate item in Windows 11.

Unlike Windows 10, Windows 11 IME window adds an extra element when descending IME window hierarchy, therefore descend one more level to handle IME candidate item and UIA eement selected event coming from it.
@nvaccessAuto nvaccessAuto added this to the 2023.2 milestone Apr 18, 2023
@josephsl
Copy link
Collaborator Author

josephsl commented Mar 9, 2024

Hi,

March 2024 update: a refinement to allow NVDA to once again announce top hardware keyboard input suggestion is being tested in Windows App Essentials add-on (dev channel build). I will submit an issue/PR pair about it soon.

Tahnks.

@josephsl josephsl mentioned this issue Mar 9, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
p4 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority triaged Has been triaged, issue is waiting for implementation.
Projects
None yet
Milestone
2023.2
Development

Successfully merging a pull request may close this issue.

Windows 11/input experience window: recognize IME interface josephsl/nvda
3 participants
@seanbudd @josephsl @nvaccessAuto