gh-130804: Fix support of typing unicode chars in pyrepl #130805

sergey-miryanov · 2025-03-03T18:57:59Z

Fixed typing of unicode chars (I tested on Cyrillic chars on Windows)

Issue: [Windows] New REPL doesn't allow to input non-ASCII Unicode characters #130804

encukou

This does fix the issue on Windows. I've also tested on Linux and saw no breakage.

Lib/_pyrepl/base_eventqueue.py

Co-authored-by: Petr Viktorin <encukou@gmail.com>

pablogsal · 2025-03-10T21:44:14Z

LGTM

Thanks a lot for the fix @sergey-miryanov

miss-islington-app · 2025-03-10T22:13:54Z

Thanks @sergey-miryanov for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington-app · 2025-03-10T22:13:56Z

Sorry, @sergey-miryanov and @pablogsal, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 7c98b0674daa3e4eb3e8f35afb61a0dba61d1780 3.13

StanFromIreland · 2025-03-10T22:22:53Z

Tests have failed.
workflow

pablogsal · 2025-03-10T22:27:24Z

How was the PR landed then?

pablogsal · 2025-03-10T22:28:17Z

Test were ok in https://github.com/python/cpython/actions/runs/13775445246/job/38523588646

StanFromIreland · 2025-03-10T22:34:30Z

It's odd, it's the same set up yet somehow they are all failing now?

pablogsal · 2025-03-10T22:34:43Z

This PR also only modifies test_multiple_statements_fail_early so I don't see how is this related

pablogsal · 2025-03-10T22:35:54Z

Reverting doesn't fix anything:

~/github/python/main main ⇡
❯ git revert 7c98b0674daa3e4eb3e8f35afb61a0dba61d1780
[main 46a7449b8e9] Revert "gh-130804: Fix support of typing unicode chars in pyrepl (#130805)"
 3 files changed, 3 insertions(+), 17 deletions(-)
 delete mode 100644 Misc/NEWS.d/next/Core_and_Builtins/2025-03-10-21-46-37.gh-issue-130804.0PpcTx.rst

~/github/python/main main ⇡
❯ make -j -s
Checked 112 modules (34 built-in, 77 shared, 1 n/a on linux-x86_64, 0 disabled, 0 missing, 0 failed on import)

~/github/python/main main ⇡
❯ ./python -m test test_repl -vvv
== CPython 3.14.0a5+ (heads/main:7c98b0674da, Mar 10 2025, 22:34:19) [GCC 14.2.1 20250207]
== Linux-6.13.5-arch1-1-x86_64-with-glibc2.41 little-endian
== Python build: debug
== cwd: /home/pablogsal/github/python/main/build/test_python_worker_1544359æ
== CPU count: 36
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 1616529274
0:00:00 load avg: 9.94 Run 1 test sequentially in a single process
0:00:00 load avg: 9.94 [1/1] test_repl
test_multiple_statements_fail_early (test.test_repl.TestAsyncioREPL.test_multiple_statements_fail_early) ... FAIL
test_toplevel_contextvars_async (test.test_repl.TestAsyncioREPL.test_toplevel_contextvars_async) ... ok
test_toplevel_contextvars_sync (test.test_repl.TestAsyncioREPL.test_toplevel_contextvars_sync) ... ok
test_asyncio_repl_is_ok (test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_is_ok) ... ok
test_asyncio_repl_reaches_python_startup_script (test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_reaches_python_startup_script) ... ok
test_close_stdin (test.test_repl.TestInteractiveInterpreter.test_close_stdin) ... ok
test_interactive_source_is_in_linecache (test.test_repl.TestInteractiveInterpreter.test_interactive_source_is_in_linecache) ... ok
test_interactive_traceback_reporting (test.test_repl.TestInteractiveInterpreter.test_interactive_traceback_reporting) ... ok
test_interactive_traceback_reporting_multiple_input (test.test_repl.TestInteractiveInterpreter.test_interactive_traceback_reporting_multiple_input) ... ok
test_multiline_string_parsing (test.test_repl.TestInteractiveInterpreter.test_multiline_string_parsing) ... ok
test_no_memory (test.test_repl.TestInteractiveInterpreter.test_no_memory) ... ok
test_runsource_show_syntax_error_location (test.test_repl.TestInteractiveInterpreter.test_runsource_show_syntax_error_location) ... ok
test_interactive_syntax_error_correct_line (test.test_repl.TestInteractiveModeSyntaxErrors.test_interactive_syntax_error_correct_line) ... ok

======================================================================
FAIL: test_multiple_statements_fail_early (test.test_repl.TestAsyncioREPL.test_multiple_statements_fail_early)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pablogsal/github/python/main/Lib/test/test_repl.py", line 304, in test_multiple_statements_fail_early
    self.assertNotIn("afterwards", output)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'afterwards' unexpectedly found in 'asyncio REPL 3.14.0a5+ (heads/main:7c98b0674da, Mar 10 2025, 22:34:19) [GCC 14.2.1 20250207] on linux\nUse "await" directly instead of "asyncio.run()".\nType "help", "copyright", "credits" or "license" for more information.\n>>> import asyncio\n>>> Traceback (most recent call last):\n  File "/home/pablogsal/github/python/main/Lib/concurrent/futures/_base.py", line 448, in result\n    return self.__get_result()\n           ~~~~~~~~~~~~~~~~~^^\n  File "/home/pablogsal/github/python/main/Lib/concurrent/futures/_base.py", line 393, in __get_result\n    raise self._exception\n  File "/home/pablogsal/github/python/main/Lib/asyncio/__main__.py", line 42, in callback\n    coro = func()\n  File "<stdin>", line 1, in <module>\n    1 / 0; print(\'afterwards\')\n    ~~^~~\nZeroDivisionError: division by zero\n>>> \nexiting asyncio REPL...\nTraceback (most recent call last):\n  File "/home/pablogsal/github/python/main/Lib/runpy.py", line 198, in _run_module_as_main\n    return _run_code(code, main_globals, None,\n                     "__main__", mod_spec)\n  File "/home/pablogsal/github/python/main/Lib/runpy.py", line 88, in _run_code\n    exec(code, run_globals)\n    ~~~~^^^^^^^^^^^^^^^^^^^\n  File "/home/pablogsal/github/python/main/Lib/asyncio/__main__.py", line 204, in <module>\n    sys.exit(return_code)\n    ~~~~~~~~^^^^^^^^^^^^^\nSystemExit: 0\n>>> \n'

----------------------------------------------------------------------
Ran 13 tests in 1.223s

FAILED (failures=1)
test test_repl failed
0:00:01 load avg: 9.94 [1/1/1] test_repl failed (1 failure)

== Tests result: FAILURE ==

1 test failed:
    test_repl

Total duration: 1.3 sec
Total tests: run=13 failures=1
Total test files: run=1/1 failed=1
Result: FAILURE

StanFromIreland · 2025-03-10T22:37:22Z

Its this PR: 6ab5c4a

pablogsal · 2025-03-10T22:38:04Z

Doesn't seem like it:

~/github/python/main main ⇡
❯ git revert 6ab5c4aa05bf35832a3ccd1e71b28b8475fa30f4
[main 1428e541980] Revert "gh-124927: Fix conversion issue between coordinates and position in REPL (#125001)"
 4 files changed, 1 insertion(+), 12 deletions(-)
 delete mode 100644 Misc/NEWS.d/next/Library/2024-10-05-13-25-07.gh-issue-124927.uzNA32.rst

~/github/python/main main ⇡
❯ make
Checked 112 modules (34 built-in, 77 shared, 1 n/a on linux-x86_64, 0 disabled, 0 missing, 0 failed on import)
./python -E ./Tools/build/generate-build-details.py `cat pybuilddir.txt`/build-details.json

~/github/python/main main ⇡
❯ ./python -m test test_repl -vvv
== CPython 3.14.0a5+ (bisect/good-ecdf6b15b0c0570c3c3302ab95bdbfd3007ea941-2-g6ab5c4aa05b:6ab5c4aa05b,) [GCC 14.2.1 20250207]
== Linux-6.13.5-arch1-1-x86_64-with-glibc2.41 little-endian
== Python build: debug
== cwd: /home/pablogsal/github/python/main/build/test_python_worker_1588881æ
== CPU count: 36
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 4038269957
0:00:00 load avg: 3.85 Run 1 test sequentially in a single process
0:00:00 load avg: 3.85 [1/1] test_repl
test_multiple_statements_fail_early (test.test_repl.TestAsyncioREPL.test_multiple_statements_fail_early) ... FAIL
test_toplevel_contextvars_async (test.test_repl.TestAsyncioREPL.test_toplevel_contextvars_async) ... ok
test_toplevel_contextvars_sync (test.test_repl.TestAsyncioREPL.test_toplevel_contextvars_sync) ... ok
test_asyncio_repl_is_ok (test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_is_ok) ... ok
test_asyncio_repl_reaches_python_startup_script (test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_reaches_python_startup_script) ... ok
test_close_stdin (test.test_repl.TestInteractiveInterpreter.test_close_stdin) ... ok
test_interactive_source_is_in_linecache (test.test_repl.TestInteractiveInterpreter.test_interactive_source_is_in_linecache) ... ok
test_interactive_traceback_reporting (test.test_repl.TestInteractiveInterpreter.test_interactive_traceback_reporting) ... ok
test_interactive_traceback_reporting_multiple_input (test.test_repl.TestInteractiveInterpreter.test_interactive_traceback_reporting_multiple_input) ... ok
test_multiline_string_parsing (test.test_repl.TestInteractiveInterpreter.test_multiline_string_parsing) ... ok
test_no_memory (test.test_repl.TestInteractiveInterpreter.test_no_memory) ... ok
test_runsource_show_syntax_error_location (test.test_repl.TestInteractiveInterpreter.test_runsource_show_syntax_error_location) ... ok
test_interactive_syntax_error_correct_line (test.test_repl.TestInteractiveModeSyntaxErrors.test_interactive_syntax_error_correct_line) ... ok

======================================================================
FAIL: test_multiple_statements_fail_early (test.test_repl.TestAsyncioREPL.test_multiple_statements_fail_early)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pablogsal/github/python/main/Lib/test/test_repl.py", line 304, in test_multiple_statements_fail_early
    self.assertNotIn("afterwards", output)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'afterwards' unexpectedly found in 'asyncio REPL 3.14.0a5+ (bisect/good-ecdf6b15b0c0570c3c3302ab95bdbfd3007ea941-2-g6ab5c4aa05b:6ab5c4aa05b,) [GCC 14.2.1 20250207] on linux\nUse "await" directly instead of "asyncio.run()".\nType "help", "copyright", "credits" or "license" for more information.\n>>> import asyncio\n>>> Traceback (most recent call last):\n  File "/home/pablogsal/github/python/main/Lib/concurrent/futures/_base.py", line 448, in result\n    return self.__get_result()\n           ~~~~~~~~~~~~~~~~~^^\n  File "/home/pablogsal/github/python/main/Lib/concurrent/futures/_base.py", line 393, in __get_result\n    raise self._exception\n  File "/home/pablogsal/github/python/main/Lib/asyncio/__main__.py", line 42, in callback\n    coro = func()\n  File "<stdin>", line 1, in <module>\n    1 / 0; print(\'afterwards\')\n    ~~^~~\nZeroDivisionError: division by zero\n>>> \nexiting asyncio REPL...\nTraceback (most recent call last):\n  File "/home/pablogsal/github/python/main/Lib/runpy.py", line 198, in _run_module_as_main\n    return _run_code(code, main_globals, None,\n                     "__main__", mod_spec)\n  File "/home/pablogsal/github/python/main/Lib/runpy.py", line 88, in _run_code\n    exec(code, run_globals)\n    ~~~~^^^^^^^^^^^^^^^^^^^\n  File "/home/pablogsal/github/python/main/Lib/asyncio/__main__.py", line 204, in <module>\n    sys.exit(return_code)\n    ~~~~~~~~^^^^^^^^^^^^^\nSystemExit: 0\n>>> \n'

----------------------------------------------------------------------
Ran 13 tests in 1.400s

FAILED (failures=1)
test test_repl failed
0:00:01 load avg: 3.85 [1/1/1] test_repl failed (1 failure)

== Tests result: FAILURE ==

1 test failed:
    test_repl

Total duration: 1.5 sec
Total tests: run=13 failures=1
Total test files: run=1/1 failed=1
Result: FAILURE

StanFromIreland · 2025-03-10T22:41:04Z

Weird? Only one left then would be yours a931a8b maybe it has something to do with the traceback modifications?

pablogsal · 2025-03-10T22:42:25Z

I'm pretty sure it's a931a8b

pablogsal · 2025-03-10T22:46:40Z

#131065

pablogsal · 2025-03-10T22:51:11Z

@sergey-miryanov Can you check #130805 (comment) ?

Seems the backport needs some adaptations to 3.13. Could you take a look?

sergey-miryanov · 2025-03-11T06:57:11Z

Seems the backport needs some adaptations to 3.13. Could you take a look?

@pablogsal
There is a conflict because a65366e doesn't backported to 3.13 and this PR fixes regression from this commit. Should we backport a65366e first?

pablogsal · 2025-03-11T12:56:19Z

That's tricky. That may be considered a feature so I am not sure we can. @ambv what do you think?

Fix support of typing unicode chars in pyrepl

3d674cf

sergey-miryanov requested review from pablogsal, lysnikolaou and ambv as code owners March 3, 2025 18:57

bedevere-app bot added the awaiting review label Mar 3, 2025

bedevere-app bot mentioned this pull request Mar 3, 2025

[Windows] New REPL doesn't allow to input non-ASCII Unicode characters #130804

Closed

sergey-miryanov added 3 commits March 4, 2025 00:45

Make mypy happy

c1f3aa8

Add assert to check if passed char is str

ef0d0ee

Merge branch 'main' into pythongh-130804-pyrepl-unicode-chars

497a0bb

encukou reviewed Mar 10, 2025

View reviewed changes

Lib/_pyrepl/base_eventqueue.py Outdated Show resolved Hide resolved

sergey-miryanov and others added 2 commits March 11, 2025 01:06

Update Lib/_pyrepl/base_eventqueue.py

bfa2e4e

Co-authored-by: Petr Viktorin <encukou@gmail.com>

Merge branch 'main' into pythongh-130804-pyrepl-unicode-chars

c667049

pablogsal approved these changes Mar 10, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting review labels Mar 10, 2025

pablogsal added needs backport to 3.13 and removed awaiting merge labels Mar 10, 2025

Add NEWS entry

7cc16ea

pablogsal enabled auto-merge (squash) March 10, 2025 21:48

pablogsal disabled auto-merge March 10, 2025 21:48

pablogsal enabled auto-merge (squash) March 10, 2025 21:48

pablogsal merged commit 7c98b06 into python:main Mar 10, 2025
47 checks passed

miss-islington-app bot assigned pablogsal Mar 10, 2025

sergey-miryanov mentioned this pull request Mar 29, 2025

New REPL on Windows exits when accented character is pasted/typed #131878

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub Sponsors

gh-130804: Fix support of typing unicode chars in pyrepl #130805

gh-130804: Fix support of typing unicode chars in pyrepl #130805

sergey-miryanov commented Mar 3, 2025 •

edited by bedevere-app bot

Loading

encukou left a comment

pablogsal commented Mar 10, 2025

miss-islington-app bot commented Mar 10, 2025

miss-islington-app bot commented Mar 10, 2025

StanFromIreland commented Mar 10, 2025

pablogsal commented Mar 10, 2025

pablogsal commented Mar 10, 2025

StanFromIreland commented Mar 10, 2025

pablogsal commented Mar 10, 2025

pablogsal commented Mar 10, 2025

StanFromIreland commented Mar 10, 2025

pablogsal commented Mar 10, 2025

StanFromIreland commented Mar 10, 2025 •

edited

Loading

pablogsal commented Mar 10, 2025

pablogsal commented Mar 10, 2025

pablogsal commented Mar 10, 2025

sergey-miryanov commented Mar 11, 2025 •

edited

Loading

pablogsal commented Mar 11, 2025

gh-130804: Fix support of typing unicode chars in pyrepl #130805

gh-130804: Fix support of typing unicode chars in pyrepl #130805

Conversation

sergey-miryanov commented Mar 3, 2025 • edited by bedevere-app bot Loading

encukou left a comment

Choose a reason for hiding this comment

pablogsal commented Mar 10, 2025

miss-islington-app bot commented Mar 10, 2025

miss-islington-app bot commented Mar 10, 2025

StanFromIreland commented Mar 10, 2025

pablogsal commented Mar 10, 2025

pablogsal commented Mar 10, 2025

StanFromIreland commented Mar 10, 2025

pablogsal commented Mar 10, 2025

pablogsal commented Mar 10, 2025

StanFromIreland commented Mar 10, 2025

pablogsal commented Mar 10, 2025

StanFromIreland commented Mar 10, 2025 • edited Loading

pablogsal commented Mar 10, 2025

pablogsal commented Mar 10, 2025

pablogsal commented Mar 10, 2025

sergey-miryanov commented Mar 11, 2025 • edited Loading

pablogsal commented Mar 11, 2025

sergey-miryanov commented Mar 3, 2025 •

edited by bedevere-app bot

Loading

StanFromIreland commented Mar 10, 2025 •

edited

Loading

sergey-miryanov commented Mar 11, 2025 •

edited

Loading