test_re fails on Alpine Linux (with musl)

[prashanthallu]

Bug report

Bug description:

718.6 0:01:10 load avg: 0.97 [35/44] test_re
719.7 test test_re failed
719.7 0:01:11 load avg: 0.98 [36/44] test_set -- test_re failed (2 failures)
724.7 0:01:16 load avg: 0.98 [37/44] test_sqlite3
725.6 0:01:17 load avg: 0.98 [38/44] test_statistics
733.0 0:01:24 load avg: 0.98 [39/44] test_str
735.1 0:01:26 load avg: 0.98 [40/44] test_struct
735.8 0:01:27 load avg: 0.98 [41/44] test_tabnanny
736.3 0:01:28 load avg: 0.98 [42/44] test_time
739.5 0:01:31 load avg: 0.98 [43/44] test_xml_etree
740.2 0:01:32 load avg: 0.98 [44/44] test_xml_etree_c
741.3
741.3 Total duration: 1 min 33 sec
741.3 Total tests: run=9,178 failures=3 skipped=203
741.3 Total test files: run=44/44 failed=2 skipped=2
741.3 Result: FAILURE
741.3 make: *** [Makefile:886: profile-run-stamp] Error 2

And while running in verbose mode the following is the output of it.



 ./python -m unittest -v test.test_re
(...)
======================================================================
FAIL: test_locale_caching (test.test_re.ReTests.test_locale_caching)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Python-3.13.2/Lib/test/test_re.py", line 2195, in test_locale_caching
    self.check_en_US_iso88591()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Python-3.13.2/Lib/test/test_re.py", line 2204, in check_en_US_iso88591
    self.assertTrue(re.match(b'\xc5', b'\xe5', re.L|re.I))
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: None is not true

======================================================================
FAIL: test_locale_compiled (test.test_re.ReTests.test_locale_compiled)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Python-3.13.2/Lib/test/test_re.py", line 2240, in test_locale_compiled
    self.assertTrue(p.match(b'\xe5\xe5'))
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
AssertionError: None is not true

----------------------------------------------------------------------
Ran 165 tests in 0.838s

FAILED (failures=2, skipped=1)

CPython versions tested on:

3.13

Operating systems tested on:

Linux

[StanFromIreland]

Do you have en_US_iso88591 installed? Check with:

locale -a | grep en_US

Both of the tests depend on it.

I presume it is setting a locale that is not available for some reason.

        for loc in 'en_US.iso88591', 'en_US.utf8':
            try:
                locale.setlocale(locale.LC_CTYPE, loc)
            except locale.Error:
                # Unsupported locale on this system
                self.skipTest('test needs %s locale' % loc)

I am unable to reproduce after deleting locale:

test_locale_caching (test.test_re.ReTests.test_locale_caching) ... skipped 'test needs en_US.iso88591 locale'
test_locale_compiled (test.test_re.ReTests.test_locale_compiled) ... skipped 'test needs en_US.iso88591 locale'

[vstinner]

@StanFromIreland: Do you link Python to the musl C library?

[StanFromIreland]

@vstinner: I should have specified, I did not. I am not sure whether it is even packaged for Fedora.

[vstinner]

musl is package in Fedora, but I failed to build Python using musl-gcc wrapper. I built Python in Alpine to get musl.

[serhiy-storchaka]

Please add the following lines at the beginning of test_locale_compiled() and show the output.

        print()
        print(f'LC_CTYPE: {locale.setlocale(locale.LC_CTYPE)}')
        print(f'LC_COLLATE: {locale.setlocale(locale.LC_COLLATE)}')
        print(f'LC_ALL: {locale.setlocale(locale.LC_ALL)}')

[vstinner]

Please add the following lines at the beginning of test_locale_compiled() and show the output.

Here you have:

test_locale_compiled (test.test_re.ReTests.test_locale_compiled) ... 
LC_CTYPE: C.UTF-8
LC_COLLATE: C
LC_ALL: C.UTF-8;C;C;C;C;C
FAIL

[vstinner]

Using musl, setlocale() doesn't fail even if the locale isn't supported:

$ ./python 
Python 3.14.0a5+ (heads/main-dirty:fe186d76cfd, Mar 11 2025, 08:54:51) [GCC 14.2.0] on linux
>>> import locale
>>> locale.setlocale(locale.LC_CTYPE, 'en_US.iso88591')
'en_US.iso88591'
>>> locale.getencoding()
'UTF-8'

Another example:

>>> import locale
>>> locale.setlocale(locale.LC_CTYPE, 'donotexist')
'donotexist'
>>> locale.getencoding()
'UTF-8'

[vstinner]

See also #131313

[StanFromIreland]

The PR covers both the tests here already. I guess the PR can be assigned this issue?

[deepwzh]

I have roughly identified the reasons:

1. Alpine does not include the en_US_iso88591 encoding by default.
2. The implementations of the setlocale function in musl and glibc are inconsistent. For musl, passing any value will not cause an error. Therefore, even if locale.setlocale(locale.LC_CTYPE, 'en_US.iso88591') does not throw an exception, it will not work properly.

When I run the code in two environments, the behavior is different

locale.setlocale(locale.LC_CTYPE, "xxx.xxx")

glibc:

locale.setlocale(locale.LC_CTYPE, "xxx.xxx")  
Traceback (most recent call last):  
  File "/home/deepwzh/opensource/python_demo/test.py", line 9, in <module>  
    locale.setlocale(locale.LC_CTYPE, "xxx.xxx")  
  File "/home/deepwzh/miniconda3/lib/python3.12/locale.py", line 615, in setlocale  
    return _setlocale(category, locale)  
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^  

musl:
No error is thrown.

Further, you can refer to the musl source code here:
https://github.com/bminor/musl/blob/c47ad25ea3b484e10326f933e927c0bc8cded3da/src/locale/locale_map.c#L99
From this, it can be seen that if a non-existent locale is passed, it will actually default to UTF-8 encoding instead of throwing an error.

[vstinner]

The PR covers both the tests here already. I guess the PR can be assigned this issue?

PR #131313 was assigned to the older issue #90548.

[vstinner]

Issue fixed by change 6146295.