SIGN IN SIGN UP

gh-92651: Remove the Include/token.h header file (#92652)

Remove the token.h header file. There was never any public tokenizer
C API. The token.h header file was only designed to be used by Python
internals.

Move Include/token.h to Include/internal/pycore_token.h. Including
this header file now requires that the Py_BUILD_CORE macro is
defined. It no longer checks for the Py_LIMITED_API macro.

Rename functions:

* PyToken_OneChar() => _PyToken_OneChar()
* PyToken_TwoChars() => _PyToken_TwoChars()
* PyToken_ThreeChars() => _PyToken_ThreeChars()
V
Victor Stinner committed
da5727a120e426ffaf68bf3a8016491205bd2f80
Parent: b69297e
Committed by GitHub <noreply@github.com> on 5/11/2022, 9:22:50 PM