gh-92651: Remove the Include/token.h header file (#92652)
Remove the token.h header file. There was never any public tokenizer C API. The token.h header file was only designed to be used by Python internals. Move Include/token.h to Include/internal/pycore_token.h. Including this header file now requires that the Py_BUILD_CORE macro is defined. It no longer checks for the Py_LIMITED_API macro. Rename functions: * PyToken_OneChar() => _PyToken_OneChar() * PyToken_TwoChars() => _PyToken_TwoChars() * PyToken_ThreeChars() => _PyToken_ThreeChars()
V
Victor Stinner committed
da5727a120e426ffaf68bf3a8016491205bd2f80
Parent: b69297e
Committed by GitHub <noreply@github.com>
on 5/11/2022, 9:22:50 PM