0%

PEP 703(使全局解释器锁在 CPython 中可选)验收

PEP 703(使全局解释器锁在 CPython 中可选)验收

(Posted for the whole Steering Council.)
(发布给整个指导委员会。)

As we’ve announced before, the Steering Council has decided to accept PEP 703 (Making the Global Interpreter Lock Optional in CPython) . We want to make it clear why, and under what expectations we’re doing so.
正如我们之前宣布的,指导委员会已决定接受PEP 703(使CPython中的全局解释器锁可选)。我们想明确说明为什么,以及我们这样做的期望。

It is clear to the Steering Council that theoretically, a no-GIL (or free-threaded) Python would be of great benefit, and the majority of the community seems in agreement. Threads have significant downsides and caveats, but they are widely adopted, both by software and hardware, and they do enable more scalable solutions to problems. The GIL clearly inhibits CPython in this, and removing that barrier would be a good thing.
指导委员会很清楚,从理论上讲,无 GIL(或自由线程)Python 将大有裨益,社区的大多数人似乎都同意这一点。线程有明显的缺点和警告,但它们被软件和硬件广泛采用,并且它们确实为问题提供了更具可扩展性的解决方案。GIL在这方面显然抑制了CPython,消除这种障碍将是一件好事。

At the same time we’re not sure if it’s possible to remove the GIL without fundamentally breaking all extension modules out there, or significantly reducing the performance or maintainability of CPython. The third-party/PyPI package ecosystem is one of Python’s strengths, and the tight, efficient integration with C libraries is one of CPython’s. It has enabled the existence of a diverse selection of packages that’s a unique selling point for Python. We need to be careful that we do not destroy those benefits, or discard decades worth of package development.
同时,我们不确定是否可以在不从根本上破坏所有扩展模块的情况下删除 GIL,或者显着降低 CPython 的性能或可维护性。第三方/PyPI 包生态系统是 Python 的优势之一,与 C 库的紧密、高效集成是 CPython 的优势之一。它使各种包的存在成为可能,这是Python的独特卖点。我们需要小心,不要破坏这些好处,或者放弃几十年的软件包开发。

Assessing the practical impact, and the practicality of adapting third-party packages to the new free-threaded situation, is difficult without a finished implementation. The unpredictable nature of thread-related issues makes it extra difficult, as some issues won’t show up until put under significant load. The changes necessary to remove the GIL are substantive enough, and require so much coordination with other CPython development happening at the same time, that we can’t reasonably do these experiments in a fork of CPython. We also want to avoid the risk of ecosystem fragmentation and unnecessarily diverging changes because of the work being done in a fork. For PEP 703 to move forward, it has to be included in CPython’s main, and released as part of regular releases (albeit not necessarily by default).
如果没有完成的实现,评估实际影响以及使第三方包适应新的自由线程情况的实用性是很困难的。线程相关问题的不可预测性使其变得格外困难,因为某些问题在承受大量负载之前不会显示出来。删除 GIL 所需的更改是实质性的,并且需要与同时发生的其他 CPython 开发进行如此多的协调,以至于我们无法合理地在 CPython 的分支中进行这些实验。我们还希望避免由于分叉中所做的工作而导致生态系统碎片化和不必要的差异变化的风险。为了使PEP 703向前发展,它必须包含在CPython的主版本中,并作为常规版本的一部分发布(尽管不一定是默认的)。

But while we think removal of the GIL is a worthy and necessary goal, and PEP 703 is the best proposal for it so far, we can’t at this stage guarantee that it will work out. We have to, as we develop PEP 703’s implementation and the necessary user-visible changes to semantics, APIs and ABIs, continually evaluate the feasibility, and be prepared to change course – or reverse it, if that turns out to be necessary.
但是,虽然我们认为取消 GIL 是一个有价值且必要的目标,并且 PEP 703 是迄今为止最好的建议,但我们不能在现阶段保证它会成功。当我们开发PEP 703的实现以及对语义,API和ABI的必要用户可见更改时,我们必须不断评估可行性,并准备改变方向 - 或者在必要时扭转它。

As mentioned before, we see this as a rollout in roughly three stages:
如前所述,我们认为这是大约三个阶段的推出:

  • Phase I: Experimental phase, which can start immediately, in which the free-threaded build is enabled through a build-time option. This should not be the default install anywhere. At least one major Python release should include this experimental free-threaded build, to allow third-party packages to test and do their own experimentation. In this stage we should make it clear the build is experimental, not supported for “production use”, and may be reverted.
    阶段 I:实验阶段,可以立即开始,其中通过构建时选项启用自由线程构建。这不应该是任何地方的默认安装。至少一个主要的 Python 版本应该包含这个实验性的自由线程构建,以允许第三方包进行测试并进行自己的实验。在此阶段,我们应该明确构建是实验性的,不支持“生产用途”,并且可以还原。
  • Phase II: Supported-but-not-default phase, which would start when the API and ABI changes have sufficiently settled, and there is sufficient community support. Exact criteria for this phase are hard to pin down at this stage, so this will have to involve some discussion among Core Devs and the community, and a decision by the SC at the time. At this point reverting should still be possible (so for example preprocessor guards should remain in place) although obviously we aren’t expecting it.
    第二阶段:支持但非默认阶段,当 API 和 ABI 更改充分解决并且有足够的社区支持时,该阶段将开始。在这个阶段很难确定这个阶段的确切标准,所以这必须涉及核心开发人员和社区之间的一些讨论,以及当时SC的决定。在这一点上,恢复应该是可能的(因此,例如预处理器保护应该保留在原位),尽管显然我们并不期望它。
  • Phase III: Default phase, at which point the free-threaded build becomes the default (but can initially still be disabled). Again, the exact criteria are hard to pin down this far ahead, but the aim is to make this as seamless and painless a default flip as possible. Like the previous phase, the SC at the time will need to make a decision as to when this occurs. Some time after the default flip, when we have a good indication it’s no longer widely used, we should start the discussion on removing the GIL build entirely.
    阶段 III:默认阶段,此时自由线程构建成为默认阶段(但最初仍可禁用)。同样,确切的标准很难确定这么远,但目的是使它尽可能无缝和无痛地默认翻转。与上一阶段一样,当时的SC需要决定何时发生这种情况。在默认翻转后的一段时间,当我们有一个很好的指示它不再被广泛使用时,我们应该开始讨论完全删除 GIL 构建。
    The details of the phases are deliberately vague, simply because we can’t know all the ecosystem impact details yet, and we don’t want to set conservative standards now and then hold people to them when in practice we’re being too cautious. (We don’t want to set overly ambitious goals and break too many things, either.)
    这些阶段的细节故意含糊不清,仅仅是因为我们还不能知道所有的生态系统影响细节,我们不想现在设定保守的标准,然后在实践中我们过于谨慎时让人们遵守这些标准。(我们也不想设定过于雄心勃勃的目标,也不想破坏太多的东西。

For some of the changes necessary for the free-threaded build, like switching to mimalloc or significant changes to the GC, it may be useful to make these separately build-time opt-ins, and perhaps make them the default before the free-threaded build becomes the default build (but not before it becomes fully supported). Having them separately enableable allows for experimentation, performance measurements and debugging focusing on the isolated set of changes. We don’t want a complex matrix of build flags, so this should probably be limited to one or two.
对于自由线程构建所需的某些更改,例如切换到mimalloc或对 GC 进行重大更改,进行这些单独的构建时选择加入可能很有用,并且可能在自由线程构建成为默认构建之前将它们设为默认值(但不是在它完全受支持之前)。让它们单独启用允许实验、性能测量和调试,重点是孤立的一组更改。我们不想要一个复杂的构建标志矩阵,所以这可能应该限制在一个或两个。

Regarding the expected performance impact of the free-threaded build, the SC thinks a significant performance penalty is expected in a free-threaded build, and the benefit is probably worth that price. At this point we’re expecting a (worst case) performance penalty of 10-15%. We don’t want to set strict limits on acceptable performance, partly because we don’t want to get stuck in arguments about how to measure performance and partly because it will depend on user expectations and, for example, how much performance work is invalidated and how much we can expect to see recuperated over time. Solutions for the free-threaded build that are fundamentally problematic for performance improvements going forward, are less acceptable than solutions that are currently suboptimal but have room for improvement.
关于自由线程构建的预期性能影响,SC 认为自由线程构建中预计会出现显著的性能损失,其好处可能值得这个价格。在这一点上,我们预计(最坏情况)性能损失为 10-15%。我们不想对可接受的性能设置严格的限制,部分原因是我们不想陷入关于如何衡量性能的争论中,部分原因是这将取决于用户的期望,例如,有多少性能工作无效,以及我们可以期望看到多少随着时间的推移而恢复。自由线程构建的解决方案从根本上对未来的性能改进存在问题,与当前次优但有改进空间的解决方案相比,这些解决方案更不可接受。

The performance impact should be isolated to a free-threaded build; a GIL build should not see any performance impact in existing code. For API and ABI changes necessary to support both GIL and free-threaded builds (e.g. avoiding APIs that return borrowed references or that rely on the GIL to protect shared data), it’s reasonable for the new interfaces to be slightly less performant, but we expect this to be very limited and usually lost in the noise.
性能影响应隔离到自由线程生成;GIL 生成不应在现有代码中看到任何性能影响。对于支持 GIL 和自由线程构建所需的 API 和 ABI 更改(例如,避免返回借用引用或依赖 GIL 来保护共享数据的 API),新接口的性能略低是合理的,但我们预计这非常有限,并且通常会在噪音中丢失。

In a similar vein, it’s important that as the free-threaded build lands in main, so that its implementation is considered in other development work that’s going on. New features can’t land without proper support for the free-threaded build, when the two intersect. It may seem tempting to ignore free-threaded when developing thread-adjacent changes, but in the end someone will have to make it work, and it’s neither fair nor particularly forward-thinking to expect the free-threaded maintainers to do all the work. We also need to be mindful of the cohesiveness of the language, the implementation and the C API. We have to assume the free-threaded build will be the only build in the reasonable future, and like other fundamental changes to CPython internals, we all have to learn the new way of approaching these problems. This may be a bit of a jump in terms of complexity – the GIL implicitly simplified so much – but it is a necessary step. We do expect, at least initially, the free-threaded build experts to help others ramp up here. The experimental phase is there for CPython and the Core Devs to get used to the free-threaded build as much as it is for users.
同样,重要的是,随着自由线程构建登陆main,以便在正在进行的其他开发工作中考虑其实现。当两者相交时,如果没有对自由线程构建的适当支持,新功能就无法落地。在开发与线程相邻的更改时,忽略自由线程似乎很诱人,但最终必须有人让它工作,期望自由线程维护者完成所有工作既不公平也不特别具有前瞻性。我们还需要注意语言、实现和 C API 的凝聚力。我们必须假设自由线程构建将是合理未来唯一的构建,并且就像 CPython 内部的其他根本性更改一样,我们都必须学习解决这些问题的新方法。就复杂性而言,这可能有点飞跃 - GIL隐含地简化了这么多 - 但这是一个必要的步骤。我们确实希望,至少在最初,自由线程构建专家能够帮助其他人在这里提升。实验阶段是CPython和核心开发人员习惯自由线程构建的,就像用户一样。

We do need a few specific things resolved before PEP 703 can leave the experimental phase. For starters, we need a solution for the ABI. The SC believes strongly that a single ABI serving both with-GIL and free-threaded builds should be possible, should be made possible, and should be required before leaving the experimental phase. If this turns out to be an unreasonable requirement, we’ll have to look at alternative solutions to ease the pressure on package maintainers (e.g. building two extension modules in the same wheel, or providing a compatibility layer through a separate library).
在PEP 703离开实验阶段之前,我们确实需要解决一些具体问题。首先,我们需要一个针对 ABI 的解决方案。SC 坚信,在离开实验阶段之前,应该有可能,应该成为可能,并且应该需要同时使用 GIL 和自由线程构建的单个 ABI。如果事实证明这是一个不合理的要求,我们将不得不寻找替代解决方案来减轻软件包维护者的压力(例如,在同一轮子上构建两个扩展模块,或通过单独的库提供兼容层)。

We also need to consider the testing matrix, both for CPython and for third-party packages. Even with a stable ABI we still need to multiply the test matrix. We probably don’t need complete coverage on all supported platforms, but we do want at least one free-threaded buildbot for each of the T1/T2 platforms, as well as some way to test the validity of the unified ABI (a way to build things in one build mode and test against an interpreter built in the other build mode). We currently rely on the stable ABI check and third-party testing of the stable ABI, but that will probably not be good enough to ensure the compatibility between the GIL and free-threaded builds. To get to the supported phase we also expect CI checks on GitHub for free-threaded builds on each of the major platforms.
我们还需要考虑CPython和第三方软件包的测试矩阵。即使有稳定的ABI,我们仍然需要乘以测试矩阵。我们可能不需要在所有支持的平台上完全覆盖,但我们确实希望每个 T1/T2 平台至少有一个自由线程构建机器人,以及某种方法来测试统一 ABI 的有效性(一种在一种构建模式下构建东西并针对在另一种构建模式下构建的解释器进行测试的方法)。我们目前依赖于稳定 ABI 检查和稳定 ABI 的第三方测试,但这可能不足以确保 GIL 和自由线程构建之间的兼容性。为了进入支持的阶段,我们还希望在 GitHub 上对每个主要平台上的自由线程构建进行 CI 检查。

There are a few specific things we want to avoid. We do not want the free-threaded build to be used as the default Python anywhere until the Core Devs and the Python community are ready for that. Obviously we can’t stop users and distributors from installing a free-threaded build by default, but we think at this stage it would be a mistake to do so for anything besides end-to-end experimentation. We also want to avoid labelling the free-threaded build “experimental” after the experimental phase. Build-time flags, defines, comments in the code should avoid the word. We want to avoid negatives in terms and flags and such, so we won’t get into double-negative terrain (like we do when we talk about ‘non no-GIL’). We’d like a positive, clear term to talk about the no-GIL build, and we’re suggesting ‘free-threaded’. (Relatedly, that’s why the build mode/ABI letter is ‘t’ and not ‘n’; that change was already made.)
我们想要避免一些具体的事情。我们不希望在核心开发人员和 Python 社区准备好之前,在任何地方将自由线程构建用作默认 Python。显然,默认情况下,我们无法阻止用户和分销商安装自由线程构建,但我们认为在现阶段,除了端到端实验之外,这样做都是错误的。我们还希望避免在实验阶段之后将自由线程构建标记为“实验性”。代码中的构建时标志、定义、注释应避免使用这个词。我们希望避免在术语和标志等方面出现负面因素,因此我们不会陷入双重负面领域(就像我们谈论“非无 GIL”时所做的那样)。我们想要一个积极、明确的术语来讨论无 GIL 构建,我们建议使用“自由线程”。(与此相关的是,这就是为什么构建模式/ABI 字母是“t”而不是“n”;该更改已经进行了。

In short, the SC accepts PEP 703, but with clear provisio: that the rollout be gradual and break as little as possible, and that we can roll back any changes that turn out to be too disruptive – which includes potentially rolling back all of PEP 703 entirely if necessary (however unlikely or undesirable we expect that to be).
简而言之,SC 接受 PEP 703,但有明确的条件:推出是渐进的,尽可能少地中断,我们可以回滚任何破坏性太大的更改——其中包括在必要时可能完全回滚所有 PEP 703(无论我们期望多么不可能或不希望)。

For the whole SC,
Thomas.

宇宙山河浪漫,赞赏动力无限

Welcome to my other publishing channels