> enable multiple interpreters in a single process.
CPython (the default interpreter) has had support for multiple interpreters in the same process since 1997, but it has only been exposed to the C API, and not in the language itself. Python 3.10, coming out later this year, will expose multiple interpreters in one process (subinterpreters, see PEP 554 [1]).
I'm excited about what could eventually come out of this. If there is one GIL per interpreter, we could have something like the `multiprocessing` library for parallel execution, but all within one process.
This is basically how the Threads extension [0] works for Tcl. Tcl has long supported creating many interpreters per thread. The Thread extension exposes some threading commands to the interpreter, which lets you create a new thread+interp. You can then send messages to that interpreter, or pass it file descriptors, but otherwise they are fairly isolated (apartment threading model).
Can multiple Tcl interpreters run truly parallel on different threads in the same process? That’s possible with Lua but AFAIK not with Python, because one of the pieces of global Python interpreter state is the Global Interpreter Lock.
Encapsulated interpreter state via HPyContext would allow replacing the per-process lock with a per-interpreter lock, enabling such parallelism.
Yes, threads from the Thread extension are real OS threads, which have a Tcl interp in them. Since they are separate interpreters then there's no impact from any kind of "per-interpreter lock".
Passing messages between Threads can be done asynchronously which does not block either thread.
Example Tcl program with threads
package require Thread
set tid [thread::create]
thread::send -async $tid {
while true { after 200; puts Jazz }
}
while true { after 100; puts Party }
This creates one thread which is in an infinite loop printing "Jazz" to stdout every 200ms, and the "main" thread which is in an infinite loop printing "Party" to stdout every 100ms.
Related but separate from the subinterpreters PEP, there is interest in moving from a global interpreter lock to one lock per subinterpreter. This would need to be in place before running Python interpreters truly in parallel within one process. There’s a great summary here: https://lwn.net/Articles/820424/
CPython (the default interpreter) has had support for multiple interpreters in the same process since 1997, but it has only been exposed to the C API, and not in the language itself. Python 3.10, coming out later this year, will expose multiple interpreters in one process (subinterpreters, see PEP 554 [1]).
I'm excited about what could eventually come out of this. If there is one GIL per interpreter, we could have something like the `multiprocessing` library for parallel execution, but all within one process.
1 https://www.python.org/dev/peps/pep-0554/