If i were to design a new programming language …
I’d see to base syntax and semantics on Python3, but strip and rebase it:
- no C: implement the interpreter in RPython, get a JIT for free and implementation bits from PyPy’s Python interpreter (parsing, IO, etc.)
- no drags-you-down batteries: lean interpreter core and a standard battery distro which is tested against the last N interpreter versions + current
- no yield: use greenlets to implement all of what yield provides and more
- no underlying blocking on IO: base it all on event loop, yet provide synchronous programming model through greenlets
- no c-level API nor ctypes: use cffi to interface with c-libraries
- no global state: just support state bound to execution context/stack
- no GIL: support free threading and Automatic Mutual Exclusion for dealing with shared state
- no setup.py: have a thought-through story and tools from the start for packaging, installation, depending/interfacing between packages
- no import, no sys.modules: provide an object with which you can access other packages’s objects and introspect/interact with one’s own package
- no testing as an afterthought: everything needs to be easily testable, empowered assert statement and branch-coverage supported from the core.
- no extensibility as an afterthought: support plugins and loose coupling through builtin 1:N calling mechanism (event notification on steroids)
- no unsafe code: support IO/CPU/RAM sandboxing as a core feature
- no NIH syndrome: provide a bridge to a virtualenv’ed Python interpreter allowing to leverage existing good crap
Anything else? Probably! Discussion needed? Certainly. Unrealistic? Depends on who would participate — almost all of the above has projects, PEPs and code showcasing viability.
Btw, did you know that when we started PyPy we initially did this under the heading of “Minimal Python”? Some of the above ideas above and their underlying motivations were already mentioned when I invited to the first PyPy sprint almost 10 years ago:
http://mail.python.org/pipermail/python-dev/2003-January/032427.html
I learned since then that Python has more complex innards than it seems but i still believe it could be both simpler and more powerful.
holger
Written by holger krekel
November 13, 2012 at 3:29 pm
Posted in metaprogramming, testing
17 Responses
Subscribe to comments with RSS.
metaprogramming and politics 
If no global state, I presume you would have a notion of immutability?
I’d add first class dynamically scoped data. You allude to it with execution stack, but I think it should be formalized in the language rather than required through exclusion. But then… I’m not sure the call stack is sufficient. Is it, when you remove yield? Hrm.
If you remove C extensions packaging becomes much simpler because it’s all just portable files.
Ian Bicking
November 13, 2012 at 3:36 pm
regarding “execution-global” state, I am thinking of what Armin Ronacher hacked for Flask’s request/app access, it’s stacked state bound to thread/greenlets which can be activated with “with XYZ: …” and the body, however nested, has direct access to it without passing it around explicitely.
holger krekel
November 13, 2012 at 3:40 pm
and as to yield: it can be implemented fully in terms of greenlets, see a little example here: https://github.com/python-greenlet/greenlet/blob/master/tests/test_generator.py
holger krekel
November 13, 2012 at 3:45 pm
You know, Holger… Not to be a downer, but I have both positive and negative feelings in response to your post…
I strongly feel approval about most of your ideas here. That is: I think the most important things in a language/library/toolset — and the things that should be the focus of any good new R&D effort — are testing, packaging, FFI, modularity, concurrency, security… There’s a pattern here! _All_ of these things are about making it practical to share and re-use other people’s code.
I think it would be a misallocation of resources to work on a new programming languages whose framing is “empowering the lone programmer even more” — how to further elevate and empower this isolated person who is writing code and debugging and extending their own code.
Such empowerment is not necessarily a bad thing to do, but where we’re really hurting is facilitating cooperation between programmers, including “casual cooperation” between people who never talk or who’ve never even heard of each other, but whose code ends up calling each other’s code.
Witness the fact that the Go language is very well liked, and some of its most celebrated innovations are extremely simple dumb stuff like a standard formatter that puts an end to style wars by enforcing a rigid coding style. Why? Because putting an end to some sources of quibbling and bike-shedding is more valuable to more people than offering some new mind-expanding abstraction that lets you express more complicated things in more elegant utterances.
So, the whole notion of putting those things — FFI and testing and concurrency and security — first in a new project is very appealing to me.
Okay, what are my negative feelings?
I hate to do this, but:
Python
I just don’t feel excitement about the propect of a new project building on top of Python. I get excited about Rust. Maybe some new things that could be built on top of Javascript. I’m fairly excited about a gleam-in-the-eye programming language being invented by my collaborator David-Sarah Hopwood, mostly just because I have such admiration for David-Sarah’s way of thinking that I want to see what they come up with.
And I’m excited about PyPy, as an *implementation* of Python2.
But I don’t feel any enthusiasm in my heart about investing in a fresh “green field” project based on Python. I’ve already invested very heavily in Python over the last 15 years, and it has paid off handsomely and is still paying off. But when I consider launching some major new project or investing serious time into exploratory new territory, I don’t want to do that with Python.
Now don’t get me wrong — I’m continuing to invest a *lot* of time and energy into Python and Python-based projects. But these are “conservative” investments — prolonging and increasing the benefits that I’m already getting, rather than taking a big risk in order to achieve something new and great. One consequence of this is that I am extremely averse to any compatibility breakage for this sort of work. One of the biggest benefits I get from using Python today are that I can continue re-using and extending my own code and the code from many others. Anything which hinders that — which narrows down the set of packages that I can use or the set of people that I can share source code with — just sucks the value out of that proposition for me. I think it is _great_ that no new versions of CPython2 are coming out after CPython 2.7, because that reduces friction in terms of source code compatibility and deployment between different users of Python2.
This is why I haven’t gotten over the Python2→Python3 gap, and I honestly don’t know if I ever will.
Thanks for reading!
Regards,
Zooko
Zooko Wilcox-O'Hearn
November 13, 2012 at 6:32 pm
Hi Holger,
what is the problem with yield? I think it’s far more readable than greenlet’s switch.
Regards,
Stefan
Stefan
November 14, 2012 at 7:40 am
Hey Stefan,
“yield” is more limited in terms of wiring communication between program parts. This is showcased by the fact that you can easily implement a “Yield” function, equivalent to what “yield” provides, when using greenlets. See the end of http://bpaste.net/show/h50hFlWrS2ZUuhmFk92E/ . The reverse is not possible (implementing greenlets in terms of “yield”).
For example, you can extend the example to have a stateful “Yield” object which offers an publish/subscribe interface and make it available to framework users. When a value/event is yielded, a number of waiting points in your program could receive it and continue. There are many other possibilities which simply are not possible with the comparatively special “yield” syntaxes.
cheers,
holger
holger krekel
November 14, 2012 at 7:53 am
I think it is possible to do this kind of event dispatching with simple yields. It just depends on the underlying scheduling function. Take a look at this SimPy example http://bit.ly/XE7NE6, where 3 generators (called processes in SimPy) are waiting for the same event. Another approach is the deferredGenerator from Twisted: http://twistedmatrix.com/trac/wiki/DeferredGenerator
I don’t like greenlets because you are responsible to schedule the execution (or may at least modify the schedule). Both approaches above properly separate scheduling concerns from the application logic.
luensdorf
November 14, 2012 at 8:40 am
Hey Zooko,
thanks for your elaborate response, i appreciate it. I still like Python a lot … Between 1990 and 2000 i switched languages, went ever higher level: Assembler, C, C++/Java, Python. I like the power of introspection and a concise clean indentation-based syntax, Therefore, i am not thrilled that much about Rust or Golang. I don’t see the point of encoding machine codes, e.g. the “{}” block delimiters, into a human handled language. I also am not thrilled about going back to have to encode type information with my programming intentions.
I do believe there is a lot of things that could be simplified about Python if you have a better implementation and revisit some design decisions. It breaks compatibility but that’s where a good Python bridge could help. Then again, almost all of what i suggest could be done with Python if using it in a certain way (especially when using PyPy).
In any case, i am not going to immediately start a new project, just wanted to get the ideas out there. I am rather bound to improve testing/integration/ packaging in Python for the time being.
best,
holger
holger krekel
November 14, 2012 at 7:41 am
Holger:
That’s a pretty good response to my negative statements! I guess if your new thing has really strong support for FFI that could help reduce the problems of incompatibility that any new thing comes with. (Like the “bridge” you mentioned.)
Also, if there were some kind of source-code-level compatibility. For example, if your new thing were mostly a subset of Python3.
This latter idea about source-code-compatibility is along similar lines to the fact that the thing that helps me move toward Python3 from my current stuck-in-Python2 world is the existence of the “Six” library: http://packages.python.org/six/
Being able to have a single codebase that works and does the same thing in both languages (no source-to-source translators!) is probably my best chance of ever getting over the gap from Python2 to Python3.
Note that for this hypothetical “source-code-level compatibility” of your hypothetical Python3-like language project, it is *not* at all necessary that all features be handled. It would be fine if there is a subset of the intersection of Python3 and HolgersLang, and only things written in that subset can have the “source-code-level compatibility”. As long as that subset is big enough to be useful. 🙂
Zooko Wilcox-O'Hearn
November 14, 2012 at 6:16 pm
Actually, i am becoming curious of how much can be prototyped in existing Python, using some hacks that could later be properly and more efficiently supported by a new interpreter version. I starting playing with a little module which helps to eliminate the need for process-global state. Stay tuned 🙂
holger krekel
November 14, 2012 at 8:04 pm
Excellent! A very good path forward! Possibly related projects:
Allen Short’s alternative Python importer:
http://washort.twistedmatrix.com/2011/01/introducing-exocet.html
Mark Seaborn’s CapPython:
http://lackingrhoticity.blogspot.com/2008/08/introducing-cappython.html
http://plash.beasts.org/wiki/CapPython
There is also a fairly detailed literature in the object-capability community about doing what they call “Taming”, which is providing wrappers around a language’s standard library so that you can use it without any globals.
There is a really interesting thing going on here, which is that you and people like you, who are presumably thinking about software engineering and sharing and re-use rather than about security, and the object-capability community, who are thinking overridingly about security, happen to meet, and find each other doing *exactly same thing* — eliminating implicit, widely-shared, and mutable state and replacing it with explicit, narrowly-shared, or immutable state.
Caja — object-capability-constrained Javascript and tamed DOM:
https://developers.google.com/caja/docs/about/
Joe-E — object-capability-constrained Java and tamed Java standard library:
https://code.google.com/p/joe-e/
Emily — object-capability-constrained Ocaml and tamed Ocaml standard library:
Click to access HPL-2006-116.pdf
Regards,
Zooko
Zooko Wilcox-O'Hearn
November 14, 2012 at 9:06 pm
April 1st already? 🙂
Mark Lawrence
November 15, 2012 at 10:31 pm
Interesting.
You bring up some issues with the language that I’ve never had a problem with, such as import, sys.modules, setup.py.
You also bring up some topics that add to my ‘to research’ list:
– I’ve been hating ctypes, especially when getting them to work across windows & windows+cygwin=posix.
Therefore, I’ll have to research cffi more. Never heard of it before. So thanks for the tip.
– Greenlets, stackless, … Threads in python… yes please.
However, I’ll add one vote to the ‘indentation based code blocks are stupid’ camp.
I love Python, but use C++ more.
I’ve never really thought that { and } were hard to type.
And I like clicking on one to jump to the other.
… just saying.
Brian Okken
November 16, 2012 at 5:07 pm
As to indentation, maybe a new language should come with two modes, optionally accepting {} 🙂
holger krekel
November 16, 2012 at 5:27 pm
Pushing everything into an event loop isn’t technically really possible without some ugly hacks. There are lots of I/O devices where either asynchronous I/O is simply not supported at all (most disk I/O), or there are practical issues with the support.
Adam Skutt
November 18, 2012 at 1:36 pm
If there is no way a process can tell the kernel to perform unblocking IO, then there is not much that can be done, indeed. Is this true on Linux/FreeBSD/Win7?
holger krekel
November 18, 2012 at 6:55 pm
There are ways, but the UNIX ways are not especially compatible with event loops and not particularly portable. Windows is better in this regard, but it’s still not trivial.
Adam Skutt
November 19, 2012 at 12:58 pm