What would it take these days to build a prototype of a new language and operating system built on top of it, just for fun?
This idea started to emerge during the 2018 winter holidays:
Woot! I just hacked the "Building an Operating System for the Raspberry Pi" https://t.co/tSvS8Bz0ft by writing the kernel in C# and using the AOT Compiler with a slightly modified version of CoreRT to compile it to ARM, and... it's running! I'm super happy, it is so much fun! 🤗 pic.twitter.com/lM3MrH04Tk— Alexandre Mutel (@xoofx) January 2, 2019
This brought me back to my teenager time when I had so much fun running assembly code directly on Amstrad/Amiga computers without using an OS... time has passed, and as I was also re-reading all the fascinating collection of blog posts of Joe Duffy about Midori and also discovered through some discussions on Twitter that Google has been actually working on a new OS called Fuchsia with its micro-kernel Zircon, it made me realized how much I would like to reinvestigate these areas...
So one year ago, I decided to re-focus the work on the stark programming language experiment and to extend its goal by building, not only a safe and efficient system programming language, but also a fast, secure and lightweight micro-kernel and capability based operating system, IoT and cloud ready (game OS also, why not?), async/await event/driven based, but also built with data oriented techniques...
There are still large and exciting areas to explore and challenge in this domain, as much as I'm eager to be technically challenged, learn and share from this experiment!
Though while I have been enjoying the early design/prototyping of stark, It became also obvious that starting entirely from scratch a whole compiler would not allow me to make enough sustainable progress and to keep enough motivation during my spare-time on the long run...
Luckily, thanks to the development of OSS, we can rely on the shoulders of existing giants to experiment and prototype more quickly what would have been impossible in the past... So during the past year, with the help of the .NET ecosystem and the seL4 micro-kernel, I was able to build a vertical prototype of the language, its core library, its front-end compiler, a native compiler and an embryo-integration with seL4:
Finally, here is a capture of the boot sequence of a HelloWorld program with #starklang on top of the seL4 micro-kernel.— Alexandre Mutel (@xoofx) February 10, 2020
This marks the end of the 1st vertical prototype by bringing up together a front-end compiler, native compiler and micro-kernel integration! ❤️ pic.twitter.com/kEQ2esGHj4
Though, apart sharing my progess on Twitter, I haven't taken the time so far to explain what this is all about... So before continuing further this experiment, It is time for me to share more about this project, and more specifically to:
- Give more details about the "why?" of this project
- Explain how this first milestone prototype was built
- And where is it going?
Stark - The language
I have realized that I never wrote down at least some high level requirements and goals for this language. They might have changed slightly since I wrote "Going Native 2.0, The future of WinRT" or more recently about stark, but in the end, they are tactical tradeoffs between the 3 following pillars:
Note: Stark is not a fork of C# nor it can be compatible with .NET
Many of the following features are requiring such significant breaking changes that they can't be retro-fitted into an existing platform like .NET.
As a programmer, this language should help me to:
|Impact on the language
|Disallow dangling references to objects.
|Remove the possibility of null reference.
|Avoid race conditions related to multi-threading.
|Remove the possibility of uninitialized state for objects.
|Don't allow arithmetic values to overflow, by default.
|Clear error model
|Design by contracts.
|Impact on the language
|Compile time optimizations.
|Zero cost for const data.
|Provide control over data locality.
|More explicit control over heap allocation and lifetime.
|Lightweight async and await.
|Minimize object runtime cost and footprint.
|SIMD and Low level.
|Impact on the language
|Functional, object and data oriented programming.
|Unified and integrated development experience.
|Concise syntax but not cryptic.
Melody - A Micro Operating System
Melody is the name of the Operating System that will be built with the Stark language.
Why developing an operating system while you would have already enough work for the rest of your entire life with just the language?
This experiment is first an opportunity to enjoy again hacking "bare metal" low level parts over the hardware, to revisit foundations that I took for granted (resource management, memory management, interrupt handlers, processes/threads lifecycle, kernel vs user code...) and to better understand the constraints of designing an operating system.
Secondly, using the language in a low-level situation should hopefully help to drive and focus the implementation of its requirements. But this should be also a two-way feedback process, where the language could benefit some OS features and vice e versa.
Lastly, I can't resign myself that the 2 existing dominant operating systems should obliterate the need to look for a better architecture or that the future of OS development will remain in these two solely. Even if the existing OS have been able to adapt some modern challenges (e.g Web, hypervisors, containers...), they feel still very brittle in their overall monolithic kernel approach, specially when security is involved. In this era of rising IoT, it feels even more surprising that object-capability based OS are still not mainstream. Also, I believe that It was a mistake to abandon the project Midori.
Melody will be driven by the following 3 pillars:
- Capabilities should be at the heart of managing HW resources and used for connecting services.
- All drivers/services/apps should live in user land.
- The operating system should be built on a secure micro-kernel.
- Should scale well from IoT to Cloud.
- IPC between services should be fast.
- Async based events should be the primary communication between services
- Processes should not be backed to disk (no-swap) but instead use a resume/suspend/restore lifecycle.
- Footprint of a HTTP client/server app with the OS should be minimal (a few Mb)
The 2019 prototype
The overall goal of this first year prototype was relatively simple:
A HelloWorld x86_64 Stark program running on a bare metal "OS" similar to the one I did with C# and CoreRT for the Raspberry Pi
The steps to achieve this goal would involve:
- Prototype the syntax of the Stark language
- Simple IDE integration (syntax highlighting)
- Use the Stark language entirely for both the HelloWorld program and its core library (so no additional C runtime on the side)
- Implement a front-end compiler for translating Stark to an intermediate binary library (e.g Assemblies in .NET)
- Implement a back-end compiler for converting the binary library to native code
- Prototype the micro-kernel with the minimum required to bootstrap the HelloWorld program
After hacking C# with CoreRT on a bare metal OS on the Raspberry Pi, I realized that the .NET ecosystem could help my experiment a lot more than I had planned when starting to work on Stark. Instead of trying to rebuild everything, let's try to find and reuse components that could significantly boost this enterprise:
For the front-end compiler, instead of fully building bottom-up a tokenizer/parser/syntax analyzer/type inference/transform to IL, why not starting from the Roslyn C# compiler instead?
For the back-end compiler responsible to generate native code, I originally thought that I could rely on CoreRT, but realized that the design of the compiler and runtime would be so different that I should proceed differently... and came to the conclusion that the most critical component I could reuse was the RyuJIT compiler for the IL to native codegen part. And I will explain more in details why.
For the OS kernel part, I was initially expecting to write the kernel in Stark itself, but while searching about what it would take, I discovered that the seL4 micro-kernel was exactly what I was looking for for a micro-kernel, and I would have already plenty of work to do to build a proper OS on top of it. We will see why I'm very enthusiastic about it.