Why I built my own programming language

I started working on my own programming language about a year ago. I released the first public version about six months ago. I think I've gone through about 75 releases since then. Some have been small patch releases, fixing a few bugs. Some have been much bigger, richer, feature-heavy releases (which have also fixed bugs).

Even in the age of frontier coding models, designing and building your own programming language is not a small or simple undertaking. It takes a lot of time to get the design and mechanics right. Appealing syntax and a rich standard library are only part of it. The engine underneath has to be halfway decent too. An engine that is usable for real workloads and real purposes. A speed that is at least ballpark-comparable to its major, much more widely known and deployed competitors.

When I first set out to build my own programming language, I wasn't trying to build something too serious, though I did want it to be usable. It was a learning exercise for which I thought the latest AI models would be useful as a guide, to go more in-depth into interpreter design, to refresh and enhance my knowledge on lexers, parsers, ASTs and all the rest of it. I chose to develop in Go, so that I could learn Go by forcing myself to go through and understand what a model which already knows Go was spitting out as it followed my instructions.

The first iterations of Geblang, pre-public release, were very simple. It had limited language features and was fully interpreted, in the sense that each script executed or imported was evaluated one statement at a time by a top-down tree walker. The evaluator engine still exists in Geblang today but has long since been superceded by the stack VM, the default runtime through which all programs are executed and which re-uses validated caches of compiled bytecode.

I always had the VM in mind, from day one, but the first iterations were heavy on learning and I'm not a very patient man; I want to be able to run programs, play around and see results, fast. So that's why I started with the evaluator and worked my way up.

Today, Geblang is a serious language that I actually use to run a couple of real web services. It's sufficiently performant, richly featured and holds up under the workloads I demand of it.

Best of all, I get to develop my way, in the language I designed to be able to program the way I want.

So...why? What is it about PHP, Node, Python or indeed any of the many alternatives that I felt weren't the way I wanted to program? Why go beyond the tree-walking evaluator? Why keep working on Geblang aiming for more than a little home-education in building an interpreter?

Dynamic typing needs to (almost) go

TypeScript exists because JavaScript's typing is so loose that these days, it's barely considered a usable language in its own right.

PHP has and continues to gradually shift to typed code being the default and more runtime type-safety guarantees being introduced in every new version.

Python won't enforce any type hints in your code, but writing your Python with type hints and having it checked with mypy is now the de facto standard.

Dynamic typing lets us effortlessly re-use the same code with many different types; something that's especially useful if you're building applications or APIs for web. It lets us write concise code focused on business logic without ceremony.

It also opens us up to all manner of silent bugs, bugs that only become apparent in production, errors or corruption by conversion and because these bugs often depend on specific state, their cause might not always be readily apparent.

So I asked myself, why not an interpreted language that's statically typed from the start, with a mechanism to opt-out of type checking only when you need it? Ah, that's TypeScript, you might say. But TypeScript loses its type information the second it transpiles to JavaScript. I wanted a language where type information is retained at runtime, where checks like instanceof list<User> work.

So Geblang's any type is inspired by TypeScript. Geblang was all about taking the best bits of other programming languages I've used and combining them into what I see as the ideal experience.

PHP - bureaucracy and historical baggage

I like PHP. I think in the PHP 8 world, most of the criticisms and jokes of old about it being a bad language, not for real programmers, not for serious websites are now firmly refuted by modern versions.

But we can't pretend there aren't still some problems. For me, the two biggest are the inefficiency, bureaucracy and capriciousness of the RFC process which makes introducing new features hard, and the commitment to backwards-compatibility which makes correcting historical mistakes nigh-on impossible.

PHP will likely never have primitives as objects. You will always be calling strlen($string) and never $string->length(). You will never be calling $list->filter(...) instead of array_filter($list, ...). You will never have lists, sets and dicts as separate types.

There are features I wish PHP had that it doesn't, and maybe never will. There have been maybe two or three serious attempts to add generics to the language, all of which have failed either for technical hurdles or because they were voted down by the internals community, or both. Operator overloading was proposed, implemented and rejected.

Some attempt at native async via greenthreading has been implemented in Fibers, but PHP remains behind with user-friendly async and await. Parallelism for CPU-bound work is off the cards without extensions.

Attributes provide metadata via runtime reflection, but there's no native or idiomatic way to make use of aspect-oriented programming, like Python's decorators.

Websockets? Don't even go there.

Then there's deployment. The interpreter, PHP-FPM sitting behind Nginx, mod-php sitting behind Apache (eww), or modern solutions like FrankenPHP. The latter are great, but you're still having to manage deployment of scripts and some kind of runtime. I want to just be able to upload a program and run it, webserver included. The lack of a production-grade built-in server in PHP that you can build or instantiate in code yourself.

Whenever I work with PHP, I feel like I'm tantalisingly close to a language that could easily deserve to be the preferred solution over Node, Python or Java for anything running on HTTP, but just falling over at the final few hurdles.

Geblang aims to correct all this. Async tasks run in goroutines and leverage the full capability of your CPUs. A websocket server and HTTP server can co-exist in the same program. Language-level features like generic types, operator overloading, function overloading or interfaces with default methods are present from day-one, alongside a familiar C-inspired syntax (curly braces, not whitespace), a module system improved over namespaces and PSR-4, and all the features you actually do like, old and new - full OOP support, a standard library geared for web, function partials, enums, match syntax and much more.

No thanks, Node.

I don't know who first came up with the concept that a language which was specifically designed to manipulate pages in a browser was a good or even vaguely reasonable choice for writing server-side code, but I feel like they've successfully pranked the whole world.

These days, of course, TypeScript is the preferred layer in between your code and the runtime. And poor as JavaScript may be for authoring code, Node is great as a runtime for serving requests where I/O bound work can be offloaded to libuv. That's what draws people in; they can async and await, fire and forget, move on to the next request. You want a websocket server in Node, no problem, that's a few lines of boilerplate plus your business logic. No one can deny the advantage over PHP there.

What I don't like is the Node ecosystem, the endless, convoluted pyramid of dependency management, the vulnerabilities, the package upon package upon package for the simplest of tasks.

This made me realise for my own programming language, I wanted "batteries included" - a phrase I hate, but an important standard for what Geblang had to provide out of the box, without downloading anything else. All the tools you need for real work, from CLI and shell scripts, to webservers with TLS, cryptography, websockets, databases, maths and statistics, data manipulation, file parsing, logging, emailing - all the stuff you have to actually do in a real-world web service. Geblang gives you all this, built-in.

Python - elegant but flawed

Geblang borrows a lot of concepts from Python. Lists, dicts and sets that are actually different things. Comprehensions. Generators. Decorators. Context managers for resources you want cleaned up reliably. Dunder methods for defining how your own objects behave with the language's operators. These are all useful ideas, and they're useful together. Once you've got used to expressing a transformation as a comprehension, going back to building an empty collection and filling it in a loop feels a bit tedious.

I wanted all of that, I just wanted it with braces and real types.

Whitespace as syntax has never appealed to me. I'm not a zealot about it and if you work with Python, you get used to it, but significant whitespace is not one of my ideals. Geblang gave me the creative freedom to choose and I chose curly braces.

The typing issue is more substantial. If I've said a function takes a list of users, I want that to mean something when the function is called. Ideally, I want a typing error called out to me before the program even runs. If I'm inspecting a value at runtime, I want the language to remember what it is supposed to contain. I don't want the agreement between my IDE, my type checker and my runtime to depend on which tools I've configured and whether everyone remembered to run them.

But another idea from Python was that a scripting language should be good at the boilerplate stuff and that feeds back into Geblang's rich standard library.

So the Python influence is fairly obvious once you start using Geblang. It's there in the collections, in the way you can customise objects, in the little conveniences that make a script pleasant to write.

Go, great foundations, different priorities

Geblang's interpreter is written in Go. So why not just write my applications in Go too? Hey, it's a good choice, no argument from me there. Go's standard library and immediate, open source ecosystem gave me a lot of the machinery I needed.

But Go also has very different priorities to me in how we shape our code. I like classes. I like inheritance where it makes sense. I like exceptions, operator overloading and decorators. I like being able to put together a small script with some top-level code and get on with it.

Geblang's defer keyword is a concept I borrowed from Go, but mostly what I took as inspiration was deployment. Building and deploying a single binary is something I wanted for Geblang. The difficulty there, of course, is that Geblang is not a native compiler, but I didn't let that stop me. Geblang's build command bundles the interpreter, its standard library and your own application code (and any dependencies), pre-compiled into VM bytecode, into a single executable which is automatically extracted out on the target machine. No mess, no fuss.

Putting it all together

So every language on this list had different features that I wanted and was missing features that I wanted. No language had it all.

The difficult part, as it turns out, isn't coming up with a list of features you like. It's making them agree with each other.

Generics are fairly easy to put on a feature list. Making them behave consistently through inheritance, reflection, decorators and an imported module is hard, even when Opus is being directed to write the actual Go code for you.

Every convenience has to survive contact with the rest of the language surface. Most of the bugs I find in Geblang these days are more obscure edge-cases, usually involving cross-module interaction, things like subtle issues with base classes imported from somewhere else. But I still find and fix bugs all the time.

This is also why I've kept the evaluator around. Having two implementations of the same language gives me something useful to compare. I use it as a sandbox to build and test new features and only consider them done when the evaluator and VM can both agree on them.

Working with AI hasn't removed the need to understand what I'm building. Whether something fits the design, whether it breaks something three layers away, those questions still need answers. The learning for me hasn't stopped.

I don't expect everyone to share my preferences. If you like PHP and Python's simplicity, Go's approach to composition or the Node ecosystem, you're probably looking at some of my complaints and wondering what the problem is. Fair enough. You might also quite reasonably decide that an established language with a huge community is a better choice for your next project.

For me, though, the result is something I actually want to use. I can write a typed script, use the object-oriented and functional features I like, serve HTTP and websocket requests, and ship the application as an executable. I find remaining problems by using my own language, and when I find a problem, I can go and fix the language engine.

That last part is probably why I kept going. It started with wanting to understand in more depth how an interpreter works. Today, I'm asking what I can build with it.

An example Geblang script

If you haven't seen the programming language I've created and have no idea what all this actually looks like, here's a quick example you can run yourself (with apologies I haven't built a Geblang syntax highlighter for this blog yet) - io and money are stdlib modules:

import io;
import money;

/* A simple class demonstrating generics and an overloaded + operator */
class Booking<T> {
    list<T> lines;

    func Booking(list<T> lines) {
        this.lines = lines;
    }

    /* Defines booking + extra of type T, returning another Booking<T>. */
    func __add(T extra): Booking<T> {
        /* Lists share storage; copy to leave the original booking unchanged. */
        let lines = this.lines.copy();
        lines.push(extra);
        return Booking<T>(lines);
    }
}

/* Python-style decorator, but in Geblang, decorators that are not defined functions simply become reflection metadata, much like PHP's attributes, so you get the best of both worlds */
func audited(callable next, string label): callable {
    /* Variadic arguments let one wrapper handle both quote signatures. */
    return func(any ... args): any {
        /* Forward the arguments, then log and return the original result. */
        let result = next(...args);
        io.println("${label}: ${result}");
        return result;
    };
}

/* This overload handles a unit price and quantity, with audit logging (well, imagine audited was a real auditing process instead of a print line). */
@audited("Tickets")
func quote(money.Money unitPrice, int quantity): money.Money {
    /* Money overloads * to return a currency amount, rather than a float. */
    return unitPrice * quantity;
}

/* The same function name also accepts different parameters of different types, via overloading. */
@audited("Booking")
func quote(Booking<money.Money> booking): money.Money {
    return money.sum(booking.lines, "GBP");
}

int guests = 6;
let tickets = quote(money.Money("19.99", "GBP"), guests);
/* Bind the generic T to Money for this booking. */
let booking = Booking<money.Money>([tickets]);
/* Price six meals or whatever it is, add that line with our custom operator */
quote(booking + money.Money("12.50", "GBP") * guests);

Try it out

If you'd like to give Geblang a try, visit the official Geblang website to learn more, see example code, read the full docs, download pre-built binaries for Linux, Windows and Mac, or build and install from the source on GitHub. Geblang is working for me; I hope it can work for you too.

Comments (0)

No comments yet. Be the first to share your thoughts!

Leave a Comment