Writing About A Metal Renderer in Swift Playgrounds
A little while ago I started a project to write a Metal renderer on my iPad in Playgrounds. A brief history is that for various reasons I am often away from my computer but have access to my iPad. I like experimenting with graphics and game development, but this has historically required a general purpose computer. Last year, Apple released Swift Playgrounds 4 in which it is now possible to write and distribute full iOS apps. Prior to this, I had tried a few different avenues of software development on the iPad:
Previous Experiments
I tried some graphics experiments with Pythonista, which is an incredible application, but offers only 2d graphics APIs. The most advanced thing I achieved was an implementation of the Boids flocking algorithm. I also used it to build a live reloading web server that converted python to js. This allowed me to write WebGL on the backend, but it was very clunky and there was no clear path to doing anything with it.
Shaderific offers a way to implement shaders, and I worked through the beginning of the Book of Shaders in this. However, the applications of shader toys were not what I was interested in developing. It’s a useful tool for shader experimentation regardless.
Swift Playgrounds prior to version 4 allowed the creation of Metal views which provides an opportunity for graphics dev. My main focus with Playgrounds was trying to implement Peter Shirley’s Ray Tracing in One Weekend using compute shaders. This was fun, but I couldn’t make enough time to get past a particular hurdle translating the recursive CPU implementation to a GPU one and so shelved it.
Having embraced vim in a big way recently, I used the Blink terminal app to ssh over to my mac and develop my C++/Vulkan project remotely in vim. The Coc plugin turns vim into a passable C++ IDE, though I couldn’t get the debugging to work effectively. Additionally, running the Vulkan app meant running a remote desktop alongside the ssh, but this was not a satisfactory feedback loop. I considered doing the dev through remote desktop, but that never felt quite right.
Swift Playgrounds
I found myself looking for another option, and then Swift Playgrounds 4 came along, providing a way to build iOS & iPadOS apps in addition to the Playgrounds it already offered. The first thing I wanted to know was if it were possible to create a Metal view and draw to that. I was able to do this, and although I had achieved something similar with Playgrounds, being able to turn it into an actual app made it more appealing in terms of time investment.
What to do with this? Well I wanted to make a game with spaceships. I’ll elaborate on why it’s always a space game another time, but why bother trying with such a developer-unfriendly device? The answer to that is simple - it’s the only device I have access to sometimes and it’s spiralled into an idea to build a game entirely on the iPad and distribute onto the app store. The challenge would be interesting, and make a fun story if nothing else. In addition to Playgrounds to write the thing and Working Copy for source control, I could use Voxel Max to make spaceship models (character animation on iPad seems like a dead end, at least at my price point) and GarageBand or SunVox to make the music. I could use Serif Designer and Pixelmator Pro for artwork. Constraint inspires creativity.
With this loose brief, the first thing to do was render a triangle - no problem, there is a lot of documentation on how to do this for Metal. Next up was the 3d math necessary to do vector and matrix maths. I’ve implemented vec3 classes for several projects previously so the standard operations, and dot and cross products were fairly easy. I wanted to add unit tests, but XCTAssert is not supported in Playgrounds 4. So I rolled my own UI that executed the unit tests at runtime. This was pretty cool and I learned a little SwiftUI in the process.
However, as I added more functionality, the compile times increased and it became apparent that Playgrounds is quite slow, particularly when dealing with the generic code I was writing for the vector and matrix implementations. It seems that while it is possible to build apps on an iPad, depending on what you’re trying to do, it might not be actually achievable.
In order to make progress, I looked at the possibility of importing external Swift Packages into Playgrounds. I had seen the option available but not tried it. After some trial and error I found the correct alignment of planets to get this working, and started to move my maths code into a library on Github. The downside to this is that to edit this code I have to either ssh to my Mac Mini, or use GitHub Codespaces (which is like a remote machine whose interface is VSCode). Codespaces are cool and will probably work if I’m out and about, while ssh is better if I’m at home.
I’m not super happy with this arrangement as it means a) not everything it happening on the iPad, and b) it depends on having an internet connection, but it has helped with the issues Playgrounds was having with the generic code. I can apply this approach and extract other areas of code to Github if necessary. One downside of this is it means that the code needs to be public (unless you can put credentials in the repo URL when adding it to Playgrounds, but I haven’t tried this). One upside of this was that for Playgrounds to make use of it, the source needs to be tagged with semantic versioning and I had to come up with a way to get this done using Github Actions. I rolled my own vector and matrix classes, but in retrospect adding extensions to the simd types might be more prudent. Something to revisit later! The code is available on GitHub at andystanton/swift-math.
Conclusion
The status as of 2022-05-19 is I have an app that can draw a triangle and can use the beginnings of some 3d maths code from a Swift Package I’ve written on Github. Next steps are to add the unit tests into the swift-maths package and add more functionality so it can be used to make 3d scenes.
This entry is part of a series on writing a Metal Renderer in Swift Playgrounds.