Yet Another Rant About AI Security OSS Externality
IMPORTANT DISCLOSURE: This blog post is written in my personal capacity reflecting on the recent Apache Spark patch releases (3.5.9, 4.0.4, and 4.1.3). These are not the views of the Apache Software Foundation (ASF), the Apache Spark project, or any of my employers past or present. Both the ASF and the AI research lab have been provided an opportunity to comment on an earlier draft of this blog post and correct any information they see as incorrect and while many individuals have provided feedback in their personal capacity, there is not (as of yet) any official comment from either.
AI has fundamentally changed the volume of security reports in all projects, especially OSS projects where the barrier to analysis is even lower. We need to revisit how we handle security disclosures in the age of AI-driven pull requests and security auditing. Open Source Software (OSS) has long been for the community, and while some of us have jobs which give us time to work on open source, it's most often largely a volunteer effort for the public good, with the ASF focused on Software For the Public Good since it was incorporated in 1999. Security is of the utmost importance and requires greater care in collaboration to avoid accidental disclosure. For security teams, when working with OSS projects, please be flexible on release timelines when issues are not actively being exploited. If we have to move our release schedule to meet your planned disclosure date, it's more likely to cause us to ship more security issues rather than fewer.
While some labs have early access programs, in my experience they're not working. Despite there being an agreement for early access to one of the labs' "frontier security models", we only got access to a scan in August. While a team at that same lab, that we'll call Team X, is, seemingly, using their frontier models from at least February to report security bugs and issue deadlines for resolution and disclosure.
The old model of "here's a bug, you have ~90 days to fix your issues before we publish how to exploit your code" came about because software vendors, mostly closed source but some open too, were in the habit of ignoring security reports (or being openly hostile to them). Previously security reports took a human to make, and we'd get a few every month, but now with AI our security report inbox is looking more like the promotions tab in my Gmail. Additionally, more bug fixes are landing later in the release candidate cycle from AI driven PRs, so these previously reasonable hard security disclosure deadlines are making it more likely we ship worse security bugs while we try and play catch up. To that end, I want to call on all frontier AI labs to put more funding into mitigating the impact of their tools on the OSS projects which are vital to their success.
To put it in “corporate� the key asks from this entire post are fairly simple:
Frontier AI labs should provide real usable access to their frontier models for OSS
AI labs should be careful in teams using their models in house for security in advance of their own programs for “trusted� access.
More flexibility on disclosure dates is needed given the impacts of AI
A pony with attached coffee cup holder^ (and unlimited coffee) (jk)
The core of the issue remains that reviewer bandwidth is the scarcest thing in large open source projects. While tools like CodeRabbit are amazing (big shout-out to them for their OSS plans), they still don't capture everything. Thanks to the increasing frequency of attempted (and sometimes successful) malicious PRs and takeovers of OSS projects, we can't just leave it to the robots to argue amongst themselves. Maintainer burnout is a real security risk that we as an industry are not prepared to handle.
While these suggestions are only a starting point, the big AI labs can do a number of things to mitigate some of the downsides. One is providing usable non-time locked free accounts to the people whose software they used. For some of the bigger projects, the AI labs have staff who are able to do things like manage releases, but they're often too busy working on internal issues to have the time to work upstream. I think giving engineers something akin to Google's "20% time" (which some jokingly called 120% time) to give back to the projects they're using could help.
I want to double down on the "usable" part. While Anthropic has done better than any of the other AI labs at trying to provide free tooling for OSS developers, their effort still doesn’t do enough. In trying to use my Anthropic provided OSS maintainer account on a recent Spark security issue I tripped its safety guards and eventually ran out of tokens*. In practice, I've found paid inference on open weight models** to be more effective at addressing security issues. Most OSS maintainers, including myself, don't yet have usable direct free access to the frontier security models, meaning we're pushed into a reactive mode rather than pro-active.
For me these issues came into focus during the recent ASF Spark 3 and 4 patch releases, available everywhere fine bits can be downloaded. Most impactful was a security issue reported by Team X where because of a combination of a lack of communication in the initial disclosure and lack of flexibility on the surprise deadline once the release managers became aware, we had to scramble for extra reviewers on complex separate security issues.
Team X reported security issues in the Apache Spark project around February 25th of this year. This report was (thankfully) accompanied by a human-reviewed proposed patch and no specific deadline in the initial report, which to be fair is better than we would get with a frontier model alone. So far, so good.
However, out of band, Team X then gave a vendor who distributes and contributes to Apache Spark (not my current employer) 90 days to fix this, or Team X would disclose the issue publicly. While the vendor requested an initial extension to line up with the planned OSS release schedule, which Team X granted, the deadline was not communicated to the broader OSS group, including myself, until much later.
While working on the most recent releases, we got roughly 30 post-first-line filter security reports in the week before the first patch release candidate, a few of which turned out to be actionable. Part way through the release cycle, I became aware of the deadline from Team X and reached out to the head of security at the lab to further extend the deadline so we could address the outstanding security issues. Some of these issues were blocked on knowledge of specialized libraries I personally don't have. Initially Team X was unwilling to extend the deadline any further, but when I explained how our 72-hour release voting normally works, they were willing to grant one extra week if the vote failed. During this negotiation, the amazing ASF security folks found a reviewer for the complex issue I was wrangling. We were able to get known issues fixed and the release candidate vote thankfully passed, but that outcome was far from certain.
If we had not been able to find reviewers for the encryption fix, we would have had to ship a partial release with a known broken encryption issue. At the end of the day, Team X has still not published the disclosures as of the end of September, strongly suggesting that they could have given us more time. As a release manager, this is of course frustrating and had personal impacts on my vacation coffee consumption being more necessary rather than enjoyed, but the impacts could have been much worse (not just missing out on limoncello).
I'd also love to give a shout-out to my point of contact at the frontier AI lab for helping unravel the game of telephone and getting me that extra week.
I don't think this problem is unique to Team X, the process is broken and compounding the existing reviewer constraint. Reviewer bandwidth has almost always been the biggest bottleneck in OSS projects, and the decreasing cost to generate code will continue to push against this bottleneck. When issues are discovered in OSS projects that are not actively being exploited, requiring releases from the OSS project with a fixed timeline is counterproductive. Release cycles are likely going to keep getting longer (see how big the latest Linux RCs are), and we should allow time for this. For the humans driving the robots, when there is a public interest-reason like an active exploit, not just a conference or PR opportunity, for providing a disclosure deadline, make sure to include that in your initial report whenever possible.
I'd like to end by reiterating my request to all of the frontier AI labs: please offer truly usable free plans for OSS developers, be flexible with your disclosure deadlines when no active exploits exist. In addition, there are some amazing engineers at these labs, if you give your staff time to participate in OSS maintenance tasks it could make a big difference. There's a strong history of industry and OSS working together in both ASF and other projects from Spark to PyTorch to the Linux kernel. That and a pony with a coffee cup holder and a coffee, but I'll take more tokens in place of a pony (I still want the coffee though :p).
I am well aware this is a bit of a “do as I say not as I did� situation as I did report some Yahoo! Mail security vulnerabilities purely through side channels back in University almost two decades ago. But I was young and did not have the resources of a multi-billion dollar company behind me, and I would argue the security implications of unencrypted e-mail passwords are a bit different than data processing software with schema inference driven class instantiation vulnerabilities.
* I try to use the account for other OSS work, not just security. It is more successful for non-security work.
** I don't currently have enough vRam to run these on my own hardware, but if anyone has a few hundred spare RTX5090s or DGXs and a few 20a circuits give me a call :p
^ Did you know that's a real thing? I mean not the unlimited coffee, but they make coffee cup holders for saddles. In practice since I live in a city and do not have space for a pony let alone the skills, I ride a Tiger 900 (without coffee cup holder) and an LX150 (with coffee cup holder). If the AI labs are reading this: coffee cup holder for motorcycle with unlimited coffee is an acceptable substitute 😛
A series by award-winning engineer Corey Boyle in the Edinburgh Inquirer. Well reasoned and well written. Some of the cited facts blew me away. ("Road traffic is the single biggest killer of 5 to 29 year olds on earth, and kills around 500 children every day. In 2024, 161 people, three of them children, died on Scotland’s roads, up 13% in a year. Twelve were in Edinburgh and the Lothians.") Spotted on edi.bike, via the Edinburgh Minute.
Corey Boyle’s three-part, evidence-rich series for the Edinburgh Inquirer on pushing the city towards a more sustainable future wrapped up this week with some practical approaches that could move the needle.
As always, it’s been refreshing hearing a new and well-informed voice calling for change - links to all three chapters are below, on the Inquirer’s SubStack:
The car was sold to us as freedom. For most of us, it has become the opposite, without us even realising.
There is a name for that. Psychologists call it motonormativity, or “car brain�: the unconscious habit of excusing from driving the harm we would never accept from anything else. Once you see it, you see it everywhere.
Dr Patrick Miner at the University of Edinburgh has helped calculate the human cost. The result? The car has killed somewhere between 60 and 80 million people, around the same as both world wars combined, and it injures roughly 102 million more every year.
Over three parts, I want to take apart the car-centric society we accept as normal, confront Edinburgh’s disgrace, its poverty, and show that a cycling city, where the bike is how most people make most journeys, is the healthier, fairer alternative.
Caption to the photo above.
The same 69 people, and the road each takes: one bus, 69 bikes, 60 cars. Sixty, because that is how many an average 69 people fill: Credit: We Ride Australia, 2012, recreating a City of Münster original
The Fibonacci sequence is so-called because in Western mathematics is
first appears in connection with a problem from Fibonacci's 1202 book
Liber Abaci. Fibonacci asks us to consider a rabbit farm where a
pair of baby rabits grows to adulthood in one month, and an adult pair
of rabbits gives birth to a new pair of babies after a month.
Representing an adult pair with a and a baby pair with b, we could
write this as:
b → a
a → ab
Starting with one pair of baby rabbits, and performing the indicated
substitutions month by month, we get:
b
a
ab
aba
abaab
abaababa
abaababaabaab
abaababaabaababaababa
abaababaabaababaababaabaababaabaab
abaababaabaababaababaabaababaabaababaababaabaababaababa
…
The total number of symbols (and pairs of rabbits) each month gives us
the Fibonacci sequence, as does the total number of as, or the total
number of bs, in each line.
A structure like this one, in which there are symbols, in this case
a and b, that are repeatedly replaced with strings of symbols, is
called an “L-system”, and this L-system in particular is the “Fibonacci L-system”.
Yesterday I happened to mention Zyzzyva, which is a genus
of weevils that I can only assume was named by a biologist who wanted
to be last in the dictionary. When I see a pattern like this I
instantly think of the Fibonacci L-system. We can use the Fibonacci L-system to generate a sequence
of increasingly long weevil names:
I had to cheat a little here because there's an a in the middle that
I didn't substitute. If we don't skip that we get the unpronounceable
“Grmammandmammamammamma”.
A quick computer search suggests several other variations, including
both “mammaries” and “titties”, but to prevent this article from
devolving into juvenile sniggering, I'm going to cut the scroll here.
Something tricky happened a couple of articles ago that I
wish now I'd explained more carefully. The recent articles talk about
“shortlex order”. But “shortlex order” isn't really just one order.
It's actually a method for turning one order into another, fancier
one.
The definition I gave said:
The rule to compare two strings in shortlex order is:
If the strings are different lengths, the shorter one comes first.
If they're the same length, compare them lexicographically.
But there's something hiding in that “lexicographically”. To compare strings lexicographically,
you have to compare them character-by-character, and that means you
already have to have some idea of the order of the characters
themselves. If we don't notice that “sort these into lexicographic order” is
ambigous, it's only because a particular standard ordering of the
characters is implicit.
But lexicographic and shortlex orderings aren't restricted to strings of
characters. They can be used to extend an ordering on
any kind of thing to an ordering on sequences of that-kind-of-thing.
If the things are characters, they extend the ordering of the
characters to an ordering on strings, which are sequences of
characters. But the things don't have to be characters.
In the previous article we took the ordinary ordering on
finite numbers (note, not characters) and used the shortlex
constructin to produce an ordering on sequences of finite numbers:
Replace any one sequence of [finite number] coordinates with one
that is earlier in shortlex order.
The phrase “in shortlex order” there is implicitly assuming the usual ordering
of finite numbers. If we had decided to start with a different
ordering of finite numbers, the shortlex extension to
sequences would be different.
A of couple of episodes ago we saw an ordering of the positive
numbers that put all the odd numbers before all the even numbers. Since that basic
ordering puts before , the shortlex extension of it would
put before . And because the basic ordering also
put before the shortlex extension would put before .
Digression
For wacky fun, put the list of words in the dictionary into a lexicographic order
that is extended from a different alphabetical order than the usual
one!
For example, suppose the alphabet was in the order QWERTY UIOP ASDF
GHJKL ZXCV BNM. The dictionary would start with “queer”, then all the
other QU words up through “quandaries”, finish out the Q section
with a few non-Qu words like “Qantas”, and then continue with the W
words. The last word in the dictionary wouldn't be “zyzzyva”, it
would be “mnemonics”.
Is your kid having trouble learning to alphabetize? Try alphabetizing
this random list into QWERTY order to help yourself feel
sympathetic:
Shortlex isn't an order, it's a method for elaborating a simpler order
Say we have some set of things, and an order relation on ,
which we'll call .
We use the symbol “” rather than “” to emphasize that
this is not necessarily the usual ordering. We still write
to mean that comes before in whatever order
represents, and analogously we can write
means the same: comes after .
The notation means the set of all finite sequences of
elements of . For example, when is a character set, is the
set of strings over that character set. Elements of are not at
all the same kind of things as elements of . Elements of
are something-or-others, maybe melons or something — let's say
melons, melons are fun — and if melon is rounder
than melon . Elements of are sequences of melons.
If and are two sequences, you can't sensibly ask whether
or , since it doesn't make to say that one
sequence is rounder than another. is meaningful for comparing
melons, not sequences.
We can't use to order elements of .
But we can lexicographicize to make a new ordering that makes sense for
. To lexicographically compare two elements of , that's two finite
sequences of elements of , whatever they are, do this:
Compare the corresponding elements of each sequence, using
, starting with the first elements
If two sequence elements are different at some posision, the sequence with the -earlier element
comes earlier
If the two elements are the same, move on to the next
corresponding elements and continue similarly.
If one sequence ends before the other, then that one comes earlier.
If both end at the same time, the two sequences were exactly
the same, so neither
comes earlier.
This turns into a lexicographic elaboration of that makes it
work on sequences. This is the same transformation that turns an ordering
on characters into the derived ordering for strings.
We can turn into an ordering for sequences in another way,
the shortlex way, producing an order I'll write as :
If the sequences are different lengths, the shorter one comes first.
If they're the same length, compare them lexicographically as above.
This is the same transformation that turns the regular ordering on digits () into the derived ordering for ordinary base-10 numerals.
I thought we'd get to today, but it seems not. Here's
a preview of the next article, though: We're not going to stop with
, we're going to go on to and .
Claude wanted to help with this article, but I wouldn't let it.
Well-Typed are happy to announce Eventlog Live, a program that streams real-time telemetry from any Haskell application to any observability platform that supports the OpenTelemetry protocol, such as Grafana Cloud, Prometheus, or local viewers such as otel-tui and otel-desktop-viewer.
In this blog post, we’ll show a variety of ways to use Eventlog Live. For each showcase, we’ll link to the relevant instructions using Eventlog Live version 0.10.0.0 as well as the script that was used to generate the showcase. If you would like to use Eventlog Live, we recommend following the up-to-date instructions in the README of the latest release of Eventlog Live.
Heap Profiles by Closure Type
The Grafana Heap Profiles dashboard, which shows a heap profile broken down by closure type (-hT) as well as a variety of metrics for memory usage. For detailed instructions, see Eventlog Live – The Basic Version. This showcase uses the oddball-with-pipe example with the Docker Compose setup in demo/docker-compose.yml.
The Grafana Heap Profiles dashboard, which shows a heap profile broken down by closure type (-hT) as well as a variety of metrics for memory usage. For detailed instructions, see Eventlog Live – The Basic Version. This showcase uses the oddball-with-pipe example with the Docker Compose setup in demo/docker-compose.yml.
Eventlog Live
Eventlog Live analyses your application’s eventlog and exports it via the OpenTelemetry protocol.
Eventlog Live is lightweight, runs side-by-side with your application in only a few megabytes of memory, and requires no instrumentation other than the -threaded and -rtsopts build options.1
All you need is somewhere to send your data.
For instance, if you have a Grafana Cloud2 account, adding telemetry to your application is as easy as:
# Create a pipe for the eventlogEVENTLOG_PIPE="/tmp/eventlog.pipe"mkfifo"${EVENTLOG_PIPE}"# Start your applicationyour-application +RTS -l-ol"${EVENTLOG_PIPE}"-hT--eventlog-flush-interval=1 -RTS&# Start eventlog-live-otlpOTEL_SERVICE_NAME="your-application"\OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$(uuidgen)"\OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-<YOUR_REGION>.grafana.net/otlp"\OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic%20<YOUR_AUTHORIZATION_HEADER>"\eventlog-live-otlp--eventlog-file="${EVENTLOG_PIPE}"-hT--eventlog-flush-interval=1
Out of the box, Eventlog Live supports -hT and -hi heap profiles3 as well as a variety of memory usage metrics, cost-centre stack profiles, log messages, productivity, detailed thread & capability usage traces. Each of these features is showcased below using an example Grafana dashboard. These dashboards can be found in the repository under demo-grafana-cloud/grafana-dashboards and demo/config/grafana-dashboards. Click any showcase for more information.
The Grafana Heap Profiles dashboard, which shows dotted red lines on each visualisation at the times that correspond to the markers, emitted using traceMarkerIO, that match the regular expression Summing \d+ numbers.
This showcase uses the oddball-with-hi example with the Docker Compose setup in demo/docker-compose.yml.
Productivity
The Grafana Threads dashboard, which shows the cumulative productivity per capability, as well as the overall productivity over a sliding window.
This showcase uses the oddball-with-pipe example with the Docker Compose setup in demo/docker-compose.yml.
The Grafana Heap Profiles dashboard, which shows dotted red lines on each visualisation at the times that correspond to the markers, emitted using traceMarkerIO, that match the regular expression Summing \d+ numbers.
This showcase uses the oddball-with-hi example with the Docker Compose setup in demo/docker-compose.yml.
The Grafana Threads dashboard, which shows the cumulative productivity per capability, as well as the overall productivity over a sliding window.
This showcase uses the oddball-with-pipe example with the Docker Compose setup in demo/docker-compose.yml.
The Tempo trace explorer, which shows a detailed trace of the scheduling of one thread, which includes the durations for each running/blocked segment as well as the reason for each time the thread becomes blocked.
These traces are not enabled by default, as their volume easily overwhelms most endpoints, and must be enabled using a configuration file.
This showcase uses the spectral-norm example with the Docker Compose setup in demo/docker-compose.yml.
Eventlog Live’s OTLP exporter is configured using OpenTelemetry’s environment variables. The output of eventlog-live-otlp --help has comprehensive documentation on its support for these environment variables.
By default, Eventlog Live aggregates telemetry data over 1 second intervals and exports every 30 seconds, but these intervals can be configured for each telemetry signal using configuration files.
Dynamic Control with Eventlog Socket
If you instrument your application with eventlog-socket, you’ll be able to start/stop profiling while your application is running using a REST API, which you can connect, e.g., to buttons on your Grafana dashboard, as we’ve done on the example Heap Profiles dashboard.
Dynamic Control with Eventlog Socket
Using the buttons on the Grafana Heap Profiles dashboard. When “Stop” is pressed, the heap profile flatlines. When “Start” is pressed, the heap profile resumes. The “Census” button can be used to request a single heap census. For detailed instructions, see Eventlog Live with Eventlog Socket. This showcase uses the oddball-with-hT example with the Docker Compose setup in demo/docker-compose.yml.
Using the buttons on the Grafana Heap Profiles dashboard. When “Stop” is pressed, the heap profile flatlines. When “Start” is pressed, the heap profile resumes. The “Census” button can be used to request a single heap census. For detailed instructions, see Eventlog Live with Eventlog Socket. This showcase uses the oddball-with-hT example with the Docker Compose setup in demo/docker-compose.yml.
Eventlog Socket lets your application write its eventlog over sockets and uses the other direction of communication for control messages. Eventlog Socket’s control protocol is generic. It isn’t bound to any specific application or library. Any library can register its own control commands under its own namespace,4 using Haskell IO actions as callbacks:
Any command that your application registers with Eventlog Socket can be invoked using Eventlog Live’s REST API.
For security reasons, Eventlog Socket’s control protocol and Eventlog Live’s REST API are hidden behind the +control feature flag, which is disabled by default. If you develop a library that (optionally) uses Eventlog Socket’s control protocol, we recommend using the same +control feature flag.
Call-Stack Profiles with GHC Stack Profiler
If you instrument your application with ghc-stack-profiler, you’ll be able to use its lightweight call-stack profiler.
GHC Stack Profiler samples GHC’s runtime call-stack, rather than a virtual cost-centre stack, and writes these samples to the eventlog. Based on our benchmarks, GHC Stack Profiler has virtually no overhead when it’s not running and we observed 2-8% overhead while it was running, compared to GHC’s built-in cost-centre stack profiler, where we observed 54-128% overhead while it wasn’t running and 98-136% overhead while it was running.
If you’ve also instrumented your application with Eventlog Socket, you’ll be able to start/stop GHC Stack Profiler while your application is running using the same REST API, which you can connect, e.g., to buttons on your Grafana dashboard, as we’ve done on the example Call-Stack Profiles dashboard.
GHC Stack Profiler can also be used without Eventlog Live, e.g., by exporting the profiles to speedscope. We’ll discuss GHC Stack Profiler in detail in an upcoming blog post.
Local Viewers
Eventlog Live does not yet include a built-in telemetry viewer, as it was primarily developed to monitor long-running processes in production. We are currently developing a built-in telemetry viewer, but the prototype is not yet ready for publication. In the meantime, some developers may feel that using either Grafana Cloud or the Docker Compose setup in demo/docker-compose.yml is too heavyweight for local development. Fortunately, the OpenTelemetry ecosystem has several lightweight telemetry viewers, such as otel-tui and otel-desktop-viewer, which are general purpose telemetry viewers that use a text and browser-based UI, respectively.
Eventlog Live with otel-tui
This showcase uses the oddball-with-otel-tui example with otel-tui version 0.7.4.
This example creates a screen session that combines otel-tui with oddball and eventlog-live-otlp.
This showcase uses the oddball-with-otel-tui example with otel-tui version 0.7.4.
This example creates a screen session that combines otel-tui with oddball and eventlog-live-otlp.
We are currently developing a built-in telemetry viewer that is purpose-built for GHC’s telemetry, with the intention of making Eventlog Live more useful for short development sessions. We’re also planning to create a library for writing structured log messages to the eventlog, expand the analyses supported by Eventlog Live, and expand the Eventlog Socket protocol to permit command arguments. Moreover, we’re always open to suggestions and feature requests from the community.
Acknowledgements
This work has been performed in collaboration with Mercury, who have a long-term commitment to the scalability and robustness of the Haskell ecosystem. Well-Typed are always interested in projects and looking for funding to improve Haskell tools. Please contact info@well-typed.com if we might be able to work with you!
If your application is built with GHC 9.2 or older, it must also be built with -eventlog. Since GHC 9.4, the eventlog is enabled by default.↩︎
Eventlog Live is developed and tested with the -hT and -hi heap profile breakdowns, but the -hm/-hd/-hy/-he heap profile breakdowns should work. The -hc/-hb/-hr heap profile breakdowns require additional support, which is planned but not yet implemented.↩︎
A little over two months ago, we released the first version of
Tricorder, a Haskell development tool built for
developers of the modern era, where responsive and frictionless developer
experience is king, and AI usage in coding is on the rise. Since then,
we’ve shipped a steady stream of improvements based on user feedback.
One of our most recent releases is tricorder-mcp, a Model Context Protocol
(MCP) server that lets AI agents use Tricorder effectively on larger code
bases.
While Tricorder is primarily developed for human use, it doubles as a bridge
to GHCi for AI agents. This article showcases tricorder-mcp and how it makes
agentic development with Tricorder more reliable.
For the uninitiated: what is Tricorder?
Tricorder is a CLI program that runs a lightweight daemon per project, each keeping
a GHCi session warm in the background. Instead of having your agent cold-call
cabal build and wait through a slow compile after every edit, it can query that
daemon for the most up-to-date build state. Tricorder uses GHCi to recompile
incrementally as files change, keeping type errors, warnings, test results,
and eval comments available on demand. The daemon does the slow part once and
keeps it warm; the agent just asks what’s true right now.
Tricorder exposes the build state through both a programmatic CLI interface
and a more human-friendly TUI interface that you might just leave
open in another terminal tab while working on your project.1
Context matters
We’ve offered a ready-to-use agent skill for some time now, and
if you’ve tried it, you might agree it works really well. However, longer
sessions reveal its limitations, as the agent forgets how to use it properly.
This is especially the case when working in large codebases, where a lot of
code needs to be brought into the context window for the agent to do its job
effectively.
The reason is the dreaded context window: once the agent decides it needs a
skill, it reads the skill definition once and reuses that original read
definition for the rest of the session. It is not automatically re-read unless
you prompt the agent to do so. That original read is just more content sitting
in the context window, and like everything else there, it’s subject to
compaction and context bloat. The agent never forgets the skill exists, but
it can lose track of how to use it.
MCP tools don’t have this problem. An agent’s harness resends the full list of
available tools—alongside skills and built-in tools—on every single turn, as
concise descriptions the agent invokes directly with the right arguments.
There’s no one-time read to lose to compaction, so tools stay just as reliable
in a two-hour session as in a two-minute one.
In effect, the agent doesn’t actually know how to use the tricorder CLI
program directly; it’s the MCP server exposing it as a tool that handles that,
turn after turn, without the instructions ever degrading.2
Words have meaning
The skill works by telling the agent how to invoke tricorder: a page of prose
describing subcommands, flags, and output formats that the agent has to read,
remember, and reproduce correctly on every call. Forget --wait, or get a flag
name slightly wrong, and the agent doesn’t fail loudly. It just gets a stale or
subtly wrong answer, or gives up on Tricorder entirely and pulls out cabal
directly, and carries on.
tricorder-mcp replaces that with typed tools: Status, TestResults,
Restart, and others, each coupled with a structured schema. tricorder-mcp
better surfaces the difference between “Tricorder found build errors” and
“Tricorder itself failed to run”. The status, test-results, and eval-comments
commands exit non-zero when they’re reporting errors or failing tests, and the server
still returns that as a normal result rather than an error, so the agent
doesn’t have to guess what a non-zero exit means.
Show and tell
To better illustrate how easy it is for an agent to use Tricorder with this MCP
server, let’s use it in some actual work!
Picking a simple enough task straight off of Tricorder’s own issue board: the
Tricorder daemon should quit after a period of inactivity. Currently, the
daemon will just hang around after you’re done with it until you happen to
remember to kill it. This is a bit of a problem, seeing as GHCi can gobble up a
sizeable chunk of memory after a few :adds and :reloads.
I will use Claude Code for this demonstration. The full conversation is
available in a public Gist.
Our initial prompt looks like this:
❯ Currently, the Tricorder daemon will remain running in the background forever
once started, until the user themselves calls `tricorder stop` or otherwise
kills the process. It would instead be preferable if the daemon would shut
down by itself after a certain amount of idle time.
...
I prompt Claude to make the Tricorder daemon shut down after a configurable
period of inactivity. I also ask it to use the Tricorder MCP server to check
the build and test status along the way, and I make sure to mention some
edge-cases that it needs to address in its implementation.
Claude provides us an overview of its plan, and begins exploring.
● I'll start by loading the Tricorder MCP tools and checking the current build
status, then explore the daemon/session architecture.
As Claude explores the repo to figure out how to achieve its goal, it runs
Tricorder through the MCP server, observing the build process for
any issues. It can view the build status itself, and as soon as the
build is finished it can view test results as well.
Read 1 file, called tricorder 5 times, ran 2 shell commands
After making a series of changes, Claude can then check in with Tricorder again
to observe the build process. Since Tricorder is running in the background, the
build is probably ready for Claude to check by the time it
gets around to doing so. Claude doesn’t have to wait around for cabal build
to finish.
● Good. Now let's build and check for type errors via Tricorder MCP.
Read 1 file, called tricorder 5 times, ran 2 shell commands
Claude can perform various actions using Tricorder as it goes along:
Check test results:
● tricorder - test_results (MCP)(wait: true)
└ atelier-core:test:atelier-test passed (214) (410ms)
test/Driver.hs
Config
extractNestedConfig
should use default value for non-object Values: OK
should fetch top-level property: OK
...
● tricorder - source (MCP)(modules: ["Data.Time.Units"])
└ {-# LANGUAGE DeriveDataTypeable #-}
{-# LANGUAGE GeneralizedNewtypeDeriving #-}
-- |This module defines types for many useful time periods, as well as
-- mechanisms for converting between them.
module Data.Time.Units(
TimeUnit(..)
, Attosecond
, Femtosecond
, Picosecond
Restart the daemon entirely, if the need arises:
● tricorder - restart (MCP)
└
A lot of these operations also support narrowing down the returned result
to reduce the size of the context window filled by Tricorder’s output.
After a little bit of back and forth, checking the build along the way, Claude
considers its task complete.
● Implementation is complete and verified (0 diagnostics, all 4 test suites
passing including the new tests). Summary of changes:
- Tricorder.Session.IdleTimeout (new): IdleTimeout newtype, defaults to 300s.
- Session.Config / Session: added idle_timeout_seconds config field
(default 300), threaded into Session and re-evaluated on every loadSession call
(i.e. on config reload), per your request.
...
Further refining Claude’s solution from here is as simple as ever, but
thoroughly sped up by Tricorder’s snappy build loop and output.
This is a simple example of how you can use Tricorder through the
Tricorder MCP server with Claude or another coding agent. Admittedly, this task
could’ve been achieved without Tricorder just as easily, or by using the
Tricorder skill, but I hope this walkthrough demonstrates the various ways
an agent can use Tricorder to optimise its work.
Want to try it out with an AI agent like Claude, Copilot, Codex, Cursor or
Grok? Check the guide on
using Tricorder with coding agents for a
full suite of plugins that manage the MCP server binary for you, and can pick
between the MCP server and the skill intelligently.
Next up for Tricorder is further improving and refining the tool’s outputs.
Much care has been given to the output of tricorder ui so far, for human
usage, but there is still a fair amount of work left to improve the
regular CLI interface in tricorder status, tricorder test-results, et al.
I am grossly simplifying the potential uses for MCP servers in this
article. They can be used for many different things, especially when
they define resources and prompts on top of the tools we describe here.
At full scope, it’s helpful to think of MCP servers as AI-oriented APIs
over resources that have a well-defined set of operations you can perform. ↩
Yesterday I reported about how much I despise home repair, how I feel
I'm really bad at it, how it sometimes goes terribly wrong for me, and
how my work diary had demonstrated that a seeming failure was catually
a glorious success.
I included this picture of our (formerly!) leaky laundry tub, and it
reminded me of something else.
When we first moved into the house, more than four years ago now, that
black outflow hose was unsecured. If you weren't careful — and
sometimes even if you were — it would thrash around while the washer
was emptying, and sometimes spray water everywhere.
In the first week or two after we moved in, there was a long list of
minor problems like that. I found some store online that was selling a
fix for the thrashing hose. The fix, which you can see in the picture,
is a stiff but flexible rod with a collar at each end. You affix the
collars around the top and bottom ends of the hose, then bend the
flexible rod to force the hose to point in the direction you want.
Then the hose can't thrash.
I observed a while back that the worst
projects are also the ones that are easiest to remember. These are
the ones that drag on for months, requiring endless adjustments,
different approaches, and a lot of thought and toil. Or they're the
ones that fail completely leaving some unfinished damage that reminds
me every day that what I did didn't work.
Here's an example in the other direction. I didn't know how to solve
this problem, but I found a solution, I executed it, and because the
solution has worked flawlessly for more than four years I completely
forgot about it until today when I was looking at this photo.
The work diary can help with this sort of thing. If I had a rollup
page for laundry room issues I would be able to open it and see a list
of all the things I had gotten right over the years, instead of
forgetting them and only remembering the stuff that went wrong.
Claude advised that it would be better to write “hub”
instead of “tub”, but I opted not to take its advice.
Addendum
Another example that occurs to me: in our house it's my job to unclog
toilets. How many times over the years has a toilet clogged and I
unfussily unclogged it with the plunger? Too many to count. And
there have been other occasions when the plunger didn't work, so I
would shrug, and go downstairs to get
the toilet auger. I don't remember
these many episodes because to me they were no big deal.
I should try to remember that no matter how incompetent I feel in
general, just owning a toilet auger puts me way ahead of a lot of
other people.
When I put up shelves in the bathroom back in May, I was a
psychological mess. For every little thing that went wrong — and there
were quite a lot — I got all stressed out and wondered why I dared to
perform this task. The outcome was good, but I had a lot of stress
getting there.
Through the whole thing I felt like I didn't know what I was doing, I
didn't know if I was about to make some terrible mistake that would
ruin the whole project, and I didn't know if I was about to produce a
humiliating disaster that a professional would have to be called in to
do over.
This is a lot of bad feeling for a project that, in hindsight, went
smoothly and ended in success. I finished timely. The shelves served
us well until we moved out eight or nine years later. I don't think
that most of the people who came through there thought anything like
“gosh, I wonder what incompetent installed these ridiculous shelves?”.
Anyway that is not what I planned to talk about this time.
I loathe home repair
In our laundry room the washer dumps its outflow into a deep plastic
sink with a drain, which we call a "tub". A while back the tub
developed a big crack in its bottom that was leaked dirty water onto
the floor. We put a bucket underneath, but that is just a workaround,
not an actual solution.
I thought with displeasure of the two unpalatable choices for
replacing the tub with a new one. I could pay a plumber to do it, and
it would probably be done competently, but would cost a lot of
money. Or I could attempt to do it myself, and it might be done
competently, and it might cost much less money… or I might be taking
the first step on the road to Hell.
If I were to attempt it myself, I would not only have to learn how to
do the task, and then get it right the first time, I would also have
to figure out on the fly how to fix the parts of the task I didn't
get right the first time. The worst case here could be very bad, much
worse than just “oh well, guess I'll have to call the plumber”. There
are a lot of ways it can go very wrong.
I'd have to buy supplies, the right supplies, and if I discovered
partway through that I didn't have the right supplies or tools I'd
have to stop work and go back to the store for the right ones,
interrupting the project. Or maybe the store has closed and we are
without laundry service until it reopens.
If I didn't get the task done right the first time, I would need new
supplies to try again, and maybe additional supplies to fix the
mistake.
I might break something else that wasn't broken to begin
with.
I do not find this kind of work fun. I hate it, and the longer it
goes on the more I hate it. By trying to replace the tub myself I
might be stepping into a tar pit.
This isn't just neurosis. There are projects in my house that I
thought would be straightforward that are perpetually unfinished for
exactly this sort of reason: I started, and something went wrong that
I didn't know how to fix.
In these kinds of jobs there is often some subtask where you have to
get two parts together by forcing them, or there is often some subtask
where you have to cajole two parts together but forcing them will ruin
them. If you mistake the second kind of subtask for the first you
have the fun of putting the broken parts in the trash and starting
over, or maybe just hurling yourself into the sea.
So instead of replacing the tub, I thought that I could try an easy
intervention: I could try to seal the crack with silicone goo. It
might not work at all, but it would be quick and easy to try, and it
wouldn't make the problem worse than it already was — the tub was
already leaking. And I had the silicone in the house already.
I did that and it seemed to work, so with relief I put it out of my
mind for a while.
Return of the leak
Time passed and last week I learned that the tub was leaking again. I
took a look into the tub and saw that the silicone from last time
seemed to be holding okay, but that the crack had gotten bigger.
I was disappointed. I had hoped that the silicone would solve the
problem. But it appeared that it hadn't, and the leak was just going
to keep coming back. The silicone hadn't worked, except as a
short-term workaround, and it seemed I would need to replace the tub
anyway.
Probably, I thought, if I knew what I was doing, I would have known
that silicone goo is not the right way to fix something like this.
Maybe someone who knew the right way would have scoffed and said
“Silicone? Pff, don't you know that stuff will start coming off in
two weeks?” Or maybe “Pff, that crack is just going to spread.”
But I went and glumly patched the new, bigger crack with more silicone
to give me some time to think about what to do to take care of the
problem for good.
Work diary to the rescue
And then something magic happened. I went to make a note about it in
my work diary.
The work diary is just an attempt to track stuff at home with a
ticketing system, similar to how I do when I'm at work. Each task I
need to do gets a ticket, which starts out as a mostly blank page. When I
work on a task, I make a note in the ticket about what I did, with the
date. When a task is done, I mark the ticket closed.
When the tub had first started leaking I had opened a ticket for it.
Then whenever there was some new development on the tub, I added a
date and a note — there was a cluster of notes from around the time I
had been trying the silicone, noting what kind I had used and how long
it had taken to cure. Then I had closed the ticket. Now I reopened
it.
Why go to the trouble of a ticketing system for chores like this one?
There are lots of reasons, and I hope to write more about this in the
future. But today's an example is an unusually good one:
When I went to update the ticket with the new status, I saw that my
previous attempt to fix the problem…
(the one I thought hadn't worked…)
had been in May 2025.
The silicone patch I had put on had solved the problem for sixteen
months. I had had no idea that it had been so long ago.
That changed things completely. I had been thinking:
I put the silicone on, it worked for a while as a stopgap, but it soon
failed and I need to come up with a different plan.
Wrong! With the new information, there was a completely different
interpretation:
I put the silicone on, it took only twenty minutes, cost next to
nothing, and worked for sixteen months.
The patch was not a failed workaround, not a short-term solution
while I figured out the real solution. The silicone was the real
solution! If all I need to do to keep the tub working is to smear
goop on it every sixteen months, I don't need a better solution!
That date recorded in the work diary changed “it didn't work” into
“problem solved”.
No artificial intelligence was involved in the
writing of this article, which is why it remains thoroughly
unpizzled.
This is the thirty-second edition of our Haskell ecosystem activities report,
which describes the work Well-Typed are doing on GHC, Cabal, HLS and other parts
of the core Haskell toolchain. The current edition covers roughly the months of
June, July and August 2026.
We offer Haskell Ecosystem Support Packages to provide commercial
users with support from Well-Typed’s experts while investing in the Haskell
community and its technical ecosystem including through the work described in
this report. To find out more, read our announcement of these
packages in partnership with
the Haskell Foundation. We need funding to continue this essential maintenance work!
We have long been working to decouple GHC from base, to make it easier to
upgrade the compiler independently from the (standard) libraries.
A key restriction that prevented disentangling base from GHC was GHC’s
treatment of known entities. These are things in the standard libraries
that GHC needs to know something about (e.g. Show for deriving, Monad for desugaring do
notation, etc). This is detailed in section 3.3
of a recent blog post by Simon Peyton Jones.
Rodrigo and Sam recently overhauled GHC’s treatment of known entities, in
collaboration with Simon (!15899), finally landing the patch in August. This
allowed known entities to be defined completely outside of the compiler for the
first time, without hard-coding the precise module that defines them.
As part of this change, it was also necessary to rework some implementation
details of Typeable, namely the KindRep datatype. Sam opened
CLC proposal #433
to get approval on this change, which required an accompanying change to the binary
package, which Rodrigo implemented in #244.
AArch64 code generation and sub-word handling
Andreas has been hunting down a slew of correctness bugs in the AArch64 native
code generator, mostly related to the handling of sub-word values. Issues
include sub-word handling across C calls (#27430), in atomic loads/stores
(#27533), in conversions (#27537), in if conditionals (#27538), in %i2f64
(#27539) and the use of invalid immediates (#27550). These were all fixed in
!16255, with a lot of the testing infrastructure being provided by Sam’s
overhaul of test-primops
(!43).
Coercion zapping
Extremely poor compiler performance for programs involving significant
type-level computation has long plagued the community (#8095, recently
re-reported as #27336). One major cause for this is blow-ups in coercion sizes
due to the representation of accumulated type family reduction steps in Core,
GHC’s typed intermediate language.
This problem resisted many attempts over the years, with promising approaches
being brought down by performance regressions on certain types of programs (e.g.
singleton-heavy programs).
However, the future is bright for type-level wizards: MR !16378 implementing
coercion zapping promises to bring the story to a satisfying conclusion.
This MR has an unusual history. In July, Sylvain Henry put up a proof-of-concept patch
that was generated using Claude Fable. This patch was a carefully balanced
“best of” previous approaches over the years, combining elements from MRs by
Ben (!611), Adam (!6476), Sam (!7787) and Richard Eisenberg (!7909), each of
which made significant progress on some programs but struggled to overcome
significant compiler performance regressions on other programs.
Sam took over this patch and rewrote it from the ground up, doing a deep
dive into GHC’s type family rewriter, and finding
several related optimisation opportunities (e.g. !16479).
The net result is a patch that significantly improves compilation speed of
type-family-heavy programs without regressing elsewhere. This patch is under
final review and we hope to land it soon. Performance results
from GHC’s testsuite:
Test case
Compiler allocations
Compilation time
T8095
768 MB ⇝ 128 MB (−83%)
2.432s ⇝ 0.160s (−93%)
T9872a
1 656 MB ⇝ 1 406 MB (−15%)
0.971s ⇝ 0.494s (−49%)
T12545
698 MB ⇝ 425 MB (−39%)
0.648s ⇝ 0.387s (−40%)
T13386
428 MB ⇝ 35 MB (−92%)
0.259s ⇝ 0.104s (−60%)
T27336
499 MB ⇝ 36 MB (−93%)
1.188s ⇝ 0.110s (−91%)
CoOpt_Singletons
634 MB ⇝ 596 MB ( −6%)
0.484s ⇝ 0.360s (−26%)
LargeRecord
1 227 MB ⇝ 967 MB (−21%)
0.872s ⇝ 0.617s (−29%)
FamAppCachePerf
496 MB ⇝ 192 MB (−61%)
0.675s ⇝ 0.191s (−72%)
Exception annotations
Ben’s GHC proposal #330 and accompanying CLC proposal #164
proposed a deep rework of exception backtraces, providing several different
backtrace mechanisms (e.g. HasCallStack, profiling cost centres, IPE).
However, the implementation of these proposals remained patchy, as detailed in
Edsko’s recent blog post.
The team made several follow-up improvements to mechanisms for
attaching information to exceptions:
Ben fixed WhileHandling dropping the ExceptionContext of the exception
it is handling (!16275, #27456).
Sam made sure SomeException does not drop the ExceptionContext of the
exception it is storing (!16274, #27455), with accompanying CLC proposal #432.
Wolfgang added rethrowSTM (!16501), incorporated the re-export into the
stm submodule (!16616), and upstreamed it to stm (#100),
with accompanying CLC proposal #436.
Zubin made annotateSTM use catchSTM# rather than catch# (!16508),
with accompanying CLC proposal #437.
Trees That Grow
Over a long period, GHC has been gradually migrating to the Trees That Grow
representation for its internal AST datatypes, and improving the modularity of
the AST within the GHC codebase (#19218, #21592). Once complete, this will make
it possible to expose the AST in a package that does not depend on the rest of
the ghc library, and unblocks doing the same for the parser. As well as
improving modularity, this will make it easier to use in tools that parse
Haskell syntax, and may unlock future simplifications to template-haskell.
Alex and Rodrigo have been working on the next steps in that process,
reorganising the codebase so that there will be no modules in the
Language.Haskell.* namespace depending on modules in the GHC.* namespace.
This included decoupling Language.Haskell.Syntax from GHC.Hs.Doc (!15601)
and from GHC.Types.SourceText (!15573), representing string literals in
the AST with Text rather than FastString (#21628, !12846), and decoupling
ModuleNames used in the AST from those in GHC (!16546). As part of the
original migration strategy, various orphan instances were introduced, which Alex has
been cleaning up (#27755, !16299, !16610).
GitLab infrastructure work
Keeping Haskell’s infrastructure available has been increasingly challenging over the
past couple of months due to the continued rise in LLM-driven crawling. See
for comparison the Creepy crawlies blog post by Konstantin Ryabitsev,
detailing the challenges faced by the git hosting infrastructure for the Linux
kernel.
The GHC GitLab instance relies on Anubis
to reduce crawler traffic. In early August, Magnus traced a fresh surge in load
to a crawler that was bypassing Anubis entirely. Blocking it cut the load on the
GitLab instance dramatically.
Other important infrastructure items:
Magnus and Ben upgraded the instance’s PostgreSQL database, after which Ben
carried the GitLab upgrade itself through to the latest release in a single
fourteen-hour session. Besides clearing a long-standing backlog item, this
made it possible to prioritise traffic from logged-in users.
Magnus upgraded the host running the GitLab container, and applied patches
to the GitLab instance (e.g. to close certain security vulnerabilities).
Zubin worked on 9.14.2, cutting a first release candidate at the end of July, and a second release candidate in September. This work fixed a significant number of bugs, many of them blockers for the next major release, 10.0, which Zubin will prepare in due course.
Andreas continued work backporting patches to the 9.10.4 branch.
Frontend
Magnus renamed ZonkAny to UnusedType and added pretty-printing
logic for it (!16212, #27390), so that it no longer leaks confusingly
in HLS output.
Magnus refactored the error reporting code for
ExplicitLevelImports (!16195).
Adam marked various language extension flags as deprecated (!16145),
implementing part of the categorisation of language extensions (#27329).
Sam relaxed representation-polymorphism restrictions on newtype
constructor patterns (!16342, #20363).
Sam fixed a couple of issues with mis-oriented coercions produced by GHC
(!16474, #27639, !16207, #27374).
Sam fixed a crash in the Data.Data instance for HsCtxt (!16142,
#27359).
Sam reviewed a patch adding explanations for unsolved Typeable
constraints (!16351).
Core & simplifier
Andreas stopped the specialiser looping on recursive dictionaries
in interestingDict (!16559, #27705).
Zubin landed various fixes for “Entered absent arg” bugs
(!16519, #27627, !16503, #27626), along with fixes for the interaction of
absent fillers with speculative evaluation of dictionaries (!16219),
following an extensive investigation of the crash reported in #25924.
Following on from Simon Peyton Jones’ earlier refactor of GHC.Core.Utils.mkTick
(!15494), which Sam helped with, Sam and Magnus tied up several loose ends:
preserving tick ordering in tickTickedExpr (!16601), fixing a
regression in which source notes were moved past each other;
allowing more ticks in argument position in rebuild_app
(!16558);
stopping ticks being dropped around variables of type IO ()
(!16123), fixing #27182 and #27225;
stopping mkTick in Core Prep from
breaking ANF (!16198).
Driver
Rodrigo made some improvements to the structure and caching of downsweep
(!16330, #27053, #27461).
During this work, Rodrigo and Sam found an opportunity to clean up some
legacy Backpack logic, establishing that implicitRequirementsShallow can
never reach a HoleUnit (!16346).
Rodrigo deduplicated and shared Modules in interface files (!16216, !16213, #27401).
Hannes shared UnitInfos across the UnitEnv using a UnitIndex
and an external unit database cache (!16115), and arranged for the
UnitIndexCache to be reused after initialising multiple home units (!16598,
#27748).
Hannes added the backendInfoTableMapValidity backend predicate
(!16234).
Hannes fixed the order of PackageDBFlags for the interactive home unit
(!16591, #27640).
Hannes diagnosed the quadratic behaviour of ghc-pkg list (#27275).
Zubin carried out a substantial overhaul of the loader (!16493, !16582,
!16583) to allow more control over unloading stale linkables, which is
useful for HLS to efficiently evaluate expressions as files change in the
IDE (#27606).
Zubin made UniqDFM.alter preserve insertion order (!16385).
Wolfgang finished up a patch sorting Usages by fingerprint, to ensure
determinism (!15484).
Code generation
Andreas fixed common block elimination silently dropping a procedure’s info
table (!16592, #27722).
GHCi and bytecode
Wolfgang added --show-byte-code, the counterpart of --show-iface for
bytecode files (!16386, #26909).
RTS
Duncan continued his sustained work to overhaul RTS I/O managers and the
scheduler. One of the main goals is to bring the non-threaded RTS in line
with the threaded one (e.g. #27335), and to improve maintainability across
I/O manager implementations.
As part of this effort, Duncan landed a collection of I/O manager refactors
(!16217), preparing the ground for fixing idle GC and deadlock detection on the
non-threaded RTS. Duncan also brought the POSIX and Win32 ticker implementations
more in line (!16023).
Andreas fixed pointer arithmetic in the RTS linker’s
flushInstructionCacheRISCV64 (!16419, #27569).
Build system
Sam made Hadrian check for cabal-install >= 3.12 upfront (!16132,
#27317), rather than failing obscurely later on.
David fixed the ghc-internal.def file name (!16190), removed
unused wrapper scripts from the Windows bindist (!16138), and made Hadrian
avoid response files when the command line is short enough (!16134),
as response files were impeding developer workflows (#27230).
David stopped ghc-toolchain throwing an exception when candidate executables are
not found (!16173, #27369).
Testsuite
Sam made the test driver normalise call-stack line numbers when referring to
locations in libraries (!16247, #27387), so that tests are no longer
sensitive to irrelevant internal changes.
Andreas made the process009 test use SIGKILL (!16429).
Zubin made various improvements to the JUnit output, most importantly
showing diffs in the GitLab summary (!16309) and reporting fragile failures as
skipped (!16250). He also landed assorted testsuite fixes (!16413).
CI and infrastructure
Magnus worked on keeping nightly CI passing (!16296, #27127),
adding a retry when fetching test metrics (!16281).
David made the lint-changelog job check for the no-changelog
label in the script rather than in rules (!16177), which avoids having to
re-run the entire pipeline when applying the no-changelog label.
Zubin arranged for tool messages to be reported as errors or warnings depending
on the tool’s exit code, rather than always as errors (!16170, #27370).
process library
Zubin fixed link errors on certain macOS configurations by allowing
posix_spawn_file_actions_addchdir to be missing when linking against
process on Darwin (#377,
#376).
Sam stopped Cabal from writing setup-config twice
(#11949), fixing access-denied
failures (#11619).
Sam made cabal-install fall back to the external setup method on a
Cabal-library mismatch (#11950),
fixing stuck rebuilds of build-type: Custom packages
(#11942).
Sam allowed preserving the local build artifacts across minor Cabal version
bumps without causing recompilation or errors
(#12289).
Tobias switched pre-processor recompilation checks to use file monitoring
(#12021, with a preparatory
refactoring in #12065).
This fixes an issue with build tools not being re-run when their version
changes (#11411).
Haskell Language Server
Andrea added whole-project loading
(#5002),
as described in his recent blog post.
This builds on the groundwork Andrea laid in hie-bios: new load modes for
upfront loading of a cradle or project
(#508) and the
componentsToLoad field
(#515), both reviewed and
merged by Hannes.
Hannes and Zubin pushed Guibou’s module-to-filename resolution over the finish line
and applied further optimisations to put HLS’s import resolution on par with GHC’s
(#4600).
Zubin made HLS unload linkables when any transitive dependency changes
(#5035),
and stopped HLS unloading all bytecode whenever new bytecode is generated
(#5033).
Hannes set the FileCache hook before typechecking
(#5030),
fixing #4675.
Hannes ensured the DocMap is never computed with stale data
(#4936).
Hannes tracked down multi-component session failures when expanding response
files (#5004),
fixed by whole-project loading.
Hannes substantially improved HLS’s error messages
(#5029).
Hannes transferred the VS Code syntax highlighting extension to the
Haskell community (#256),
fixing a mishap that had caused
the extension to disappear from the VS Code marketplace, breaking syntax
highlighting for many users.
Runtime observability
Haskell Debugger
Rodrigo continued to lead development of the
Haskell Debugger (hdb), an
interactive step-through debugger for Haskell, with contributions from Andrea
and Hannes. It also gained a logo, designed by George Thomas (#326).
Hannes and Rodrigo added support for the future release of GHC 10.2 (#332, #359).
hdb has two mechanisms for listing threads, decoding stacks and collecting
exception information: external compiler commands where GHC supports them,
and a fallback that inspects the heap directly on older GHCs. Rodrigo
separated the two into distinct modules selected by a Cabal version
conditional, rather than interleaving them (#356).
Rodrigo made the singleThread flag of a continue request optional, as
the Debug Adapter Protocol specifies (#38).
Hannes made the stack frame decoding logic use the source locations that
StackAnnotation carries from ghc-internal-9.1402 onwards
(#333).
Andrea made each stack frame report the variables bound in its own scope
(#355,
#160).
Live monitoring
Wen continued to develop
eventlog-live, which allows
GHC’s eventlog to be monitored as a program runs, streaming telemetry data
from the runtime system to an external observability platform.
One focus was on improving the OpenTelemetry integration.
Releases 0.6.0.0 through 0.9.0.0 came out this quarter. See the
eventlog-live changelog
for more details.
Stack and IPE profiling
Hannes and Wen worked on packages supporting stack profiling functionality, with
a release of ghc-stack-profiler 0.4.0.0 (changelog) and ghc-stack-annotations 0.2.0.0
(changelog).
Wen rewrote ipedb’s LSM-tree database API
(#18) and released 0.2.0.0/0.2.0.1
with other improvements (see the changelog).
The GHC developers are very pleased to announce the availability of the
first alpha prerelease of GHC 10.0.1. Binary distributions, source
distributions, and documentation are available at downloads.haskell.org and
via GHCup.
GHC 10.0 will bring a number of new features and improvements, including:
Type and Constraint are now completely distinct types, just as much as
Int and Bool (#24279).
The new QualifiedStrings language extension
(GHC Proposal #723) allows string literals to be
qualified using module qualification syntax, so that M."my string"
desugars toM.fromString "my string".
The treatment of static forms has been simplified, implementing
GHC Proposal #732 (#26556). The free term variables of
(static e) must now be bound at top level.
-fpolymorphic-specialisation is now (re)enabled by default (#23559),
Bytecode can now be written to disk with -fwrite-byte-code and packaged
into bytecode libraries with -bytecodelib, which the interpreter can use
to satisfy package dependencies (#26298).
Executables can now be properly statically linked with the new
-fully-static and -static-external flags (#26434).
SIMD support in the AArch64 native code generator (#26536).
A new poll I/O manager, which is now the default in the single-threaded
RTS on all POSIX platforms except macOS (!9677).
Version 2 of the GHC Jobserver Protocol for GHC’s -jsem
jobserver on Linux and other POSIX platforms (#25087, #27253). This is
sadly a necessary breaking change. -jsem requires cabal-install 3.18 or
later, and is otherwise ignored with a warning, falling back to sequential
compilation.
Changes to base:
GHC.Num.{BigNat,Integer,Natural} are no longer exposed
(CLC #359) and GHC internals in GHC.Num are deprecated
(CLC #360).
Data.List.init and Data.List.last now carry an x-partial warning
(CLC #292).
The minimum bootstrap compiler is now GHC 9.10, LLVM versions up to 23 are
supported (#26813, #27764), and the Debian 9, Debian 10, Ubuntu 18.04
and Ubuntu 20.04 bindists have been dropped in favour of a new Debian 13
bindist (#25876).
… and many more
A full accounting of changes can be found in the release notes.
See the migration guide for guidance on migrating programs to this release.
As always, GHC’s release status, including planned future releases, can be
found on the GHC Wiki status.
Given the many changes in this major release, we would very much appreciate
testing on downstream workloads.
We would like to thank these sponsors and other anonymous contributors
whose on-going financial and in-kind support has facilitated GHC maintenance
and release management over the years. Finally, this release would not have
been possible without the hundreds of open-source contributors whose work
comprise this release.
As always, do give this release a try and open a ticket if you see
anything amiss.
Over the last months, I’ve participated in a Google Summer of Code internship at
Tweag in collaboration with Haskell.org, with the goal of bringing type class
support back to Liquid Haskell.
In this post, I will share the major insights and contributions that resulted
from this project, walking through the main breaking points that I identified
and their solutions, before briefly discussing the current state of the project
and the limitations that emerged while expanding the test suite.
What is type class support, anyway?
Ask a Haskell programmer what makes a Monoid instance well-behaved, and the
answer will certainly involve laws: <> should be associative and mempty
should act as an identity. In Haskell, it is common practice to describe
structures, many of them implemented as type classes, in terms of the properties
that should hold for them.
However, the responsibility of checking that the instances are well-behaved
usually falls on the developer. This is where Liquid Haskell (LH) can be really
convenient, as it will delegate to an SMT solver the task of proving that the
properties hold for instances of a class.
That feature used to exist back when LH targeted GHC 8, but as the library moved
on to GHC 9, it broke and its test suite was disabled. During my GSoC 2026
project, I worked on bringing it back; the work is documented in
this PR. I am thrilled
to say that type classes are once again supported by LH, and that the test suite
has been re-enabled.
Most of the machinery for working with type classes isn’t new. It was built
during the GHC 8 era, and credit goes to its original authors: Yiyun Liu, James
Parker and Niki Vazou. What the restoration required was a proper integration
with the new GHC 9 API, and making targeted adjustments that account for recent
developments that have taken place in LH since then.
When a class becomes a proof obligation
Before we dive into the work that was done, let’s take a look at an example
(taken from
All.hs
in the test suite):
classSemigroupawhere{-@ mappend :: a -> a -> a @-}mappend::a->a->aclassSemigroupa=>VSemigroupawhere{-@ lawAssociative :: v:a -> v':a -> v'':a -> {mappend (mappend v v') v'' == mappend v (mappend v' v'')} @-}lawAssociative::a->a->a->()dataPNat=Z|SPNatinstanceSemigroupPNatwheremappendZn=nmappend(Sm)n=S(mappendmn)instanceVSemigroupPNatwherelawAssociativeZ__=()lawAssociative(Sp)mn=lawAssociativepmn
There are a few things worth noticing here. The laws that describe the behaviour
of a class are no longer an agreement invisible to the user, but through
VSemigroup, we actually have a way of allowing the user to provide a proof to
the specification declared as a method.
For PNat, the body of lawAssociative serves as the proof that the law holds
for that instance. The recursive call on the first argument carries out the
induction: LH verifies the base case directly and the recursive call on the first
argument tells the solver which induction hypothesis to use for the successor
case.
In order to accomplish this, LH mirrors what GHC already does: while GHC
compiles classes into record types, LH will generate a specification of the
respective dictionary; GHC turns methods into record selectors, and LH lifts them
into the refinement logic; and instances are elaborated into evidence by GHC,
whereas LH will check the class’s laws for a concrete type.
What broke in the GHC 8 to 9 migration
In order to do its work, Liquid Haskell hooks into GHC as a plugin: It
consumes GHC’s intermediate representation, calls its API, and even asks GHC to
elaborate expressions on its behalf. This closeness between the two makes their
integration seamless, but also puts pressure on the LH
maintainers to keep up with the changes made to the compiler.
In particular, the release of the compiler’s version 9 brought many
changes to LH, but the feature to check type classes was left behind,
and thus the corresponding tests were guarded for earlier GHC versions:
executable typeclass-pos
...
-- Fails on GHC 9.0.1
if impl(ghc < 9)
other-modules: ...
So, the first thing to do was to run the broken tests and identify the errors.
The exploration was not as straightforward, since often an error would only show
up after solving a different error at an earlier stage in the pipeline.
But looking back, errors can be classified in three buckets:
Name resolution: Refinements that mention type class methods failed
because their names weren’t visible to the resolver.
Elaboration of expressions: Methods (or laws) elaborated through GHC came
back incomplete, especially in subclasses; e.g. unbound symbols and wrong number
of binders.
Measure generation: The generated measures had dictionary constructors of
the wrong shape, producing arity errors and mismatched types in the
constraints.
The way I approached these problems was to use the behavior of GHC 8 as an
oracle: I traced each error to find the pieces of code
where the behaviour diverged from the working implementation, and re-aligned the
two.
I will address these issues in the same order in the sections that follow.
Teaching the resolver about class methods
After the migration to GHC 9, Liquid Haskell introduced a significant change: the
name resolution module.
This new phase in the compilation serves as an entry point through which all names
used by LH have to go. However, this new pipeline did not account for type class
methods.
If you tried to run a test involving type classes after the introduction of name
resolution, even doing something as simple as mentioning an imported class
method in a specification, LH would throw an error. For example, this module
imports check and uses it in the specification of a class law:
When building this, Liquid Haskell could not resolve check in the specification:
Unknown logic name `check`
Cannot resolve name
|
7 | {-@ lawValid :: v:a -> {check v} @-}
| ^^^^^
The way to solve this consists of making LH aware of the existence of such class
methods during the name resolution phase. If the lookup in LH’s environment
fails, we can leverage GHC’s environment and check if there is a match for the
name we are interested in. If that is the case, the name can be registered in
the logic, as long as the entry’s parent is a class (thus making sure it is a class
method).
For the time being, we assume that if the user wants to use class methods in
specifications, they will also want to activate type class checking. Therefore we
show an error message inviting the user to enable type class support:
Unknown logic name `check`
This is a class method, but typeclass support is disabled.
Enable it with {-@ LIQUID "--typeclass" @-}.
Where did dictionaries go?
When compiling code that uses a type class method, GHC inserts the dictionary
that corresponds to the instance on which it is being called. Similarly, in
order to reach a superclass method, it is necessary to first extract the parent
dictionary. GHC stores that parent dictionary as a field of the subclass
dictionary and generates a selector to retrieve it. For example, $p1VSemigroup
is a compiler-generated selector1 for the Semigroup dictionary stored
inside a VSemigroup dictionary, so GHC produces
mappend ($p1VSemigroup vsemigroupDict) x y instead of
mappend vsemigroupDict x y. This applies only to subclasses, since only they
have superclass dictionaries.
For LH to check a specification referencing these methods, it hands the
refinement to GHC to obtain a Core expression. Therefore, GHC decides what
dictionaries to insert at each location. These dictionaries are the evidence LH
needs: they carry the implementations of methods and, for subclasses, the
dictionaries of their superclasses. However, there were some slight changes in
how those dictionaries are given: in GHC 8, the dictionaries used to be inlined
in the application, whereas GHC 9 uses let-bindings to pass those arguments.
This was a breaking change, because the previous code was expecting abstract
syntax trees with a
different shape. In particular, the non-inlined dictionary bindings were ignored,
discarding important evidence in the elaborated expressions. The expressions
therefore referred to free variables, resulting in errors such as “Unbound
symbol”.
The fix, then, was to not discard that evidence when elaborating the
expressions. As with the rest of the restoration, I used GHC 8 as the source of
truth and mimicked the behaviour it previously had: to inline the evidence instead.
Once the elaborated Core expression is returned by GHC, a function will traverse
the tree to find evidence bindings (which can be easily identified via the GHC
API) and substitute their occurrences by their definitions.
This produces a valid expression with no unbound symbols.
Counting fields on dictionary constructors
I previously mentioned how classes are transformed into record types: every
class declaration gives rise to the specification of a record that describes
what the shape of the dictionary is, its fields and their respective types. That
description is consumed in several places, most noticeably during measure
generation2.
To use these record types, LH must determine the number of fields each
record has and reconstruct the shape of the corresponding GHC dictionary. This
must account for fields storing superclass dictionaries.
GHC 9 had some changes on what fields were returned by the
GHC API, and the expected arities started
disagreeing in a few places because superclass fields were missing from the
information LH received. This caused a series of errors and panics: “GHC
gave back more/less binders than I expected”, “Liquid Type Mismatch”, “GHC and
Liquid specifications have different numbers of fields”, and so on.
Previously, different phases had different counting rules. The solution required
finding the correct rule to decide the arity of constructors, and applying the
adjustment throughout the code base. In particular, it was necessary to always
count superclass dictionaries as fields for class dictionaries, by including the
data constructor’s
theta information
– the type class constraints that accompany the constructor. This ensures that
all constraints, especially subclass constraints, are correctly included.
This was a targeted fix that only executes when working with type classes,
correctly solving the previous errors.
Known limitations
With the original tests passing again, I started adding new ones to cover some
scenarios that were missing. That surfaced a few remaining gaps that were already
present in GHC 8. Tickets were opened for each one, with small code examples to reproduce
the errors:
Issue #2727: For
a class such as class M m where ret :: a -> m a, where m is a higher-kinded
parameter, m will be elaborated as if it had kind Type instead of Type ->
Type. This triggers a kind error during the verification process.
Issue #2743: Some
special classes, such as Eq, Ord, and the numeric classes, are embedded
directly in the SMT solver. Their methods are translated into SMT primitives
instead of using the definitions supplied by their instances. As a result, LH
assumes that these instances have the standard behaviour of their classes and
cannot verify laws against their actual implementations.
Issue #2744: A
class is lifted into the LH logic only when at least one of its methods carries
an LH specification. Without one, its methods cannot be used in refinements,
and its instances are not checked against class specifications. It could be
worth eliminating this requirement to make type class support easier to use,
or documenting better how it is needed.
Issue #2745: When
a specification uses a class method whose constraints cannot be satisfied, LH
reports an unbound dictionary symbol instead of a clear error resembling
GHC’s error messages.
These are the immediately identified issues. As type class support remains
experimental, we should expect some other issues to be uncovered still by
more tests to come.
Conclusion
We now have a working test suite that runs the type class group on GHC 9: all
previously working tests pass again, unchanged, showing that the restoration was
done properly. A few more tests were added to check some specific critical
points in the pipeline. Negative tests were included to verify that LH fails
when that is expected behaviour.
The user documentation was updated to match the implementation. Comments were
added as well, to explain some design choices and non-obvious steps in the code.
The whole restoration fits in about 150 lines of code, a small number because
none of it is new machinery. Every change followed the same approach: find the
place where GHC 9 and Liquid Haskell’s pipeline had drifted apart, and re-align
them, trying to stay as close as possible to the GHC 8 implementation.
I want to thank my mentor, Facundo Domínguez, for his guidance and for steering me in
the right direction as each new error came up; to the Haskell.org committee and
Google Summer of Code for making the project possible; and to Tweag, for their
work on Liquid Haskell and for hosting this post.
The exact name is an implementation detail which may vary and is internal
to GHC. ↩
Measures are functions that refine the types of data constructors and lift
them to the logic. Generating a measure for a data constructor (or a record
type) requires identifying the shape of that constructor. This is important
for type classes because of the transformation they undergo during
compilation into record types. ↩
The GHC developers are very pleased to announce the availability
of the second release candidate for GHC 9.14.2. Binary distributions, source
distributions, and documentation are available at downloads.haskell.org and
via GHCup.
The changes since the first release candidate are:
Several fixes to the specialiser and demand analyser for miscompilations
causing incorrect runtime results, crashes, or hangs
(#27626, #27627, #27629, #27703, #27704, #27705, #27717).
Fixes for a number of runtime system bugs:
a missing memory barrier in the parallel garbage collector, which could
crash or silently corrupt the heap on weakly-ordered architectures such
as AArch64 (#27477).
an “unknown/strange object” crash in the compacting collector when
collecting the large data constructors that the bytecode interpreter
allocates (#27649).
a segfault when querying the label of an unlabelled thread (#27618).
a crash when capturing or resuming a delimited continuation that adjusts
the asynchronous exception masking state (#27651).
stack corruption when returning large unboxed tuples in GHCi (#27633).
Further fixes to the AArch64 native code generator, addressing bugs that
caused incorrect runtime results from subword operations
(#27430, #27533, #27537, #27538, #27539, #27550).
toException for SomeException no longer drops the attached
ExceptionContext, and a catchSTM handler no longer breaks retry or
asynchronous exception delivery (#27455, #27657).
Fixes for panics and spurious out-of-scope errors involving type arguments
in pattern synonym right-hand sides (#27440, #27583, #27586).
The pattern-match checker now recognises considerAccessible under ticks,
and no longer emits a spurious -Wincomplete-uni-patterns warning under
-finfo-table-map (#27314, #27360).
The deprecation warning for static forms, added in rc1, no longer fires on
free type variables (#27664).
A fix for a regression preventing modules from being loaded into GHCi after
startup (#27202, #27640).
A fix for representation-polymorphism checks producing a coercion that
fails Core Lint (#27639).
Updates for libffi (3.8.0), directory (1.3.11.0), file-io (0.1.6),
haskeline (0.8.5.0) and os-string (2.0.10).
GHC 9.14.2 is a bug-fix release fixing many issues of a variety of
severities and scopes, including:
Version 2 of the GHC Jobserver Protocol for GHC’s -jsem
jobserver on Linux and other POSIX platforms (#25087, #27253). This is sadly
a necessary breaking change: -jsem requires cabal-install 3.18 or later,
and is otherwise ignored with a warning, falling back to
sequential compilation.
A new deprecation warning for static forms that mention nested let-bound
variables. This will become an error in GHC 10.0
(GHC Proposal #732, #26718).
A fix for a CorePrep miscompilation that could project a field out of an
absent dictionary, resulting in a segfault (#25924).
A fix for demand analysis giving an absent demand to an argument that was
still used by the function’s stable unfolding, which could cause a runtime
crash (#26416).
Numerous fixes for register allocation and Cmm register-conflict analysis
bugs, preventing incorrect code generation and corruption of vector registers
when spilling and reloading (#26411, #26526, #26537, #26542, #26550, #26668).
Many NCG fixes on AArch64: MOVK clobbering live values (#26980), register
clobbering and an incorrect overflow bit in MUL2 (#27046, #27047), and
incorrect sign extension (#26978) and unsigned right shift (#26979) at subword
widths.
A fix for a stack alignment bug on x86 that could cause segfaults or corrupted
registers when using AVX/AVX-512 vector code (#26595, #26822).
Fixes for several black hole handling bugs that could lead to deadlocks or
crashes in multithreaded programs, showing up as hangs or “END_TSO_QUEUE
object entered” errors (#26922, #26936).
Fixes for several compiler panics, including issues with SetLevels (#26681),
the type-class specialiser (#26682), mkTick (#26772, #27121), CoreToStg
(#27182, #27386), the pattern-match checker (#25926, #27124), module graph
reachability (#26568), and continuation trimming for bottoming functions
(#27261).
Fixes for several cases where the compiler could hang or loop, including with
negative type literals (#26861), deep subsumption (#26823), and when
simplifying the right-hand sides of rewrite rules (#26323, #26826).
A fix for cast worker/wrapper incorrectly firing on INLINE functions (#26903).
A fix for exponential-time desugaring of nested case expressions
(#27383, #20251).
A fix for a regression that caused overloaded functions to no longer be
specialised as effectively as in previous releases, hurting runtime
performance (#26831).
A fix for associated type family and data family instance changes not
triggering recompilation (#26183, #26705).
Improvements to determinism of compiler output, including the order in which
:info lists instances (#26846, #26858, #26877, #27532).
Additions for GHCi and GHC API users: the -fimport-loaded-targets flag
(#26866), custom external interpreter commands (#26652), decodeStackWithIpe
and related stack-decoding helpers (#27065), an optional SrcLoc on
StackAnnotation (#26806), and a gc_sync_elapsed_ns counter in
RTSStats (#26944).
… and many more.
A full accounting of these fixes can be found in the release notes. As
always, GHC’s release status, including planned future releases, can be found on
the GHC Wiki status.
This release candidate will have a two-week testing period. If all goes well
the final release will be available the week of 1 October 2026.
We would like to thank these sponsors and other anonymous contributors
whose on-going financial and in-kind support has facilitated GHC maintenance
and release management over the years. Finally, this release would not have
been possible without the hundreds of open-source contributors whose work
comprise this release.
As always, do give this release a try and open a ticket if you see
anything amiss.
I’m pleased to announce jj-stack, a new tool for managing stacked GitHub pull requests quickly
and safely with Jujutsu.
Stacked changes let you (or your agent) split a large body of work into small, independently
reviewable changes without waiting for each one to be reviewed or merged.
I built this because I wanted the stacked-review workflow I’d grown used to, with the local
editing experience I’ve found so delightful in Jujutsu.
Lately I've been reading Orwell's collected work from 1945–1950, which
was being deaccessioned from the library. It opens with a short but
fascinating essay on
“You and the Atom Bomb”,
published 19 October 1945.
It seems clear that Orwell's thinking about this development led
directly to his novel Nineteen Eighty-Four, which was written in late 1948.
Orwell's initial focus is:
The question that is of most
urgent interest to all of us, namely: “How difficult are these
things to manufacture?”
This is important, he says, because:
Ages in which the dominant weapon is expensive or difficult to make
will tend to be ages of despotism, whereas when the dominant weapon
is cheap and simple, the common people have a chance. … A complex
weapon makes the strong stronger, while a simple weapon — so long
as there is no answer to it — gives claws to the weak.
Obviously, the atom bomb is in the first category.
Then we come to the part that is clearly the road to Nineteen Eighty-Four:
We have before us the prospect of two or three monstrous
super-states, each possessed of a weapon by which millions of people
can be wiped out in a few seconds, dividing the world between them.
It has been rather hastily assumed that this means bigger and
bloodier wars, and perhaps an actual end to the machine
civilisation. But suppose — and really this is the likeliest
development — that the surviving great nations make a tacit
agreement never to use the atomic bomb against one another? Suppose
they only use it, or the threat of it, against people who are unable
to retaliate? In that case we are back where we were before, the
only difference being that power is concentrated in still fewer
hands and that the outlook for subject peoples and oppressed classes
is still more hopeless.
By 1945 the three super-states were taking shape. Orwell writes about
how in 1941 it had appeared that the Axis powers would win the war
(he mentions James Burnham's The Managerial Revolution)
so that Germany (not Russia) might control Europe
and Asia, while Japan (not China) might control East Asia, but he said
this was just a “miscalculation”:
More and more obviously the surface of the earth is being parcelled
off into three great empires …. The haggling as to where the
frontiers are to be drawn is still going on, and will continue for
some years, and the third of the three super-states — East Asia,
dominated by China — is still potential rather than actual.
The frozen and oppressive political milieu of Nineteen Eighty-Four is clearly laid
out:
Mr H.G. Wells and others have been warning us that man is in danger
of destroying himself with his own weapons…. Nevertheless, looking
at the world as a whole, the drift for many decades has been not
towards anarchy but towards the reimposition of slavery. We may be
heading not for general breakdown but for an epoch as horribly
stable as the slave empires of antiquity. … The kind of world-view,
the kind of beliefs, and the social structure that would probably
prevail in a state which was at once unconquerable and in a
permanent state of “cold war” with its neighbours.
Nineteen Eighty-Four was published in 1949, and Orwell died the following year after
a long battle with tuberculosis. His essay on
“You and the Atom Bomb” is available online
courtesy of the Orwell Foundation and Orwell's estate.
In this episode, we sat down with Michael Hanus. Michael is a professor at the University of Kiel and is one of the founders of functional logic programming, the combination of logic and functional programming in a single formalism. Michael is also the creator of Curry, a functional-logic language that resembles Haskell, so you can dive right in if you're familiar with Haskell. Naturally, we talked about what functional logic programming is, how it improves over plain functional programming, and the challenges of semantics and implementation. We learned a lot - enjoy!
Well-Typed was strongly represented at this year’s ZuriHac, with our team of Haskell experts giving
six talks across ZuriHac itself and the Haskell Ecosystem and Implementors’ Workshops. If you couldn’t make it to ZuriHac, the recordings are now available. In addition, we are delighted that
Dominik will be giving a keynote talk at MuniHac 2026
on 11th October, and there’s still time to register for MuniHac!
ZuriHac Advanced Track
Andres and Andrea hosted the Advanced Track at ZuriHac, delivering two one-hour presentations on implementing dependently typed languages in Haskell, with a focus on dependent pattern matching.
They provide a good introduction to the world of dependent types as well as an in-depth discussion for language implementors.
Haskell Ecosystem Workshop
Andreas gave an excellent talk on how to contribute to the Haskell ecosystem and why companies in particular should want to do so.
Haskell Implementors Workshop
The Haskell Implementors Workshop was a great opportunity to share our progress on
improvements to GHC over the last year. It’s always nice to take a moment to reflect
on the progress we’ve made and the work we’ve done.
Andreas kicked things off with the annual GHC status report. This report
provides a summary of the essential maintenance and community stewardship work that
Well-Typed performs for the GHC project.
Next, Hannes introduced a prototype implementation of ghc-stack-profiler, a lightweight profiler that allows you to profile your application without compiling with -prof.
The profiler uses source locations from -finfo-table-map for human readable profiles and can be used with GHC 9.10 and onwards.
Expect an in-depth post about ghc-stack-profiler soon!
You can also read more about our work in this area in posts tagged with profiling.
Finally, Rodrigo presented his progress on the haskell-debugger, a standalone step-through debugger for GHC.
Since last year’s first introduction, many new features have been implemented, such as support for backtraces with source locations, custom debug visualisations of values and using the external interpreter for debugging multi-threaded applications.
Conclusion
Well-Typed offer Haskell Ecosystem Support Packages
in partnership with the Haskell Foundation, to provide commercial
users with support from Well-Typed’s experts, while investing in the Haskell
community and its technical ecosystem.
These projects were made possible by funding from our clients, notably Mercury, who
are improving the experience for Haskell developers by supporting foundational work on Haskell tools.
It was great to meet everyone who attended the workshops and asked interesting
questions during and after the talks. We hope to see you all again at MuniHac,
or at next year’s ZuriHac!
Formal proofs have been fashionable in the recent months, after being at best an
afterthought of the programming community for half a century. By formal proofs I
mean writing mathematical statements and their proofs in a computer readable language such
that a program (often called a proof assistant) can check that the proof indeed
proves the theorem.
Formal proofs belong to the family of formal methods. A loose term describing all
the computer-assisted methods to gain assurance about programs beyond mere types
and tests. Formal methods have garnered quite a lot of interest of late because
many want to generate more code with LLMs, and without constraint it’s really
difficult to guarantee that the code does what you set out to do. Formal methods
can, at least in principle, provide such guarantees.
Formal methods sit on a spectrum. On one end you have fully automated methods
(such as abstract interpretation, model checking,…), which take a program and prove properties
about the program on their own. Fully automated methods are necessarily limited both in the
kind of properties they can prove, and on the programs they can prove it on. On
the other end of the spectrum, you have formal proofs, which lets you prove any property you
want, in the entirety of maths, you just have to do it all by yourself (and
believe me, convincing a computer of a proof is quite an ordeal compared to
convincing a human).
All formal methods incur a cost, until recently, most software projects came to
the conclusion (rightly or wrongly) that they can’t recoup that cost. This
sentiment is changing. Not only that, but there’s a sentiment that LLMs can be
used to generate formal proofs, making formal proofs themselves much cheaper,
giving you access to mostly arbitrary properties1. Hence the
interest.
As a certified formal-proof nerd (I have a true, bona fide, government-backed
nerdiness certification called a PhD), I’m certainly curious to find out how
grounded these hopes would turn out to be. So I got myself a company LLM
account, and set out on a little project to form an opinion of my own. I don’t
know if it’s entirely fair to make a pronouncement about the entire discipline
based on a single small-scale experiment, but I have to be honest: this is what
this blog post is.
TL;DR While I find the capability of LLMs to generate formal proofs very
promising, I’m not convinced that code generation constrained by tight
specification is a very good or efficient way to program (at least not in its
absolutist sense where most of the code is generated by an LLM).
What I did
Specifically, I wanted to test the maximalist proposition of me writing only
specifications, and let the LLM generate all the code (I allow myself to write
a tiny amount of code, mostly types), and to write all the proofs itself. There
are, of course, a lot of potentially less maximalistic ways to use this
combination of technologies (see for instance this successful attempt at
optimisation). But the maximalist approach seemed like the fastest way
to get an overview of a variety of scenarios.
To that effect I decided to build a tiny SAT solver. Because it’s small enough
while being non-trivial. And because I mostly knew what I wanted to get, but
it’s also not something I’m seasoned at building. I thought it was a realistic
middle ground. And it has decently clear specification, which gave the project a
good chance of success.
I wrote all of this in a single proof assistant2 (Lean is quite popular these
days, but I picked Rocq since it’s been my tool of choice since long before
Lean was even an idea). I assume that a more realistic situation if we ever
use proof assistant for code generation is to have the program in one language
and the specifications and proofs in another. But there aren’t really any mature
system to program this way today. So I didn’t want to get into this particular
can of worms. To simulate the more realistic situation, though, I tried as much as
possible to separate implementation and specifications in different files.
What I found most remarkable
I’d like to start with what I feel worked best, because I truly learnt
something. Well, I suppose I already knew it somewhere deep down; but it was
unconscious and I couldn’t voice it.
Here’s the thing, LLMs appear to be pretty good at writing formal proofs, and
one of the tools that they use and in which they differ significantly from
traditional automation methods is that they act non-locally. An LLM seeing an
error message will often realise that a lemma is missing somewhere else, and
they will go and add them, including behind an abstraction barrier. Traditional
automation just gives you the error message, and you have to go and create the
additional lemma on your own.
Before doing this little exercise, I had never consciously realised the mental
tax caused by having to jump to another file to add a trivial lemma. Whether
it’s via LLMs or via dedicated tools, this non-local automation is something
that formal proof workflow desperately needed. So I was impressed.
Admittedly, the same is true of regular programming, to some degree. Having to
switch file and add an extra function to an abstract data type is rather costly,
and non-local code generation is certainly a contribution of LLM-backed coding
agents. It’s just a bigger burden, and so a bigger relief, in formal proofs.
Speaking of, one of the biggest costs of formal proofs is proof maintenance.
When a program changes a little, proofs about its correctness need to change. A
small change, say, in a data type, can ripple through many proofs. And since
proofs are traditionally much more expensive than implementation, the cost of
change is that much bigger. It’s quite
time consuming, and often painful to us humans to go through the proofs one by
one and fix them. This is something else LLMs are rather effective at. You
change a bit of code, they will propagate that change and fix your proofs. It
really relieves the tension between writing the program you want and keeping
your correctness proofs.
And, on the other hand, when an LLM convinces itself that something can’t be
proved, it will cook up some counterexamples for you to figure out what you got
wrong. The explanations have been quite good in this project. Plus, these are
interactive, you can prod further if you need more. This is a form of
realisation of the long dream of having interactive compiler errors (especially
useful here because figuring out why you can’t prove something is often just as
hard as understanding why you can). I’ve been rather impressed honestly.
What I found most amusing
Ok, so, this is only a side note, but LLMs have a very unusual way of writing
proofs. First they guess an entire proof, and only then they will run the
compiler. The first guess will almost always be quite far from an
actual proof. And only in the second step will it start mutating to converge to
an actual proof. Mutations are sometimes fairly local, and sometimes involve
more guesses about future goals.
There’s nothing necessarily wrong about this approach. It’s just very alien to us
humans. And I admit that I’m quite easily amused.
What it wasn’t
Here’s the thing. I don’t think that programming this way ends up being faster than programming
by hand. To be specific, it’s been considerably faster than writing a program with its
proof by hand, but not than writing a program alone.
There are a number of obstacles there. First, writing specifications is hard. Not
everything has a clear specification, but even when it does, the specification can be
quite complex.
In the case of a SAT solver, the SAT solver itself has a simple specification,
the parser and CLI don’t really admit a specification, and the main invariant
(which is required to prove the SAT solver’s specification) is about 50 lines long.
I got
it wrong, forgot some bits. The agent takes a long time to prove, or fail to
prove. We iterate. It takes time.
The other thing is that specifications are partial. Most of the time you don’t
fully specify the functional behaviour of your functions, but even when you have
a full functional specification, it doesn’t capture the algorithmic aspects of
the function. So you still have to read the generated code quite carefully, and
it’s really hard to get a good feel of what’s going on in code you didn’t write.
This is compounded by the fact that an LLM-backed agent will try to sneak every
possible trick past you: it will mess up algorithmic features of a function
if it helps it prove something, it will alter your specification, pierce your
abstractions, assume lemmas as axioms instead of proving them. Once it even
fully duplicated my program in order to avoid having to prove a refactor
correct!
Generally, current coding agents have exceptionally bad taste, they will produce
awful, slow code, suggest the wrongest possible fixes. Their output needs to be
monitored quite carefully, and no amount of formal methods will meaningfully
improve that.
A recommendation (that, admittedly, I didn’t apply to myself in this project): auditing lemma
statements lost in a sea of updated proofs is very difficult, your eyes will
skip past it more often than not. But you need to audit them because they will
change, either because the program changed and the lemma needs to be adjusted,
or because the agent is trying to be sneaky. So if you ever find yourself
letting a coding agent do the proofs for you, try and make sure that the
statements you truly care about are visually isolated – with at most trivial
proofs – so that their changes stand out.
So in the end, I would have probably got to a finished product faster had I
programmed it by hand. Even as someone who did proofs of programs for my PhD.
But it’s a very different experience than writing a program by hand, so some of
us might like it better. And you get the added benefit of having some formal
proofs to go with it.
Automation is still relevant
LLMs, as it happens, are both quite slow and quite expensive. Formal proofs
currently stress both. To give some idea, I almost always burnt through my weekly
allowance of tokens in under two days’ work; once it just took until the middle
of the afternoon of a single day to spend a week of credits3. I tried using a
cheaper model, but the results were disastrous, formal proofs with LLMs
currently still seem to require some of the most high-powered models.
Generally speaking, bespoke automation will almost always be faster and cheaper
than a generic automation. The strength of coding agents is their generality:
they are capable of making progress on any sort of problem. Their ability to both
non-sequentially and non-locally directly affect the text of the proof gives them a
flexibility that most automation doesn’t have.
But they will spend your time and money doing that. So an agent, to me, seems to
be at its best when it has many tools to orchestrate. A powerful automation will
let the agent skip a lot of steps; bespoke automation tools are also more robust to refactors, which
can help a tonne.
What is lost
Throughout this project, I was struck by how little I understand the code
or the proofs
(though admittedly, I gained quite a bit of understanding of most, not all, of
the implementation
via the process of writing the specification, and responding to proof failures;
the proofs though, I’ve long since lost track).
One way in which it manifests is that I couldn’t intervene when proofs were
taking a long
time. I didn’t know if it was taking time because the agent was getting lost. I
didn’t know if it was taking time because the proof were badly engineered and
what I could do to improve that. I didn’t get a feel for what was happening at
all. So I couldn’t build
automation4 to help proofs be more robust, or easier to deal with, because I
didn’t know what took the agent effort. On a different axis, maybe with some
understanding I could have avoided having a 7000+ line-of-code proof file and
saved a lot of tokens, but I genuinely don’t know what’s going on in there.
When you program or prove by hand, if you’ve done something wrong, you’ll feel
the pain of having to work in a badly designed setting. This pain, in turn, will
both cause you to seek ways to reduce the pain and tell you what part of the
systems are the problem. With coding agents, the cost remains. In fact it’s my
experience that what humans need and what LLM-backed coding agents need to be
effective is quite similar (small, limited-responsibility, well-isolated units).
But the pain is missing. And I don’t know what to replace it with.
Conclusion
I’m really not convinced that code generation constrained by formal proofs is
the future of programming. There are too many challenges, and it’s not even that
efficient. But I want to reiterate that fully generating the code isn’t the only
way that you could plausibly use an agent in conjunction with formal methods.
If you already have formal proofs in your program, I think it’s rather clear
that coding agents help refactoring quite significantly. A rather extreme
example can be found in this blog post from Nik
Swami, where an entire verified
garbage collector was ported from one version of F* to another. It appears to be
still quite labour intensive. But considerably less so than a manual port would have
been.
But formal proofs can’t been seen as a panacea. They don’t fully specify your
code (if anything because they don’t specify the algorithmics of your code). And
this means yet another cost: writing specifications requires expertise, and this
expertise doesn’t replace the programming expertise. Both are required (conflict
of interest declaration: I obviously have services to peddle).
However, I think that the evidence is pretty strong that formal proofs are
getting cheaper. Not necessarily cheap, but noticeably cheaper (see also this
experiment, which is much more seriously conducted than mine). And if formal
proofs really do become cheaper, it means that it will become reasonable to use
them for applications where it would have been unthinkable a few years ago.
Isn’t that exciting?
I suppose you could say that it would make LLMs a form of automation for
formal proofs. Probably because it is. But it doesn’t make formal proof into
an automated formal method. For one thing, you still need to intervene to
suggest proof strategies. For those well-versed in proof assistants, I’d say
LLMs are more like a hammer, and less like an automated tactic. ↩
For completeness, the Rocq implementation is then extracted to OCaml, where I
implement the unspecified parts of the program. This way the SAT solver
is packaged as a normal CLI. ↩
I don’t know exactly how much weekly budget I was given. I can’t give
a concrete number. But, however hard it is to figure out precisely, I seemed
to me that almost all the tokens spent were spent on the proof part. We can
give a guesstimate that proofs have been two orders of magnitude more
expensive to generate than the implementation. ↩
Rocq proofs are built with programmable tactics. It’s customary, when
writing proofs in Rocq, to use the programmability to build yourself a
little toolkit of automated solvers which discharge proof cases of a common
shape in your problem space. ↩
I’ve been a lifelong reader, which has proved a fragile habit in our era of endless trivial
distraction. While books and e-ink readers solve the problem of competition for attention, they
can’t compete with a phone for convenience. That tiny device in my pocket has made reading both
easier and more vulnerable to displacement whenever a notification pops up.
A few days ago, I learned for the first time about a recent generation of tiny e-ink readers.
Since they’re cheap, it was easy to give into curiosity: I bought an Xteink
X3. Not only is it astonishingly tiny, I could immediately
install the delightful open-source
CrossPoint firmware. I was very
quickly able to install a few books. I also appreciated being able to install custom fonts,
though I was a little surprised by their indifferent rendering (foreshadowing…).
I while ago I added a minor mode, haskell-ng-project-mode, to my Haskell mode.
The idea I had was that I could use it to add keybindings for running various
tools (via compile), e.g. I bound cabal build and cabal test, p b and
, p t respectively. Then I hooked in the minor mode in my haskell-ng-mode.
Soon I realised it'd be nice to have those shortcuts available in dired too,
so I hooked it in there as well. When I later wanted to re-run the tests after
modifying a JSON file that was part of a golden test I realised that this wasn't
a very nice solution at all.
After looking around a bit I found emacs-multi-compile. I removed the minor mode
and add this configuration instead
I've had some vague thoughts that it'd be rather easy to write something
slightly more custom, but emacs-multi-compile works very well for me so I'm
very happy with it.
Jumping between implementation and test, again
I wrote about this a while ago. At the time I came up with a way of bending
ff-find-other-file to my will, but soon after that I ran into a case where
there may be 2 "other files" – one of the Haskell projects at work put unit
tests in files ending in Spec.hs and property tests in filed ending with
Prop.hs. Unfortunately this is a situation that ff-find-other-file can't
handle.
I ended up hacking together consult-kith to deal with it. The configuration to
handle two kinds of test files looks like this
From Haskell Language Server2.15.0.0 onwards, users can opt-in to loading all
components of their current project up front, rather than as needed. Doing so
ensures information about every module is fully available from the start, and
avoids pauses to re-initialize the session when opening a file from a new
component.
The feature is enabled by setting componentsLoading to "multi: whole-project" in the HLS configuration.
The next release of Haskell Debugger will load all components up front too.
Debug sessions will then be able to support breakpoints from any module in the
project, not just the entry point’s component.
The feature is provided by hie-bios >= 0.20 and the cradle config (hie.yaml)
can be used to fine-tune the list of components to load. The new
componentsToLoad
field
lets you list the exact targets to pick, which are otherwise the components
listed in the cradle or inferred from the cabal/stack project.
Multi component sessions
Haskell Language Server has supported multi component sessions for a while, e.g.
when working on a Cabal package with both a library and an executable their files
can all be loaded together, so jumping to definition from app/Main.hs can
land you in lib/Foo.hs.
However, so far each new component has been loaded into the session lazily, i.e.
only when files belonging to the component are opened. There are caches
involved, but each time this happens the GHC session is re-initialized from
scratch: the new set of components, and how to build them, is discovered by
querying the underlying build system, e.g. by invoking cabal repl/stack repl
and extracting the flags they would pass to ghci. Ultimately the new component
might have to go from being pre-compiled on disk to interpreted in-memory,
forcing all its descendants to be reconfigured too, so even if we wanted to be
more clever not much work could be saved.
This is not a big deal with just two components, but in a larger project it can be
surprising to have HLS behave differently in a given file according to which
other files have been opened. Also, when the session needs to be re-initialized
because of a new component, the induced pause can be a bit jarring.
Whole project loading
The new whole project mode instead loads all project components right away,
so that everything is available from the start. The downside is doing everything
up front means memory use and initialization time are also paid up front. Also,
a misconfigured component can stop the others from loading.
In Haskell Language Server
Eventually we hope to make whole project loading the default. For the moment,
starting with haskell-language-server-2.15, the new feature is enabled by
setting the new config field componentsLoading to multi: whole-project. The
other options are multi: needed-only, which is the default, and single, which
loads only a single component at once. The componentsLoading field replaces the
previous sessionLoading one which is now deprecated: sessionLoading: "multipleComponents"
is now componentsLoading: "multi: needed-only" and interpreted as such.
In Haskell Debugger
Haskell Debugger was actually the primary motivation for this work, as finding
out in the middle of a debug session that you cannot add a breakpoint in an
imported module is more dire than the issues with HLS.
Prior to whole-project loading, the typical setup where a package contains both
a library and an executable that depends on it would end up with only the
modules from the executable interpreted and so available to be stopped at.
The library was installed in a temporary package database and included only as a
dependency, so trying to set a breakpoint in a module from the library would just fail.
Haskell Debugger builds its session only once, as soon as it learns what the
entry point is, i.e. the module and function to start execution from. At this
point the debugger doesn’t know where the breakpoints will be, and so which
modules need to be interpreted. As new breakpoints can be set while execution
already started, waiting to know all of them to initialize the session is not an
option.
Because of the above we have gone ahead and made whole-project loading the
default and only way Haskell Debugger behaves, starting from the next release.
How it works and further configuration
What “all project components” means depends on the underlying cradle, either
specified in a hie.yaml file or inferred from what can be found in the root
directory. Recent enough Cabal and Stack are the two kinds of cradles that
actually support this: if the hie.yaml file specifies components we load
those, otherwise we ask the package manager to infer them. Inferring means
asking Cabal for target all while enabling tests and benchmarks, or asking
Stack for its default targets.
The above logic, like most of this new feature, is implemented by hie-bios
starting from version 0.20. All details on how to fine-tune your cradle can be
found in hie-bios’s README, including
the new componentsToLoad
field.
Before explaining the new field let’s go through a few examples with Cabal cradles to make
things more concrete. For each case we will list the cabal repl invocation
that hie-bios uses to fetch the correct flags.
A very minimal cradle (hie.yaml) will let cabal infer the components by
using the target all, i.e.
cradle:cabal:
will result in cabal repl --enable-multi-repl --enable-tests --enable-benchmarks all. The --enable-* flags are included so that the
session will cover everything in the project.
The --enable-* flags will override what specified in a cabal.project file,
so if the cradle includes one we omit them, i.e.
cradle:cabal:cabalProject:"./cabal.project"
will result in cabal repl --enable-multi-repl --project-file=./cabal.project all.
If components are specified then those are the targets used, e.g.
will result in cabal repl --enable-multi-repl lib:foo exe:foo.
Finally, if some component doesn’t play nice with cabal repl or the tooling that uses hie-bios,
the field componentsToLoad can be used to specify the exact list of targets to use, for example
will result in cabal repl --enable-multi-repl lib:foo lib:bar.
Conclusion
Hopefully whole-project loading will provide a less surprising and smoother dev
experience on projects with multiple components. It’s been my new default and
working well for me since it got merged. Feedback is welcome on the HLS
issue tracker.
Issue #5005 is the
issue about the feature introduction.
This work has been performed in collaboration with Mercury, who have a long-term
commitment to the scalability and robustness of the Haskell ecosystem.
Well-Typed are always interested in projects and looking for funding to improve
HLS and other Haskell tools. Please contact info@well-typed.com if we might be
able to work with you!
In today’s episode, we’re joined by Brent Yorgey, Associate Professor of Computer Science at Hendrix College and the creator, among other things, of the Diagrams vector graphics library and Swarm, a 2D programming and resource gathering game. We talk about teaching, domain specific languages, and competitive coding.
Adam McKay, director of Don't Look Up, points out what should be obvious but is easy to ignore.
Like all grand lies, “we have more time” takes many different forms, often focus-grouped. There are the lofty-sounding promises to reach “Net-zero emissions by 2050,” as if 24 years from now was soon enough. There are the queries of “What’s your carbon footprint?,” as if individual people with plastic bags or straws were the problem and not gigantic multinational corporations. There’s the accusation that anyone who expresses urgency is a “doomer.” There are those who treat climate as an abstraction to win political campaigns, first arguing that the Biden administration’s Inflation Reduction Act was a victory despite doing nowhere near enough to address the scale of climate collapse, and more recently that Democrats should just stop talking about climate and focus on more “winnable” issues. Most damaging and common of all, there are those who simply don’t mention it.
All of these distortions have robbed us of the most critical element people need to understand a threat and adjust their actions accordingly: context.
And make no mistake, when it comes to how much time we have, the data and the geological record are crystal clear.
For some badly-needed context, the last three years, likely for the first time in over 100,000 years, have averaged above the Paris Accords’ “do not cross” 1.5 degrees celsius planetary warming. Only ten years ago, that was considered catastrophic, but now mainstream news tells us it’s just a number.
...
Let’s do the context thing again: 2C will mean we have completely left the stable climate period known as the Holocene. This is the 11,000-year-old era that has given us large-scale crop and livestock agriculture, which gave us cities, written language, modern science and medicine, and warm beds and showers.
Before that, for hundreds of thousands of years in the Pleistocene era, humans (and our hominid ancestors) spent most of their time running from drought, fires, floods, extreme cold and heat, predators, and each other.
The population was likely no more than a few million people spread across the whole planet. At one point we may have almost gone extinct, with just 10,000 Homo sapiens clinging to life. But after the advent of the Holocene and its remarkably stable climate, in only 100 centuries the world population has risen to 8 billion.
That is what we stand to lose. In fact, if our institutions, news and elected officials continue to feed and water the oil company-conceived mega-falsehood that climate breakdown is something just “our great-grandkids need to worry about,” human civilization as we know it will collapse and billions could die. And we’re not talking about the far distant future. We are talking about collapse within years, not centuries. For real.
I've searched with little success for ideas about how we can handle AI so that it benefits rather than harms society. Here is a suggestion from Jaron Lanier. It was published by the New Yorker in 2023, when GPT-4 was the latest model. Time named Lanier to its list of 100 most influential people in 2015.
A program like OpenAI’s GPT-4, which can write sentences to order, is something like a version of Wikipedia that includes much more data, mashed together using statistics. Programs that create images to order are something like a version of online image search, but with a system for combining the pictures. In both cases, it’s people who have written the text and furnished the images. The new programs mash up work done by human minds. What’s innovative is that the mashup process has become guided and constrained, so that the results are usable and often striking. This is a significant achievement and worth celebrating—but it can be thought of as illuminating previously hidden concordances between human creations, rather than as the invention of a new mind.
As far as I can tell, my view flatters the technology. After all, what is civilization but social collaboration? Seeing A.I. as a way of working together, rather than as a technology for creating independent, intelligent beings, may make it less mysterious—less like hal 9000 or Commander Data. But that’s good, because mystery only makes mismanagement more likely.
...
This concept, which I’ve contributed to developing, is usually called “data dignity.” It appeared, long before the rise of big-model “A.I.,” as an alternative to the familiar arrangement in which people give their data for free in exchange for free services, such as internet searches or social networking. Data dignity is sometimes known as “data as labor” or “plurality research.” The familiar arrangement has turned out to have a dark side: because of “network effects,” a few platforms take over, eliminating smaller players, like local newspapers. Worse, since the immediate online experience is supposed to be free, the only remaining business is the hawking of influence. Users experience what seems to be a communitarian paradise, but they are targeted by stealthy and addictive algorithms that make people vain, irritable, and paranoid.
In a world with data dignity, digital stuff would typically be connected with the humans who want to be known for having made it. In some versions of the idea, people could get paid for what they create, even when it is filtered and recombined through big models, and tech hubs would earn fees for facilitating things that people want to do. Some people are horrified by the idea of capitalism online, but this would be a more honest capitalism. The familiar “free” arrangement has been a disaster.
...
Consider what might happen if A.I.-driven tree-trimming robots are introduced. Human tree trimmers might find themselves devalued or even out of work. But the robots could eventually allow for a new type of indirect landscaping artistry. Some former workers, or others, might create inventive approaches—holographic topiary, say, that looks different from different angles—that find their way into the tree-trimming models. With data dignity, the models might create new sources of income, distributed through collective organizations. Tree trimming would become more multifunctional and interesting over time; there would be a community motivated to remain valuable. Each new successful introduction of an A.I. or robotic application could involve the inauguration of a new kind of creative work. In ways large and small, this could help ease the transition to an economy into which models are integrated.
...
There are also non-altruistic reasons for A.I. companies to embrace data dignity. The models are only as good as their inputs. It’s only through a system like data dignity that we can expand the models into new frontiers. Right now, it’s much easier to get an L.L.M. to write an essay than it is to ask the program to generate an interactive virtual-reality world, because there are very few virtual worlds in existence. Why not solve that problem by giving people who add more virtual worlds a chance for prestige and income?
Last time, we took a stroll down memory lane to review some ancient code I wrote, and see what/how I’d do things differently now. That was fun, and I have a new banger for us today.
Today’s example comes from the same place — a little vim-like text editor I was working on. Vim has this “repeat” operator which just does the same thing you did last time. It’s nice. Delete a word. Running repeat deletes another word. Kinda dinky in this example, but it can also rerun “increment every number in the next four paragraphs.”
Anyway, so that’s the feature I was trying to duplicate. But the action I was running might have done some IO, and I didn’t want to repeat the IO (because maybe the IO was to gather some input from the user, and repeating shouldn’t ask again.)
-- | A monad transformer adding the ability to record the results of IO actions-- and later replay them.dataAgain m ainstanceMonad m =>Monad (Again m)instanceMonadIO m =>MonadIO (Again m)instanceMonadTransAgain-- | Runs an action and records all of its sampled IO. Returns a action which-- when invoked will use the recorded IO.runAgain ::Monad m=>Again m a-> m (m a)-- | Marks an IO action to be memoized after its first invocation.again :: ( MonadIO m , Typeable r )=>IO r->Again m r
I’m actually pretty impressed with past-me on this one. That’s a clean API, and the implementation was pretty cute.
newtypeAgain m a =Again { runAgain' ::RWST () [Dynamic] [Dynamic] m a }once ::forall m r. ( MonadIO m , Typeable r)=>IO r->Again m ronce action =do a <- dequeue >>= \caseJust x ->return. fromJust $ fromDynamic xNothing-> liftIO action tell [toDyn a]return awhere dequeue ::MonadState [r] m => m (Maybe r) dequeue =do get >>= \case [] ->returnNothing (x:xs) ->do put xsreturn$Just x
The trick here is to keep MonadState [Dynamic] and a MonadWriter [Dynamic] instances around. This code uses the MonadState to pop off IO actions it’s already run, and the MonadWriter to keep track of new IO actions it needs to cache. The “magic” happens in runAgain, which runs it through once with an empty state, and then returns a new action with the MonadWriter results moved into the state for the next time around:
runAgain ::Monad m=>Again m a-> m (m a)runAgain action =do (a, w) <- evalRWST (runAgain' action) () []return$do evalRWST (runAgain' action) () wreturn a
Simple. Clean. Elegant. Too bad it doesn’t actually work.
The following program crashes with an error coming from the unsafe use of fromJust in once:
oops ::Again (StateTBoolIO) ()oops =-- Branch on the StateT lift get >>= \case-- The first time around...False->do-- Cache a unit value once $print ()-- Change the state value lift $ put True-- Since the state changed at the end of the first branch, we will take this-- branch the second time around...True->do-- Calling 'once' the second time around gets its value from the cache,-- which has type @()@. But at this callsite, it ought to have type-- @Bool -> Bool@. Uh oh! f <- once $purenot lift $print ("uh oh!", f True)ohNo ::IO ((), Bool)ohNo =flip runStateT False$do m <- runAgain oops m
The attack here comes from the fact that we can force the Again program down a different code path its second time around. And that second time, it encounters a call to once which has a different type than the one it cached. Thus, trying to reuse the cache leads to what is effectively a type error at runtime.
What actually went wrong here? The problem is that monads are too powerful. Since the only way to compose monads is via bind (m a -> (a -> m b) -> m b), we are forced to extend a monadic value of type m a by a function of type a -> m b. Which is to say that the “next thing to do” in a monadic computation is always going to be a function. And functions are completely opaque. So, once we’re given a monad, there’s simply no way to know what it’s going to do without actually running the continuation.
The problem arises from the fact that that continuation can invisibly branch. My Again monad was attempting to statically analyze the monadic computation, and cache the IO results in a queue. But as oops shows, it’s trivial to break this sort of static analysis. Because the branch is hidden inside of a function, and there’s no way to determine which branches a function didn’t take.
But I didn’t know that eleven years ago, so I give myself a pass on this one. Nevertheless, let’s tackle this problem with the wisdom of ages.
No More Monads
So if the problem with my previous implementation of Again is that it’s a monad, what options do we have? As it happens, there are two very nice options available to us here: arrows and selective applicative functors. We’ll discuss arrows for now, and come back to selectives.
Arrows are a generalization of functions. Like functions, they take two type parameters (an input and an output). Like functions, they come with an identity arrow. They compose “end to end” just like functions do. These three properties are expressed as a Category superclass, which we usually call c and write infix:
classCategory (c ::Type->Type->Type) where id :: x `c` x (.) :: (y `c` z) -> (x `c` y) -> (x `c` z)
In addition to being categories, arrows also come with a means of transforming regular functions into arrows, and of mapping arrows over pairs:
classCategory c =>Arrow c where arr :: (x -> y) -> (x `c` y) (***) :: (x `c` x') -> (y `c` y') -> ((x, y) `c` (x', y'))
What’s neat about arrows is that their values are function-like things, rather than being actually functions. That means when we compose arrows, we compose values whose internals we get to choose. Which in turn, means that static analysis is possible again!
Attentive readers will notice that there isn’t actually any way for arrows-as-such to be able to branch. To gain that capability, they require an additional ArrowChoice, which equips them with the ability to lift arrows over Either:
classArrow c =>ArrowChoice c where (+++) :: (x `c` x') -> (y `c` y') -> (Either x y `c`Either x' y')
To wet our whistle, let’s write a little helper Arrow that shows that the composition of an applicative functor with an arrow is itself an arrow:
newtypeStatic m c x y =Static { unStatic :: m (c x y) }
The arrow instances for (Applicative m, Arrow c) => Static m c are the usual trick of using applicative functions to lift operations into the right place. For example, identity is just pure id, and composition isliftA2` of composition:
instance (Category c, Applicative m) =>Category (Static m c) whereid=Static$pureidStatic g .Static f =Static$ liftA2 (.) g f
We can do the same trick for arr and (***):
instance (Arrow c, Applicative m) =>Arrow (Static m c) where arr =Static.pure. arrStatic f ***Static g =Static$ liftA2 (***) f g
as well as for (+++):
instance (ArrowChoice c, Applicative m) =>ArrowChoice (Static m c) whereStatic f +++Static g =Static$ liftA2 (+++) f g
What’s nice about this Static construct we’ve built is that it gives us two degrees of freedom. We’re free to choose an arrow whose structure we’re going to reuse, and an underlying applicative functor (which can give us “static” effects as we’re building the underlying arrow.)
The idea here is that we can put the cache-building machinery inside of the applicative functor, but the cache-reading machinery inside of the underlying arrow.
Constructing Again, Again
It’s worth knowing about the Kleisli arrow:
newtypeKleisli m a b =Kleisli { runKleisli :: a -> m b }instanceMonad m =>Category (Kleisli m)instanceMonad m =>Arrow (Kleisli m)instanceMonad m =>ArrowChoice (Kleisli m)
which says that any monadic bind function is itself an arrow. Implementing the instances yourself is a fun exercise if you’re new to this stuff. We can reuse Kleisli as our underlying arrow, which when you expand everything out, we get:
newtypeAgain m a b =Again { unAgain ::Static m (Kleisli m) a b }derivingnewtype (Category, Arrow, ArrowChoice)runAgain ::Monad m =>Again m a b -> m (a -> m b)runAgain =fmap runKleisli . unStatic . unAgain
What I think is particularly cool about this is that runAgain is just the composition of the newtype unwrappers, and yet its type is extremely reminiscent of my ten-year-ago implementation:
runAgain -- the original ::Monad m=>Again m a-> m (m a)
This is an instance of a more general principle, which is that the final “observation” you want to make of your type is always a good representation of that type. It’s not the only representation, but it’s always a reasonable choice. Appropriately enough, this is known as a final encoding.
Anyway. How can we implement once? By way of onceF:
onceF :: (MonadIO m, Ord a) => (a ->IO b) ->Again m a bonceF m =Again$Static$do cache <- liftIO $ newIORef memptypure$Kleisli$ \a -> liftIO $do cacheMap <- readIORef cachecase M.lookup a cacheMap ofJust b ->pure bNothing->do b <- m a modifyIORef' cache $ M.insert a bpure bonce ::MonadIO m =>IO b ->Again m a bonce m = arr (const ()) >>> onceF (const m)
onceF works by constructing a cache :: IORef (Map a b) at the Static level. Each call to onceF gets a unique cache. Then we drop into the implementation of the Kleisli arrow (which recall happens at runtime.) Inside the Kleisli, we check to see if the map has a value at the requested a, and if so, return the cached b. If not, we run our function and cache it.
Since there’s no Dynamic tomfoolery here that is existentializing away our types, we don’t need to worry about the oops attack; Haskell’s type system guarantees we can’t construct such a thing.
Because Again isn’t a Monad, it can’t be a MonadIO either. But it’s nice to be able to provide the equivalent of liftIO:
againF ::MonadIO m => (a ->IO b) ->Again m a bagainF =Again.Static.pure.Kleisli.fmap liftIOagain ::MonadIO m =>IO a ->Again m x aagain m = arr (const ()) >>> againF (const m)
So that completes our arrow-based implementation of Again. But there’s still something left undone.
Circling Back to Selectives
I said earlier that selective applicative functors would be an alternative approach to implementing Again. Selectives rightfully sit between Applicative and Monad in the functor hierarchy, but were discovered too late, and everyone was still kind of annoyed about having had just stuck Applicative into the hierarchy.
What, precisely is a selective functor? It’s a different solution to the problem of “monads are too powerful” which gives us a branching primitive to play with. Behold:
classApplicative f =>Selective f where select :: f (Either a b) -> f (a -> b) -> f b
select says that you might have an a, in which case you must run the provided f (a -> b). But maybe you have a b, in which case you may run the effects of the f (a -> b). But nobody’s forcing you to.
Since there are still no raw function continuations to be seen here, we can use select to provide branching to our programs. But all of the branches can be identified statically, since we’re only operating on f values, again, whose internals we can control.
In fact, by aggressively nesting select calls, you can get back a version of monadic bind — so long as you have a finite number of select calls you’d need to make:
bindS :: (Selective f, Enum a, Bounded a, Eq a)=> f a-> (a -> f b)-> f b
(the implementation of which is fun if you’re looking for a challenge)
Anyway, all of this is to say that we could have equally structured our Again as a selective functor rather than as an arrow! But ArrowChoice is at least as powerful as Selective, so given all of our hard work already, we can pull out a Selective instance for free.
What do I mean when I say “at least as powerful?” It means we can derive it for free, given an ArrowChoice instance. Behold, the GenericArrow newtype, which exists only for us to have something to attach some instances to.
newtypeGenericArrow c x a =GenericArrow { unGenericArrow :: c x a }derivingnewtype (Category, Arrow, ArrowChoice)
Here we use newtype deriving to say that if c is a category/arrow/arrowchoice, then GenericArrow c is too. But now we can show that having Arrow is enough to get Functor and Applicative instance:
instanceArrow c =>Functor (GenericArrow c x) wherefmap f c = arr f . cinstanceArrow c =>Applicative (GenericArrow c x) wherepure= arr .const liftA2 f x y = (x &&& y) >>> arr (uncurry f)
Then, given an ArrowChoice, we can get an implementation of select:
instanceArrowChoice c =>Selective (GenericArrow c x) where select te tk = proc b ->do e <- te -< bcase e ofLeft x ->do k <- tk -< b returnA -< k xRight y -> returnA -< y
Here we don’t have the option of choosing to invoke tk in the Right case. But that’s OK, because in Again, doing so would correspond to creating caches for branches that don’t need it.
Given all of this, we can now derive Functor up to Selective for Again:
newtypeAgain m a b =Again { unAgain ::Static m (Kleisli m) a b }derivingnewtype (Category, Arrow, ArrowChoice)deriving (Functor, Applicative, Selective) via GenericArrow (Again m) a
which means that users can now choose between the arrow hierarchy and the functor hierarchy for how they’d like to think about building Again actions. And to nudge them even a little further, we can introduce something that looks like a monad transformer, removing the input parameter:
typeAgainT m =Again m ()runAgainT ::Monad m =>AgainT m a -> m (m a)runAgainT =fmap ($ ()) . runAgain
In 1837 Pierre Wantzel proved that no compass-and-straightedge construction can trisect every angle. The counterexample is 60°: trisecting it would mean constructing cos 20°, which is a root of an irreducible cubic, while a compass and straightedge can only build stacks of square roots: objects of degree 2, 4, 8, never 3. And since 60° is easy to construct, any general method for trisection would have to trisect that one, too.
None of this has stopped people trying. For well over a century, mathematics departments have received a steady stream of constructions from amateurs convinced they’d done the impossible, enough of them that Underwood Dudley filled a whole book, The Trisectors, with them.
I used to be one of those amateurs. Well, I was twelve, and my construction never made it into any mathematician’s inbox. But I had the essentials: a method, a compass and straightedge, a very exasperated geometry teacher, and a conviction that the experts had given up too soon.
What you can trisect
So you can’t trisect an angle, but there are things you can trisect. Say the angle is AVB, with vertex V and measure θ, and mark A and B the same distance out from V. Draw the arc centered at V from A to B. Cutting that arc into three equal pieces is the same as cutting the angle into three, so the whole problem comes down to trisecting that one arc, and that arc is the one Wantzel showed is off limits. But it belongs to a whole family of arcs from A to B, some fatter, some flatter, and other members of the family can be trisected exactly.
Pick one, trisect it, and read the result back as an approximate trisection of the angle. How wrong you are depends on how far apart the two arcs sit.
Idea #1: Trisect the chord instead
Start with the flattest member of the family. Push the arc’s center infinitely far back and the arc straightens into the segment AB itself. Segments can be trisected exactly, by a standard similar-triangles construction, or more slickly through the centroid of a triangle. Either way, trisect AB, draw rays through the two division points, and you have approximate trisectors. Call this chord trisection.
For narrow angles, chord trisection works remarkably well. As the angle shrinks the arc flattens toward its chord, and the error vanishes like θ³/81 (in radians, plus higher-order terms). That cubic is fast: a 4° angle gets trisected to within a few ten-thousandths of a degree. Wide angles are another story. The arc bulges away from the chord, the chord’s trisection points crowd toward the sides of the angle, and as θ approaches 180° the middle third of the trisection swallows the whole angle.
So chord trisection is wonderful for narrow angles and hopeless for wide ones.
Idea #2: Trisect a semicircle instead
This is the method I found that year in geometry class. Slide the arc’s center the other way, up from V as far as it will go, to the midpoint of AB. The arc swells into the semicircle on AB, pointing away from V. A semicircle is easy to trisect, because its thirds are 60° arcs, and you mark those off by stepping the radius along the arc with your compass. Rays from V through the two marks are your approximate trisectors. Call this semicircle trisection.
You’d expect semicircle trisection to do well at wide angles, and it does. At θ = 180° the semicircle stops being a stand-in altogether: V lands at the midpoint of AB, the semicircle coincides with the angle’s own arc, and the trisection is exact. Narrow the angle from there and the true arc pulls away from the semicircle, so the error grows…
Then it does something surprising: it turns around, comes back to exactly zero at θ = 90°, then rises and falls once more on the way down to zero at θ = 0, never getting more than 1.226° from the true trisector anywhere along the way. The worst cases sit at arcsin(1/√3) ≈ 35° and its supplement near 145°.
The zero at 0° is just collapse. A and B merge, the whole angle closes onto a single ray, and the constructed trisectors converge onto it along with everything else. The zero at 90° is the interesting one, and it turns out to be the inscribed angle theorem. A right angle inscribed in a circle subtends a diameter, so a 90° vertex V lies on the very circle the trisection marks were drawn on, and each 60° arc of that circle subtends exactly 30° at V.
You can see how I fell for this. The three cases I knew how to check — 0°, 90°, 180° — were all exact, and an error under 1.226° passes the eyeball test everywhere else. The weakness hides at small angles, which are the hardest to check on paper. As θ approaches 0, semicircle trisection hands back a ray a quarter of the way across the angle instead of a third. Less than a degree off, sure, but when the whole angle is a few degrees, that’s a low bar. The error remains a substantial fraction of the answer no matter how small θ gets.
We have two approximations, then, with opposite strengths. Chord trisection converges cubically for small angles but blows up for large ones. Semicircle trisection stays within about a degree everywhere but converges to the wrong ray as the angle shrinks. It would be nice to get the best of both worlds, wouldn’t it?
Bridge: The exact residual trick
Draw the arc centered at V through A and B, and suppose someone hands you a guess: a point G on the arc, with VG claimed to be a trisector. Set your compass to the chord distance from A to G. Starting at B, step backwards that distance twice along the arc toward A, and call the landing point H.
Claim: the trisector of the (probably small) angle GVH adjacent to VG is exactly the trisector of AVB adjacent to VA.
Proof: Write the guess as arc(A→G) = θ/3 + ε, where ε is its unknown error. Each compass step from B moves an arc of θ/3 + ε, so after two steps H sits at θ − 2(θ/3 + ε) = θ/3 − 2ε from A. The residual arc from H to G is therefore exactly 3ε, and a third of the way from G toward H lands at θ/3 + ε − ε = θ/3, for either sign of ε.
The residual angle GVH is exactly three times the guess’s error, and nothing in the reduction is approximate. Trisecting AVB has been converted, exactly, into trisecting a much smaller angle.
Putting the pieces together
So here’s the plan. Feed any angle at all, 5° or 175°, to semicircle trisection, and it hands back a guess within 1.226° of the true trisector. The residual trick then turns that guarantee into a fresh problem: an angle of at most 3.68° whose trisector is exactly the ray we’re after. And 3.68° is chord territory, small enough for the cubic convergence to bite, handing us a remarkable 0.674 arcseconds of error: roughly the angle spanned by a penny viewed from three and a half miles away.
Nothing stops you from running the residual trick again on the improved answer; each round cubes the error. But one pass already lands within an arcsecond, and iterating would be polishing a quantity that stopped mattering a step ago.
I’ve been wanting to get back into the habit of blogging, and thought a fun project would be to go back through my old code and review it as the programmer I am now. So let’s do that.
Eleven years ago, I was so proud to written my first monad that I typed up a big olblog post about it. This seems like a fun thing to revisit, so let’s do it.
To save you the trouble of reading that ancient-ass blog post, here’s the gist. Given a big piece of state, that has many smaller stateful subcomponents, for example, a World is full of Buffers:
my monad acted like StateT except that you can restrict the piece of the state you’re allowed to manipulate without losing the ability to look at the rest of the state.
In the blog post, this monad is called Jurisdiction, but at some point in the source code it got renamed to JailT. Cool name. Furthermore, the implementation changed rather dramatically from the blog post, so let’s chase the implementation as given. I’ll reproduce it here:
typeRLens r s =Lens s s r rnewtypeJailT s r m a =JailT { runJailT' ::Bool->RLens r s-> s-> m (a, s, RLens r s, Bool) }deriving (Functor, Typeable)instance (Applicative m, Monad m) =>Applicative (JailT s r m) wherepure x =JailT$ \v l s ->pure (x, s, l, v) (<*>) = apinstance (Applicative m, Monad m) =>Monad (JailT s r m) wherereturn=pure ac >>= k =JailT$ \v l s ->if vthendo (x, st', l', v') <- runJailT' ac v l s runJailT' (k x) v' l' st'elsereturn (undefined, s, l, v)
Already there’s a lot to look at. The most immediate thing that strikes me is the formatting; now I’m very much a “use two spaces for indent” kind of guy — Haskell doesn’t provide many opportunities for natural linebreaks, and our horizontal space is much more limited than our vertical space. So don’t throw away your horizontal budget on initial spacing. It’s minor, but being an artisan means caring about the minor details.
newtypeJailT s r m a =JailT { runJailT' ::Bool->RLens r s-> s-> m (a, s, RLens r s, Bool) }deriving (Functor, Typeable)
Much nicer.
The other exciting thing to see in the earlier snippet is that this code comes from before the Functor-Applicative-Monad transition. Which is to say that it’s from the long long ago before Applicative was a superclass on Monad. There’s some history for you.
And then there is the elephant in the room. Whatever this Bool is being passed around by runJailT', the Monad instance branches on it and returns undefined when it’s False. What the hell is that???? Digging into the MonadPlus instance provides some insight:
instance (Applicative m, Monad m) =>MonadPlus (JailT s r m) where mplus x y =JailT$ \v l s ->do x'@(xa, xs, xl, v') <- runJailT' x v l sif v'thenreturn x'else runJailT' y v l s mzero =JailT$ \v l s ->return (undefined, s, l, False)
So mzero returns undefined and some False value, presumably which is there to try to warn someone that there’s an undefined floating around.
This is a stupid design. If you’re ever in the business of needing to store data-which-might-not-be-valid and a separate tag stating whether or not it’s valid, a much better design here would be to just use Maybe a instead of (a, Bool).
In fact, perhaps I realized this later on, because runJailT does exactly this logic:
runJailT :: (Functor m)=>JailT s s m a-> s-> m (Maybe (a, s))runJailT ac =fmap (\(a, s, _, v) ->if v thenJust (a, s) elseNothing). runJailT' ac Trueid
Let’s roll back a bit. The pivotal primitive of JailT is the aptly-named jail:
jail :: (Monad m)=>RLens r' r->JailT s r' m a->JailT s r m ajail l' m =JailT$ \v l s ->do (a, s', _, v') <- runJailT' m v (l . l') sreturn (a, s', l, v')
Pretty reasonable definition here. I’m not sure what the idea behind RLens being a flipped version of Lens' was for. The only odd choice here is that in the big tuple returned in jail, l gets passed along unchanged. We can tell from looking at it that l here is the current lens we’re looking at. But jail is the only primitive that changes the lens, and it takes a JailT as an argument. Which is all to say that there isn’t anything actually stateful going on with the RLens. So without looking any further, I’d bet dollars to donuts that this l getting returned is purely vestigial and serves absolutely no purpose. Which in turn means we should be able to chop it out of the definition of JailT.
Speaking of stupid anti-patterns, here’s another one:
into ::RLens a (Maybe a)into = lens fromJust (constJust)jailMaybe :: (Applicative m, Monad m)=>RLens (Maybe r') r-> a->JailT s r' m a->JailT s r m ajailMaybe l d m = (gets $ view l) >>= \caseJust _ -> jail (l . into) mNothing->return d
This is horrendous for a few reasons. into is an unsafe lens that will crash if you try to read out of a Nothing. But then that behavior is guarded in jailMaybe by checking that it isn’t already nothing. Stupid.
The last function I want to point out is parole:
parole :: (Applicative m, Monad m)=> a->JailT s r m (Maybe a)->JailT s r m aparole d m =do this <- get m >>= \caseJust a ->return aNothing-> put this >>return d
which… does… something. I guess it attempts to run a Maybe-valued JailT, and if it fails, unwinds the state and returns some default a. Rearing its ugly head here, however, is the ghost of the undefined nonsense from mzero. Note that if we invoke parole a mzero we will get a crash rather than the roll-back-and-default behavior that it promises on the tin.
So, all in all, I am not particularly impressed with my first monad. Here’s how I’d write it today.
First, I don’t think I’ve written a “control”-y sort of monad from scratch in the last five years. Whatever behavior you want is almost always just the composition of a few monad transformers. In this case, we have a Reader for the current lens, a State for the original state, and a Maybe to give us mzero semantics.
Since we want our state to roll-back when we take an alternative path, we must put MaybeTunderneath our StateT.1 This gives us the nice formulation of a JailT as:
newtypeJailT s r m a =JailT { unJailT ::ReaderT (ALens' s r) (StateT s (MaybeT m)) a }
(we use ALens' s r here instead of Lens' s r since the latter requires -XImpredicativeTypes. Which happens to be turned on in the original implementation!)
Once nice thing about defining our monad as a newtype over a series of monad transformers, is that it allows us to newtype-derive all of the relevant instances:
No implementation for such things is required, and therefore we know that the derived instances must be correct. Which means we have many fewer things to worry about.
I prefer to use unWhatever as my naming scheme for these newtypes, so that I can provide runWhatever as a more user-friendly function.
runJailT :: s ->Lens' s t ->JailT s t m a -> m (Maybe (a, s))runJailT s l = runMaybeT .flip runStateT s .flip runReaderT l . unJailT
What’s also nice about using standard monad transformers for your implementation is that you can reuse all of their existing combinators. For example, to implement jail all we need to do is invoke withReaderT :: (r' -> r) -> ReaderT r m a -> ReaderT r' m a, which lets us change the type of the reader.
jail ::Lens' t u ->JailT s u m a ->JailT s t m ajail ltu =JailT. withReaderT (\lst -> cloneLens lst . ltu) . unJailT
Notice here that I like to minimize my points without going point-free; jail as written is very legible, but writing it entirely pointfree is a crime against humanity:
Rather than implementing parole directly, we can note that it is the composition of two pieces — (1) attempt something, (2) return a default if it failed. Whenever you notice de-composition opportunities like this, take them!
We can write attempt, which is wildly general and doesn’t care one fig about jails or the legal system:
attempt :: (Alternative m, Monad m) => m (Maybe a) -> m aattempt m =maybe empty pure=<< m
and then parole is merely
parole ::Monad m => a ->JailT s t m (Maybe a) ->JailT s t m aparole a0 j = asum [ attempt j , pure a0 ]
{-# LANGUAGE GeneralizedNewtypeDeriving, FlexibleInstances, MultiParamTypeClasses, DeriveFunctor, ImpredicativeTypes, LambdaCase, DeriveDataTypeable #-}importControl.Applicative (Alternative(..))importControl.Monad (ap, MonadPlus(..))importControl.Monad.StateimportControl.LensimportData.Maybe (fromJust)importData.TypeabletypeRLens r s =Lens s s r rnewtypeJailT s r m a =JailT { runJailT' ::Bool->RLens r s-> s-> m (a, s, RLens r s, Bool) }deriving (Functor, Typeable)instance (Applicative m, Monad m) =>Applicative (JailT s r m) wherepure x =JailT$ \v l s ->pure (x, s, l, v) (<*>) = apinstance (Applicative m, Monad m) =>Monad (JailT s r m) wherereturn=pure ac >>= k =JailT$ \v l s ->if vthendo (x, st', l', v') <- runJailT' ac v l s runJailT' (k x) v' l' st'elsereturn (undefined, s, l, v)instance (Applicative m, Monad m) =>MonadState r (JailT s r m) where get =JailT$ \v l s ->return (view l s, s, l, v) put x =JailT$ \v l s ->return ((), set l x s, l, v)instanceMonadTrans (JailT s r) where lift x =JailT$ \v l s -> x >>= (\a ->return (a, s, l, v))instance (Applicative m, MonadIO m) =>MonadIO (JailT s r m) where liftIO = lift . liftIOinstance (Applicative m, Monad m) =>MonadPlus (JailT s r m) where mplus x y =JailT$ \v l s ->do x'@(xa, xs, xl, v') <- runJailT' x v l sif v'thenreturn x'else runJailT' y v l s mzero =JailT$ \v l s ->return (undefined, s, l, False)instance (Applicative m, Monad m) =>Alternative (JailT s r m) where (<|>) = mplus empty = mzerorunJailT :: (Functor m)=>JailT s s m a-> s-> m (Maybe (a, s))runJailT ac =fmap (\(a, s, _, v) ->if v thenJust (a, s) elseNothing). runJailT' ac TrueidtypeJail s r =JailT s r IdentityrunJail ::Jail s s a-> s->Maybe (a, s)runJail ac = runIdentity . runJailT acjail :: (Monad m)=>RLens r' r->JailT s r' m a->JailT s r m ajail l' m =JailT$ \v l s ->do (a, s', _, v') <- runJailT' m v (l . l') sreturn (a, s', l, v')into ::RLens a (Maybe a)into = lens fromJust (constJust)jailMaybe :: (Applicative m, Monad m)=>RLens (Maybe r') r-> a->JailT s r' m a->JailT s r m ajailMaybe l d m = (gets $ view l) >>= \caseJust _ -> jail (l . into) mNothing->return descape :: (Applicative m, Monad m)=>JailT s s m a->JailT s r m aescape m =JailT$ \v l s ->do (a, s', _, v') <- runJailT' m v id sreturn (a, s', l, v')freedom :: (Applicative m, Monad m) =>JailT s r m sfreedom =JailT$ \v l s ->return (s, s, l, v)-- happens to be unused, because it's kinda insane??overwrite :: (Applicative m, Monad m)=>RLens r' r->JailT r r' m a->JailT s r m aoverwrite l' m =JailT$ \v l s ->do (a, s', _, v') <- runJailT' m v l' $ view l sreturn (a, set l s' s, l, v')parole :: (Applicative m, Monad m)=> a->JailT s r m (Maybe a)->JailT s r m aparole d m =do this <- get m >>= \caseJust a ->return aNothing-> put this >>return dinspect :: (Applicative m, Monad m)=>RLens i r->JailT s r m iinspect l = gets $ view linquire :: (Applicative m, Monad m)=>RLens i s->JailT s r m iinquire l = view l <$> freedomarrest :: (Applicative m, Monad m)=>RLens r' r-> r'->JailT s r m ()arrest l v =do state <- get put $ set l v statearrests :: (Applicative m, Monad m)=>RLens r' r-> (r' -> r')->JailT s r m ()arrests l f =do state <- get put $ set l (f $ view l state) state
vs the
New Implementation
{-# LANGUAGE DerivingStrategies #-}{-# LANGUAGE FlexibleInstances #-}{-# LANGUAGE GeneralizedNewtypeDeriving #-}{-# LANGUAGE MultiParamTypeClasses #-}{-# LANGUAGE RankNTypes #-}importControl.ApplicativeimportControl.LensimportControl.Monad.StateimportControl.Monad.ReaderimportControl.Monad.Trans.MaybenewtypeJailT s r m a =JailT { unJailT ::ReaderT (ALens' s r) (StateT s (MaybeT m)) a }derivingnewtype (Functor, Applicative, Monad, MonadIO, Alternative)instanceMonad m =>MonadState r (JailT s r m) where get =JailT$do l <- asks cloneLens gets $ view l put r =JailT$do l <- asks cloneLens modify $ set l rrunJailT :: s ->Lens' s t ->JailT s t m a -> m (Maybe (a, s))runJailT s l = runMaybeT .flip runStateT s .flip runReaderT l . unJailTjail ::Lens' t u ->JailT s u m a ->JailT s t m ajail ltu =JailT. withReaderT (\lst -> cloneLens lst . ltu) . unJailTescape ::JailT s s m a ->JailT s t m aescape (JailT j) =JailT$ withReaderT (constid) jfreedom ::Monad m =>JailT s t m sfreedom =JailT getattempt :: (Alternative m, Monad m) => m (Maybe a) -> m aattempt m =maybe empty pure=<< mparole ::Monad m => a ->JailT s t m (Maybe a) ->JailT s t m aparole a0 j = asum [ attempt j , pure a0 ]arrest ::Monad m =>Lens' t u -> u ->JailT s t m ()arrest l u = modify $ set l u
Maybe this was interesting to you. It certainly was fun for me, because I like tearing apart bad code, and I don’t get to see much of it now that I have very talented colleagues. Thankfully I have a huge amount of crap code from my past, including an even more egregious monad that we’ll tackle next time around.
Unwrap the definitions of StateT s Maybe a and MaybeT (State s) a to see why.↩︎
I regretted that as soon as I opened a YAML file though. It turns out the
yaml-ts-mode is rather unusable.1 I switched back to yaml-mode and
adjusted the value of treesit-enabled-modes to include what I have use for
(the change):
The customization of treesit-enabled-modes contains a list of all the
supported modes.
I hope someone will take pity on yaml-ts-mode and make it usable soon.
Tree-sitter grammar for Cabal files
When I began tinkering with my own treesitter-based mode for Haskell I failed to
find a tree-sitter grammar for Cabal files so I wrote a rather rough one myself.
My hope was that it could serve as a base, or motivation, for someone else to
write a proper one. It took more than 2 years, but now it's finally happened!
I've switched my mode to use Curtis's grammar and I'll be archiving mine. (The
change.)
Hello editable xref, good bye wgrep
Rahul's post also mentions that editable xref buffers. I had the same itch to
have editable xref buffers, and I'd added a function to turn them into grep
buffers to be able to edit them. I've now switched to using the built-in
support, and removed the function.
Magix is a neat tool that lets us run Haskell programs as scripts1. We put a shebang on top mentioning Magix, list the Haskell packages we need, and ./script.hs just works. This post is about running such a script fast(er) on GitHub Actions.
As our example, we’ll take the static site generator (SSG) I wrote some time ago: BlogShake. It is written as a single Haskell file. It uses Shake to build the website, Pandoc to render posts, and Mustache for templates. The script starts with these Magix directives2:
Magix compiles the script into an executable and runs it. Nothing else to install, no cabal, or nix-shell required.
Nix and Cabal can also run scripts by providing shebang directives3, so why reach for Magix instead? The nix-shell shebang reinterprets the script with runhaskell on every run, which is slow. Cabal compiles the script, but it fetches dependencies from Hackage and builds them from source, leading to very slow first build and rebuilds. Magix compiles the script once into a binary executable and caches it for the next runs. It also fetches dependencies from the prebuilt Nix cache. So running via Magix is faster than either case. But when running on GitHub Actions, we have a problem.
GitHub Actions gives us a fresh runner for every build, with no Nix store and no Magix cache. So every run, we have to install Nix and Magix, download all the dependencies, and compile the script. In one instance, the build from scratch took 105 seconds, with installation, dependency download, and compilation taking 92 seconds.
That is a lot of wasted work because the script and its dependencies change rarely. The actual run itself takes only a few seconds once the executable exists.
What if we could persist the compiled executable across runs? Magix’s build is deterministic: the same script and the same nixpkgs revision produces the same executable. If we could stash that executable somewhere durable, a cache hit could skip Nix and Magix installation, as well as the script compilation entirely, and just run the executable.
Magix creates a Nix derivation for compiling the Haskell script with GHC and builds it. The resultant executable lives in Magix’s cache, as a symlink with a path like:
GHC statically links all the Haskell libraries into the executable. The only dynamic dependencies are a handful of libraries—zlib, libffi, gmp etc. So if we have the executable plus those few libraries as a self-contained unit, we don’t need the hundreds of other packages in the Nix store4.
NAME is the script file name without its extension, and HASH is a SHA-256 hash of the script contents; together they key the bundle directory. MAGIX_CACHE mirrors how Magix itself resolves its cache directory: $XDG_CACHE_HOME/magix or ~/.cache/magix, overridable with the MAGIX_CACHE_DIR environment variable. Bundles live under ~/.cache/magix-bundles by default, overridable with the MAGIX_BUNDLE_DIR env var.
The bundle function locates the script’s build result in Magix’s cache, copies the executable out of the Nix store, and gathers the libraries it links against:
copy_lib_deps(){localsrc="$1"localdepsdepbasedeps=$(ldd"$src"|awk'/\/nix\/store\//{print $3}')for dep in$deps;docase"$dep"in*/*-glibc-*/*)continue;;esac[-e"$dep"]||{echo"error: dependency '$dep' of '$src' not found">&2exit 1}base=$(basename"$dep")if[!-f"$DEST/lib/$base"];thencp-L"$dep""$DEST/lib/$base"copy_lib_deps"$dep"fidone}bundle(){mkdir-p"$DEST/bin""$DEST/lib"RESULT=""for f in"$MAGIX_CACHE/"*"-$NAME-result";do[-L"$f"]||continue[-e"$f"]||continueif[-z"$RESULT"]||["$f"-nt"$RESULT"];thenRESULT="$f"fidone[-n"$RESULT"]||{echo"error: no build result for '$SCRIPT_FILE' found in $MAGIX_CACHE">&2exit 1}RESULT=$(readlink-f"$RESULT")BIN="$RESULT/bin/.$NAME-wrapped"cp"$BIN""$DEST/bin/$NAME"chmod +w "$DEST/bin/$NAME"copy_lib_deps"$BIN"case"$(uname-m)"inx86_64)INTERPRETER=/lib64/ld-linux-x86-64.so.2;;aarch64)INTERPRETER=/lib/ld-linux-aarch64.so.1;;*)echo"error: unsupported architecture: $(uname-m)">&2exit 1;;esacpatchelf\--set-interpreter"$INTERPRETER"\--set-rpath"\$ORIGIN/lib"\"$DEST/bin/$NAME"for lib in"$DEST"/lib/*;do[-f"$lib"]||continuechmod +w "$lib"patchelf--set-rpath"\$ORIGIN""$lib"doneecho"bundle created at $DEST"}
Here is what it does:
It picks the latest <hash>-<name>-result symlink for the script by modified time, skipping dangling ones5.
It dereferences the symlink to a Nix store path, copies the compiled executable at bin/.<name>-wrapped from the Nix store path into the bundle6, and makes the copy writable.7.
It copies the dynamic library dependencies of the executable by calling the function copy_lib_deps. copy_lib_deps finds the dependencies with ldd, and copies the ones that live in /nix/store into the bundle’s lib/ directory, skipping glibc. Note that it does this recursively, copying the dependencies of dependencies as well.
It rewrites the executable with patchelf, setting the interpreter and the library search path for it. It also sets the library search path for libraries themselves so that transitive dependencies work as well.
The two patchelf flags are the interesting part. The --set-rpath flag makes the dynamic loader find the bundled libraries, so we don’t need LD_LIBRARY_PATH or the Nix store at run time. The --set-interpreter flag is needed because Nix rewrites each executable’s dynamic loader to point at its own glibc inside the Nix store. Since we don’t bundle glibc, we reset the interpreter to the host’s loader according to the system’s architecture8.
We deliberately do not bundle glibc. Unlike the other libraries, glibc cannot simply be shipped alongside the executable9. The dynamic loader needs to be at an absolute path and needs to be matched with the glibc version. So we simply don’t bundle it and rely on the host’s glibc. One caveat here is that the host’s glibc must be at least as new as the one the executable was built against. That works because glibc is backwards-compatible10.
The rest of the script is the run subcommand and the command dispatch:
run(){localbin="$DEST/bin/$NAME"if[!-f"$bin"]||[!-d"$DEST/lib"];thenecho"error: no bundle for '$SCRIPT_FILE' at $DEST">&2exit 1fiexec"$bin""$@"}case"$CMD"inbundle)if["$#"-gt 0 ];thenecho"error: 'bundle' takes no extra arguments">&2exit 1fibundle;;run)run"$@";;*)echo"error: unknown command '$CMD' (expected 'bundle' or 'run')">&2exit 1;;esac
run computes the same SHA-256 hash of the script, checks that the bundle exists, and execs the executable, passing the remaining arguments through untouched.
We resolve the nixpkgs branch to its latest commit with git ls-remote at the start of the job, use that commit in the cache key, and pin the whole build to the same commit via MAGIX_NIXPKGS_REF, as we see below.
The cache key has two parts: the hash of the script and the resolved nixpkgs commit hash. When the branch moves or the script changes, the cache misses because of the key change, and we rebuild against the new commit and/or script. We also restore the previously cached bundle to ~/.cache/magix-bundles, if found.
install Nix adds Magix’s binary cache to Nix, so that we get the prebuilt packages for Magix. The compile blog step runs ./blog.hs with no arguments12: This causes Magix to compile the script, but Shake has nothing to build, so it exits immediately. The build is pinned to the same nixpkgs commit that keys the cache, via the MAGIX_NIXPKGS_REF env variable used by Magix. Finally, create bundle packages the compiled executable by running the bash script we saw earlier.
The rest of the workflow runs on every run, cache hit or not:
-name: build siterun: ENV=PROD ./scripts/magix-bundle.sh run blog.hs -j4 build
I’ve cut down the rest of the workflow to the only step that is relevant to us: build site that runs the bundled executable with the script’s arguments. Other steps are specific to BlogShake.
This post showed how to speed up a Haskell script running on GitHub Actions by caching a compact bundle of its compiled executable. The approach works for any Haskell script that runs on Linux: compile once using Magix, bundle the binary with its library dependencies, and let the cache do the rest. One caveat though: the bundler script uses the internal details of Magix, which may break if Magix changes how it works.
Why not cache the entire Nix store between runs? Because the size of the full Nix store closure required to build Haskell scripts is usually in GBs. Caching that per run would defeat the purpose of caching by taking way too much time to download the cache. You may also want to reach out for the Nix bundle feature, which produces self-contained compressed executables. But these executables are still too big: 45 MB compressed/178 MB uncompressed for BlogShake. Our approach in this post results in a 9 MB bundle, compressed.
Another completely different option is to build a fully statically linked executable, which I wrote about in Nix for Haskell: Static Builds. However, that requires a custom toolchain, running which on GitHub Action is too complex and/or slow.↩︎
A result symlink can be left dangling if the store path has been garbage-collected. We skip those and pick the newest live one.↩︎
Magix wraps the built executable with wrapProgram, which renames the real executable to bin/.<name>-wrapped and puts a wrapper script in its place. We copy the real executable.↩︎
Nix store files are read-only mode, and we are about to modify the file, so we make it writable.↩︎
We hardcode the loader paths here, but it should work on most mainstream Linux distributions with glibc.↩︎
Well, it can be actually. That’s what Nix bundle does. It copies the glibc in the Nix store to the bundle, and points the program interpreter at the bundled loader inside a chroot.↩︎
An executable built against glibc 2.42 runs fine on a host with glibc 2.43, but not the other way around.↩︎
The glibc constraint dictates the runner image. Here we build against nixpkgs branch release-26.05, which has glibc 2.42. Ubuntu 26.04, the GitHub runner image we use, ships with glibc 2.43. So they are compatible.↩︎
That no-argument behavior is Shake-specific: with no actions given, Shake runs nothing, so a bare ./blog.hs compiles the script and exits. A general script won’t do that by default. If you adapt this for a non-Shake script, give it a mode that does nothing, say a --compile flag, so running it bare just produces the executable. This step’s only job is to get Magix to build the script, not to run it.↩︎
Thanks for reading this post via feed. Feeds are great, and you're great for using them. ♥
Disclosure: I’m the original author and lead for the Bombadil project at Antithesis.
Carlos Catala, interning at Antithesis and working on testing our webapps, found a strange crash while I was out camping. The tests he works on runs Antithesis’ webapp inside Antithesis and explores it using Bombadil. That surfaced a particular sequence of events leading up to the following crash (logs are simplified for readability):
state machine error: unhandled transition: Running(10) + ActionAccepted(Click("Filters"))
What does this mean? To understand it, we need to back up a little.
The heart of the browser driver is a finite state machine, an abstract model of a tab in Chrome or Chromium, and how we’re interacting with it over CDP. This is necessary due to how you can only invoke certain commands in certain browser states, or it might hang or behave in other confounding ways. In our state machine, some transitions are valid, and those are handled in a big match (state, event) { ... } expression that returns a new state. Invalid transitions are covered by the last branch that returns an Err with a message like the above.
There’s another important component involved which is called the runner. It decides the control flow of the test, and uses the driver to run the system under test. The driver emits a state, and the runner decides what action to take in that state. It hands back the action to the driver so that it can apply it. The browser driver actually pauses the JS runtime when capturing a state, and doesn’t resume execution until the driver has given back the next action.
Hence, the ActionAccepted event (which is when the driver gets a new action from the runner) is not valid in the Running state. It really is only valid in the Paused state. How could this happen? The driver and runner are supposed to operate in lock-step, where the driver is only in control while the JS runtime is paused. This was the nut to crack.
When Carlos told me about this during my PTO, my camping brain was definitely not helpful, so I let it rest until I was back at work. Staring at logs and code also didn’t help much, except for a poorly conceived theory about what could be going on, mislead by the presence of network faults in one test where we saw the bug. We tried but failed to reproduce this locally, so I decided to find a simpler test subject and, crucially, try to reproduce it in Antithesis.
A Simple Reproduction with Causality Analysis
To reduce possible confusion I wanted to first see if I could trigger the same bug in Bombadil with a simpler system under test. I found a Dockerized version of the-internet, an example application by Sauce Labs made for automated browser testing. Based on what we had seen before, I wrote a very small specification that only performs three actions:
In addition to finding various bugs in the Ruby code, it also immediately reproduced the Bombadil bug! Time to dig in. I went through a few instances and started to see a pattern. In addition, this gave me an excellent opportunity to try out Causality Analysis, a mind-bending feature for learning what behavior increases the likelihood of a bug appearing. Here’s the report I got:
The causality analysis report plots the likelihood and helps you understand what events caused a certain bug.
As you may see in the chart, there are three points where the probability increases markedly. Correlating those with the logs (which is easy, you see the bumps highlighted in the log too), I could find the chain leading up to the bug:
Click a link that navigates to /frames (probability goes to 20%)
On that page, click a link that navigates to /iframe (probability goes to 47%)
Go back (I was a bit surprised that this didn’t bump probability noticeably)
Go forward (probability goes to 99.98%)
After the last “Go forward” action, it’s a done deal. Digging into the details in between these points, I could then see how going back to a page which immediately throws an exception, after restoring from bfcache, takes a different path than most page loads do.
The Bug
When going back or forward in history, the state machine immediately transitions to Running, waiting for confirmation by the browser to know what actually happened. Some pages prevent bfcache through JavaScript or HTTP headers, so a history navigation can also end up as a full page load with new HTTP requests. When the page is confirmed to have been restored from bfcache, the page and thus the state machine model skips the Loading state entirely, and goes straight to Running.
In this case, that was followed by the exception being thrown immediately (you can see it in the bottom of the screenshot above), which triggers another path: exceptions thrown while in Running aren’t collected and reported at a later state capture, as is the case for the Loading state — they immediately pause the runtime and trigger a new state read. This in turn let the runner pick a new action to perform. But the CDP event that confirms that the page was restored from bfcache hadn’t arrived yet.
This is a race between the asynchronous communication of runner and browser state machine, selecting the next action, and the CDP event sent by Chromium over a WebSocket confirming the bfcache restore. The unfortunate ordering that triggers the bug is:
Back or forward, restoring from bfcache
Exception thrown on restored page, pausing the debugger and capturing a new state
Runner picking a new action and passing it back to the browser state machine
Confirmation of bfcache restoration coming in, transitioning state machine to Running
State machine applying the ActionAccepted event, which, as we noted in the beginning of this excursion, is not valid when in Running
This sequence causes the crash. It only happens with bfcache-enabled history navigation to a page with an exception being thrown before the confirmation from Chromium comes back. Antithesis’ deterministic fuzzing environment messes with process scheduling and action selection and discovers this sequence reliably.
In Conclusion
I’m not going to describe the fix in too much detail, but it involves a more robust use of the already existing Generation value in the browser state machine. It’s a monotonically increasing number that is used to detect stale timeouts and actions, so that the state machine can discard them. The browser state now carries the current generation, and that is threaded through the runner’s action selection process, so that a stale action (as in step 5 above) can be safely ignored. Additionally, some transitions now increment the generation in cases where all existing timeouts or actions are stale.
I’ve had bugs of this kind before, and understanding them from staring at logs, let alone reproducing them, has been painful. This, in contrast, feels a bit like cheating. Moreover, it extends to race conditions and complicated bugs in the webapps themselves, which makes me very excited about continued work on the Bombadil and Antithesis integration.
Welcome back to Five-Point
Haskell! This is my attempt to codify principles of writing robust,
maintainable, correct, clear, and effective code in Haskell and to dispel common
bad practices (or, heresies) I have run into in my time.
In the last post, we talked about Total
Depravity, which is about treating any mentally tracked constraint or
condition as inevitably leading to a catastrophe and denouncing the reliance on
our flawed mental context windows.
However, stopping here gives us an incomplete picture. Firstly, types aren’t
just about preventing bad behaviors. They’re about designing good code.
Secondly, there is only so much you can do by picking careful structures and
making invalid states unrepresentable. These are still human tools with human
flaws.
The next point, to me, is about an aspect of the type system that I see
little coverage of, but is a doctrine of design that I reach for in almost
everything I write. It’s about leveraging the unyielding properties of math
itself to take care of our fate, even when we are unable to structure
our types well.
So, when writing Haskell, remember Unconditional
Election.
Unconditional Election: The power of the forall to elect or
reprobate instantiations and implementations through parametric polymorphism.
These properties aren’t based on any conditional ad-hoc aspect of types, but are
truly unconditional, predestined by universal quantification.
Surrender your control to parametric polymorphism in all things. Embrace the
“free”-dom of “Free Theorems” from one of Haskell’s greatest unexpected
strengths: the type parameter.
Choice is a Prison
Conditional Election
Learning Haskell can be a journey full of surprises, but this was one of the
ones that blew my mind the most.
Let’s think of a polymorphic function in Java that takes a value of any type
and returns something of that same type:
static<T> T foo(T x)
What could that function do?
Well, it could do IO or throw an exception, mutate the input, or possibly be
non-terminating, but let’s assume all it (and every other example here) does is
purely return a value without mutation. What could it do?
The answer: pretty much anything. It could return the same value it was
given, except if it is an Integer, in which case it negates it:
static<T> T foo(T x){if(x instanceofInteger i){return(T)Integer.valueOf(-i);}return x;}
Same deal in TypeScript or pretty much any other typed language without
parametricity:
functionfoo<T>(x: T): T
Yet again, we have conditional election:
functionfoo<T>(x: T): T {if (typeof x ==="number") {return (-x) asanyas T; }return x;}
But how about Haskell?
foo :: a -> a
(For the rest of this post, let’s ignore non-termination1
and other escape hatches2)
Because Haskell has type erasure and no runtime reflection, the only
possible implementation is simply:
foo :: a -> afoo x = x
So, there is an invariant that appeared somehow in our code: a function of
type forall a. a -> amust leave its value
unchanged!
But wait…says who? Did we insert some sort of const compiler
annotation? Did we add some sort of annotation or pre- and post-condition that
the value cannot change? Are we relying on any sort of foreseeable property of
the value given?
No, this behavior is actually intrinsically fixed! We got this
theorem for free. No need for any sort of work, no need for any
foreseen faithfulness. We didn’t even have to write the function before
knowing all it possibly could be.
This is the power of the forall. The above
foo :: a -> a can be considered “sugar” for:
foo ::forall a. a -> a
If you don’t add a forall a, it is implicitly added. Some
languages, like PureScript and Dhall, require the forall in every
case to be explicit. You can think of the “forall” as like
template <typename T> in C++, a declaration of
what type variable is being quantified over.
Anyway, let’s consider another type signature:
static<T>Stringfoo(T x)
Even if we disallow IO (like getting the time, or system state), in Java (and
most other languages), this again could literally be anything. You can serialize
the object with toString, or you can get its class using
getClass…
In Haskell? A forall a. a -> String cannot use its input! It
must be a constant string!
foo :: a ->Stringfoo _ ="hello"-- orfoo _ ="goodbye"-- orfoo _ ="i always ignore my input"
In fact, there is an isomorphism between String and
forall a. a -> String (fun exercise: write it!)
You can “selectively” bring in capabilities using typeclasses:
foo ::Show a => a ->Stringfoo _ ="hello"-- orfoo x =show x-- orfoo x ="reversed: "<>reverse (show x)
But there are still properties you can enforce: the resulting string can
only depend on the input as far as what is revealed in its
Show instance. Any property not in its Show instance
is off-limits.
Alternatively, you can think of
foo :: Show a => a -> String as:
foo :: (a ->String) -> a ->Stringfoo showVal x =-- ...
and you can see that the only way you can ever inspect a is
through the singular inspection-lens a -> String that you are
given. No getClass(), no back doors, etc.
The Guessing Game
One game Haskellers often play day-to-day is “guess the properties that the
forall ensures” on different type signatures. Let’s try it out!
For example:
mystery :: a -> b -> a
mysteryhas to be \x _ -> x — there is
no other option.
How about:
turmeric :: ((a, b) -> c) -> a -> b -> c
If you think about it, the only option is:
turmeric :: ((a, b) -> c) -> a -> b -> cturmeric f x y = f (x, y)
You can go pretty far down this lane using Haskell as a theorem
prover, in that the type signature represents a proposition and the
implementation represents a proof of that claim. But we’re not going to go down
that route for now, since most practical code is not theorem-proving.
But let’s look at something a bit more structural. How about:
theThing :: [a] -> [a]
What could this do?
Well, we know that all items from the result list must be from the
input list. It must be a “subset” — but the ordering or multiplicity can change.
And more importantly, it can’t depend on anything about the properties
of any a. We also know that if the input is empty, so must be the
output.
From this, we can derive what are called free
theorems to look at properties that any implementation must have.
Namely, mapping a function over the list and calling
theThing must be equivalent to calling theThing and
then mapping:
theThing .map f==map f . theThing
Can you see why? Think of any possible implementation — reverse,
take 3, etc. — and see how this must be the case. However, this is
not true for i.e. sort :: [Int] -> [Int]. Because
sort depends on the actual properties of the items,
map f could change the properties that sort depends
on!
Think about what this can’t do. It clearly selects a single item,
but:
The single item cannot be determined based on any quality or merit of that
item — it can’t be the smallest, the largest, etc.; it has to depend purely on
the position in the list and the length of the list
If given an empty list, it must return Nothing
And again we have the same free theorem,
doIt . map f == fmap f . doIt. No matter how you implement
doIt, it is guaranteed to commute with map and
fmap!
What could this possibly do? Well, we can rule out things like
sum because we can’t use any property of the values themselves. The
only things that this could return are constant functions and functions that
depend on the length but not the contents of the list. We also
have another free theorem, collapse . map f == collapse: mapping a
function shouldn’t change the output, because none of the actual values
matter.
From this type signature, we can conclude that the final list must contain
the same item! The only possible inhabitants are
replicate n for some n, or repeat.3
Again, we have a free theorem: map f . duper == duper . f
-- no free theorem: `take 3 . iterate (+1) :: Int -> [Int]`ghci>take3.iterate (+1) .negate$4[-4,-3,-2]ghci>mapnegate.take3.iterate (+1) $4[-4,-5,-6]-- free theorem: `replicate 3 :: Int -> [Int]`ghci>replicate3.negate$4[-4,-4,-4]ghci>mapnegate.replicate3$4[-4,-4,-4]
How about continuations?
consumeInt ::forall r. (Int-> r) -> r
This might look exotic, but think about what it must do. It receives
a function Int -> r and must produce an r. The only
way to produce an r for all possibler is to
apply the given function to some fixed Int chosen ahead of time.
This type is actually isomorphic to Int itself: the only inhabitant
is \f -> f x for some fixed x.4
consumeInt ::forall r. (Int-> r) -> rconsumeInt f = f 42-- must be a fixed `Int`, cannot dynamically changeconsumeString ::forall r. (String-> r) -> rconsumeString f = f "hello"-- must be a fixed `String`, cannot dynamically change
What’s better than one type variable? How about two?
mapMaybe :: (a ->Maybe b) -> [a] -> [b]
Because this has to work for alla and b,
we know that the only possible bs you can find in your
result list are bs that you get from the
a -> Maybe b function. So, you can be sure that the
implementation doesn’t conjure out any arbitrary b except for the
specific ones producible by your a -> Maybe b. For example, if
you pass it a function that returns only even integers, the resulting list will
only ever contain even integers!
One final one, with a higher-kinded type variable:
traverseIO :: (a ->IO b) -> [a] ->IO [b]-- vstraverse ::Applicative f => (a -> f b) -> [a] -> f [b]
What invariant does the second add over the first? Even if you only
ever plan on calling things with IO, the second gives you a new
invariant: there won’t be any “stray” IO actions other than what is
given in the a -> f b. In the first one, you never know if the
resulting IO action might include a putStrLn "hello"
or a launchMissiles. You definitely don’t want any functions doing
sneaky IO behind your back!
The More you Surrender
Practically, this becomes similar to the principle of least power, the idea
that you should use the tools with the least power necessary to do your job. Say
you are writing a function that shuffles a list of items, important for
your business logic. You can encode exactly what business logic is
being done by adding more and more parametricity.
If your type is [Int] -> [Int], you know your function has
pretty much no restriction on what it can do. It can even look at the machine
representation of your values.
If your type is Num a => [a] -> [a], you know that it can
possibly numerically transform the items in your list, or even conjure up new
items.
If your type is Ord a => [a] -> [a], you know that your
business logic is allowed to look at the ordering between items in the list, but
cannot return any items that weren’t in the original list.
If your type is [a] -> [a], you know that your logic can
only affect the permutation and multiplicity of items in your list.
If your type is Foldable t => t a -> [a], you know that
results come from the input, but you can still reorder, duplicate, or drop
elements. Furthermore, the resulting permutation is “pre-determined” before you
receive any input items.
If your type is Functor f => f Int -> f Int, if you call
with [], you know that the length and ordering of the result will
be preserved, and also any mappings of Ints will be done
purely.
If your type is Monad m => m a -> m a, if you call with
[], you know that the lengths of your results will always be
integer powers of the length of the input, and each input element will be
duplicated the same number of times. So, if you give it [1,2,3],
you know the result’s length has to be of the form 3^k and must
contain 3^(k-1) copies of 1, 2, and
3, in some order.
By switching from concrete types slowly to parametric types, you surrender
control of what your functions can do, and create stronger and stronger
guarantees. In other languages, or with refinement types, you might have to
explicitly declare a post-condition like “the final values must all come from
the original list”. With parametric polymorphism, this is already guaranteed and
elected, no matter what the implementation is.
You can also play this game in the other direction: instead of writing a
post-condition like “the length of the list must be preserved”, you can try to
figure out the level of power you need to get that. For example, if you wanted
to ensure “the result must have the same number of Int items (if
you give it a list), but the order can depend on the actual inspection of the
items”, how would you give that type?
And (spoilers) why does Traversable f => f Int -> f Int
work? And, why, when you pass it a tree of Ints, does it even
preserve the shape of the tree while freely allowing shuffling between
actual leaves based on the Int values themselves?
It’s Only Natural
As an aside, did you wonder where I got those free theorems from? In the
examples above, they come from naturality. Basically, any
forall a. (Functor f, Functor g) => f a -> g a corresponds to
a natural
transformation in category theory, and so must commute with any
fmap.
Basically, if you have a natural transformation
h :: forall a. F a -> G a, with Functor F and
Functor G, then we have:
h .fmap f==fmap f . h
The above examples, forall a. [a] -> [a],
forall a. [a] -> Maybe a, etc. all arise from this. But you
might have to think carefully to see that forall a. a -> [a] is
really forall a. Identity a -> [a]. And can you think of the
Functor that gives us naturality for forall a. [a] -> Int?5
Maybe more surprising than the fact that these free theorems exist is the
fact that their root is intrinsically tied to a branch of math as obscure and
esoteric as “category theory”!
How can we enforce that the userId is not changed?
Maybe if we were in C, we could have a const field:
struct User {constint userId;char userName[20];int userAge;};
But, this applies to all usage of the User struct…what
if we only wanted to preserve this property on a single function? You can’t
declare struct-level const on a single argument!
Instead, we can enforce this by making userId’s type
parameterized:
dataUser uid =User { userId :: uid , userName ::String , userAge ::Int }-- | Guaranteed not to change the IDprocessUser ::User uid ->IO (User uid)
Is this constraint enforced because we carefully designed the structure of
our type? Is it constrained because we added compiler annotations or refinement
types or static analysis? Not quite! It truly did come for free.
What if I wanted to write a function that processed items without adding or
removing any? Just each item in-place?
-- | Invariant: Preserves the ordering of items, and their number.updateItems ::Checklist->IOChecklist
How can we make sure all our implementations are elected to only be
implementations that don’t modify the length of items?
Again the answer can be: add quantification!
dataChecklist t =Checklist { updated ::UTCTime , items :: t (Status, String) }-- | Guaranteed to preseve the same number of `items` (or, number of item-- "slots"), but can still perform IO to get the new Status and StringupdateItems ::Traversable t =>Checklist t ->IO (Checklist t)
Try as you might, you can’t make an implementation that gives you a different
number of items than when you started (even though you can rearrange or replace
them).
Note that we are not adding type parameters for abstraction or to be
able to use “exotic checklists” (Checklist Maybe). Instead, we are
intentionally using them universally quantified in functions that process them,
in order to take advantage of these automatically enforced properties.
This intersects a lot with the Higher-Kinded
Data pattern. Maybe we do have data we want to have multiple
structural versions of:
dataUserF f =User { userName :: f String , userAge :: f Int }typeUser=UserFIdentitytypeNullableUser=UserFMaybetypeUserParser=UserFParsertypeUserDocs=UserF (ConstDoc)typeUserPrinter=UserF (OpString)
In this case, a function like
processUser ::Functor f =>UserF f ->UserF f
will give you a different, unique guarantee for every “shape” your user
has:
For UserF Maybe, the quantification ensures that the
null-or-present property of each field is preserved
For UserF Parser, it ensures that all of the “parsing” logic,
and the set of strings that are validly parsed, is preserved
For UserF (Const Doc), it ensures that the per-field
Doc/documentation is never changed or updated.
For an example, we can write:
processUser ::Functor f =>UserF f ->UserF fprocessUser user =User { userName =fmap (maptoUpper) (userName user) , userAge =fmap (+1) (userAge user) }
This is guaranteed to keep nullable fields null if UserF Maybe,
preserve all successful parses if UserF Parser, and leave any
field-level documentation unchanged if UserF (Const Doc).
All of these properties are mathematically enforced,
unconditionally. It doesn’t depend on any foreseen property of the
types or values we use. These guarantees free us to be able to confidently use
these functions without fear of invariants breaking.
This game becomes even stronger when you consider dependent typing, where we
can express more complex relationships between type variables. For example, in
the case where you have a phantom type (like in this
singletons tutorial):
dataDoorState=Opened|Closed|LockeddataDoor (s ::DoorState)processDoor ::Door s ->IO (Door s)
processDoor, by virtue of taking forall s, must
leave the door state unchanged! It can never open a closed door, unlock a
locked door, etc.
For things like fixed
length vectors, where the length n parameter is the size, what
invariant do you think is preserved in:
something ::Vector n a ->Vector n a
We know that the length of the result must be the same as the length of the
input. Furthermore, with the forall a, we know that every item in
the result must come from the input, but we might rearrange or change the
multiplicity of the occurrences as long as they add to the same original total
number. This might be a good candidate for a function like
reverse.
Or, consider:
somethingElse ::Vector n a ->Vector (n -1) a
From this, we know that the original vector must be non-empty!
Because of how the types must flow for whatever n you give it, this
requires n >= 1.
Ranking Up
Now that you see how useful it is to use type parameters and
forall, can we use this fact at the meta-level even within
our code itself?
Ensuring structural preservation
Let’s say we want to map an IO function over every item in our
UserF, and return a new one. We know that whatever IO function we
use must leave the actual “result” type unchanged. So that means we
must take a forall a. f a -> h (f a).
traverseUser ::Applicative h=> (forall a. f a -> h (g a))->UserF f-> h (UserF g)traverseUser f u =User<$> f (userName u) <*> f (userAge u)
Here again we use the trick above to generalize for all
Applicative h instead of concretely IO, so we can know
that the final action can’t sneak in stray IO.
Ensuring lexically-confined
resources
We can also use this property in phantom types to enforce lexically-confined
resources. Let’s say we are simulating local variables in an
IntMap:
newtypeVar=VarIntnewtypeMemory v =Memory { getMemory ::IntMap v }initVar :: v ->State (Memory v) VarinitVar x = state $ \(Memory mp) ->case IM.lookupMax mp ofNothing-> (Var0, Memory$ IM.insert 0 x mp)Just (i, _) -> (Var (i +1), Memory$ IM.insert (i +1) x mp)readVar ::Var->State (Memory v) vreadVar (Var i) = gets ((IM.! i) . getMemory)writeVar ::Var-> v ->State (Memory v) ()writeVar (Var i) x = modify (Memory. IM.insert i x . getMemory)runWithMemory ::State (Memory v) a -> arunWithMemory = (`evalState`Memory IM.empty)
(By the way, what do we gain from having the state be IntMap v
parametric on v? What guarantees/invariants do we get, what sort of
actions do we forbid the library itself from doing? Is it possible to have a
default-initialized variable?)
We can run operations like:
getFib ::Int->State (MemoryInt) IntgetFib n =do a <- initVar 0 b <- initVar 1 replicateM_ n $do newSum <- (+) <$> readVar a <*> readVar b writeVar a =<< readVar b writeVar b newSum readVar b
ghci> runWithMemory (getFib 10)55
But now our variables are not actually scoped. We could, for instance, run
runWithMemoryinside itself:
myAction ::State (MemoryString) amyAction =do-- new memory starts out empty v <- initVar "hello"-- memory is now (0, "hello")let x = runWithMemory $do-- new memory starts out empty readVar v -- runtime error, looking up '0' in empty map!-- ..-- ..
Now readVar v will fail! Remember that v is
Var 0, but that 0 key only has meaning in the outer
scope. In the inner scope, 0 refers to a different
IntMap, where it is undefined.
We can also do something silly like returning a Var:
ghci> runWithMemory (initVar "hello")Var0
And now that var exists outside its scope. Its binding is gone, so the name
no longer refers to anything meaningful.
We can prevent this by associating every variable with the scope that created
it. Then we can ensure that runWithMemory requires the scope
phantom to never be a part of the final output:
newtypeVar s =VarIntnewtypeMemory s v =Memory { getMemory ::IntMap v }initVar :: v ->State (Memory s v) (Var s)readVar ::Var s ->State (Memory s v) vwriteVar ::Var s -> v ->State (Memory s v) ()runWithMemory :: (forall s.State (Memory s v) a) -> arunWithMemory = (`evalState`Memory IM.empty)
Here, a Var s must come from a Memory s v with the
same scope s. It is associated with that scope, and no
others. The forall here ensures that the action being given cannot
unify with any external s: the scope is freshly created by
runWithMemory.
Right off the bat, this prevents passing variables into nested calls (the
first var’s s is different than the inner scope’s s),
but this also prevents variables from leaking. That’s because the result type
a must be fully independent of the s, so
returning a Var s is illegal, since that would require the
a to depend on s, which escapes the scope of the
forall. (This is exactly how the ST monad works in GHC
standard libraries, actually.)
By requiring the caller to give up control of the s, we
ensure lexical confinement both of the library and of the user-given
continuation. Now this safety doesn’t come from carefully tracking where
variables came from. Instead, it is assured through the universality of the
forall and the unconditional properties it enforces.
Habits to Build
Let’s look at what it looks like to recognize this principle in practice, and
use it in your code. Let’s imagine we have a function that you can use to deploy
a new Config in your environment:
deployConfig ::Config->IO ()
But, deployment is a bit expensive. So we want to deduplicate our deploys:
deploying the same Config twice would be a no-op. We can do this by
keeping a Config in an IORef:
This works, but after learning about the principles in this post,
that type signature should feel a little bit suspicious to you. Note that our
function never actually inspects the Config at all. The
logic is independent. Would there be any value in pulling out the caching logic
generically?
cachedUpdate ::Eq a => (a ->IO ()) ->IORef a -> a ->IOBoolcachedUpdate action cache newVal =do oldVal <- readIORef cacheif oldVal == newValthenpureFalseelsedo action newVal writeIORef cache newValpureTrueupdateConfig ::IORefConfig->Config->IOBoolupdateConfig = cachedUpdate deployConfig
Let’s presume that we never intend to reuse cachedUpdate. So, we
just increased our total lines of code…and for what? What does
cachedUpdate get us?
Firstly, in the original monomorphic updateConfig, written
directly against Config, there is so much that could go wrong.
Maybe you could mis-handle the configuration or accidentally modify it. You
could set certain fields to fixed values. You might end up deploying a
configuration that was never passed in directly.
In our cachedUpdate implementation, we are sure that
any Config deployed will only come directly from
calls to updateConfig. No other operations are possible.
Secondly, the type signature of cachedUpdate tells us a lot more
about what cachedUpdate’s intended logic is and what exactly it can
support. Let’s say in the future, a new requirement comes: Deploy a “default”
configuration if deployConfig ever fails.
You want something as drastic as this to require you to change your
types and your contracts. In fact, if a new requirement comes along and you are
able to implement it without changing your types, that should be extremely scary
to you, because you previously allowed way too many potentially invalid programs
to compile.
If we were to add such a change (“deploy a default Config”), it
should have us go back to the type signature of
cachedUpdate and see how it must change in order for us to support
it. That process of interrogation makes us think about what we actually want to
do and how it would fundamentally change our data flow.
If you subscribe to “SOLID”
programming, this should all remind you of “Dependency Inversion”.
Basically: treat all monomorphic code with suspicion. It may be a symptom of
you trying to hold on to more control, when you should be letting go.
Another example of the same instinct: say you have a function that broadcasts
a notification to a list of users:
notifyUsers :: [User] ->Text->IO ()
How can this branch on the users? Really, in any way. It could inspect each
user, craft different messages for admins vs. regular users, silently skip
certain users, log personal data, vary behavior based on user properties…
Compare that to:
notifyAll ::Foldable t => t recipient -> (recipient ->IO ()) ->IO ()
notifyAll is parametric in recipient. It cannot
inspect who it is notifying. It cannot skip recipients based on their properties
or treat admins differently from regular users. All of that “policy” logic is
forced to live in the caller. The broadcasting logic itself must be
uniform: it calls send indiscriminately on some fixed subset of the
elements. It can pick a subset to send to, but it can’t pick what gets included
in that subset based on the properties of the recipient.
Embracing Unconditional Election
What sort of control are you trying to hang on to in life, in a way that puts
you in your own prison?
To me, the fact that making code more polymorphic and giving up information
is valuable not just for abstraction, but for taking advantage of universal
properties, was a surprising one. But ever since I started writing Haskell, it’s
a fact that I take advantage of every day. So, next time you see the
opportunity, try thinking about what that parametric forall can do
for you. Take advantage of the doctrine of Haskell predestination that arrives
from properties of logic determined before our universe ever existed.
The Next Step
Embracing Total Depravity and Unconditional Election should redefine your
relationship with your code. But not all code lives in the nice pure world where
we can cordon off effects. forall a. [a] -> [a] is very
different than forall a. [a] -> IO [a], after all.
To extend these boundaries to useful code, we have to deal with that boundary
between the world of the pure and the world where things actually happen for real. We’ll explore
the nuances of that boundary in the next chapter of Five-Point
Haskell, Limited Atonement.
Special Thanks
I am very humbled to be supported by an amazing community, who make it
possible for me to devote time to researching and writing these posts. Very
special thanks to my supporter at the “Amazing” level on patreon, Josh Vera! :)
This is the “Fast and Loose Reasoning” condition (Danielsson et
al.).↩︎
Mind that the very important overhaul of the semaphore-compat library to v2 is included in this release and that the semaphore feature will only start working with cabal-install 3.18 again, which is sadly a breaking but necessary change.
A full accounting of these fixes can be found in the
release notes. As always, GHC’s release status, including planned future
releases, can be found on the GHC Wiki status.
We would like to thank these sponsors and other anonymous contributors
whose on-going financial and in-kind support has facilitated GHC maintenance
and release management over the years. Finally, this release would not have
been possible without the hundreds of open-source contributors whose work
comprise this release.
As always, do give this release a try and open a ticket if you see
anything amiss.
The GHC developers are very pleased to announce the availability
of the release candidate for GHC 9.14.2. Binary distributions, source
distributions, and documentation are available at downloads.haskell.org and
via GHCup.
GHC 9.14.2 is a bug-fix release fixing many issues of a variety of
severities and scopes, including:
Fixed a CorePrep miscompilation that could project a field out of an absent
dictionary, resulting in a segfault (#25924)
Fixed demand analysis giving an absent demand to an argument that was still
used by the function’s stable unfolding, which could cause a run-time crash
(#26416)
Numerous fixes for register allocation and Cmm register-conflict analysis
bugs, preventing incorrect code generation and corruption of vector registers
when spilling and reloading (#26411, #26526, #26537, #26542, #26550, #26668)
Many NGG fixes on AArch64: MOVK clobbering live values (#26980), register
clobbering and an incorrect overflow bit in MUL2 (#27046, #27047), and
incorrect sign extension (#26978) and unsigned right shift (#26979) at sub-word
widths
Fixed several black hole handling bugs that could lead to deadlocks or crashes
in multithreaded programs, showing up as hangs or “END_TSO_QUEUE object
entered” errors (#26922, #26936)
Fixed a stack alignment bug on x86 that could cause segfaults or corrupted
registers when using AVX/AVX-512 vector code (#26595, #26822)
Fixed an “unknown/strange object” crash in the compacting garbage collector
(#27434)
Fixed a regression that caused overloaded functions to no longer be
specialised as effectively as in previous releases, hurting runtime
performance (#26831)
Fixed exponential-time desugaring of nested case expressions (#27383, #20251)
Fixes for several compiler panics, including issues with SetLevels (#26681),
the type-class specialiser (#26682), mkTick (#26772, #27121) and
CoreToStg (#27182, #27386)
Fixed cast worker/wrapper incorrectly firing on INLINE functions (#26903)
Fixed negative type literals causing the compiler to hang (#26861)
Fixed associated type family and data family instance changes not triggering
recompilation (#26183, #26705)
Improvements to determinism of compiler output, including the order in which
:info lists instances (#26846, #26858, #26877, #27532)
Fixed split sections support on Windows (#26696, #26494) and the LLVM
backend (#26770)
The JavaScript backend now supports more than 128 registers, fixing runtime
ReferenceError failures for functions taking very many arguments (#26558)
Fixes for the Wasm backend, and the RISC-V and PowerPC native code generators
… and many more
A full accounting of these fixes can be found in the release notes. As
always, GHC’s release status, including planned future releases, can be found on
the GHC Wiki status.
This release candidate will have a two-week testing period. If all goes well
the final release will be available the week of 13 August 2026.
We would like to thank these sponsors and other anonymous contributors
whose on-going financial and in-kind support has facilitated GHC maintenance
and release management over the years. Finally, this release would not have
been possible without the hundreds of open-source contributors whose work
comprise this release.
As always, do give this release a try and open a ticket if you see
anything amiss.
At Tweag, we have been auditing Cardano smart contracts since 2021. We started
out reviewing contracts mostly by hand, but it did not take long before we began
building tooling to make our audits more efficient and systematic — most notably
cooked-validators, which we have maintained and improved
ever since. In its earliest form it could only handle one transaction at a time
and was built on top of plutus-apps — a library that has been
archived for a good while now. From this early reliance on a now deprecated
dependency, it grew over the years into the powerful framework it is today. At
its heart are convenient transaction bodies (or skeletons), focused solely
on what the user actually cares about, and several layers of automation that
build properly shaped and balanced transactions out of
them. On top of these sit fine-grained tweaks that alter skeletons in a
stateful way1 before submission (such as redirecting an output or
minting an additional token), an expressive LTL-inspired DSL that
schedules those tweaks at chosen points along a trace, and the automated attacks
built as instances of that DSL. The most recent improvement was to give this DSL
a proper semantics for negation and its derivatives, such as implication — the
subject of an upcoming post.
As painful as it is to admit, not every tool we built survived.
Pirouette, an experimental static analyser for Plutus Core, was
retired, and Graft, an attempt to generalize our LTL DSL to
a wider range of domains, has been on hold for a long while, waiting for
additional underlying logics and target domains. But even the tools that did not
last taught us a great deal, and each of them contributed to improving our
auditing expertise.
The smart-contract ecosystem did not stand still either. Developers were given
access to more advanced languages, such as Plutarch, a monadic take
on writing Plutus script through a high-level lambda calculus, sticking close to
Plutus Core, or Aiken, a DSL living outside of the Haskell world
specifically geared towards writing and testing smart contracts on
Cardano. Cardano itself kept growing underneath us through its
consecutive eras: Babbage brought reference inputs and inline
datums, Conway introduced a full governance system, and Dijkstra, with nested
transactions, is peeking over the horizon.
Through all of this we audited, we learned, and we occasionally stumbled — our
first audit of Minswap, where a critical vulnerability slipped
past us, is the case we keep coming back to. We also succeeded — more often than
not with critical findings of our own, growing and learning in the process,
while bringing increasing value to our customers. This post is a retrospective
on those five years. It is not a client-by-client account, but a look at the
recurring patterns we keep finding, the lessons we have drawn, and the way our
methodology evolved. Whether you are a smart-contract developer, a fellow
auditor, or simply someone who cares about high-assurance software in general
(or on Cardano in particular), we hope you take away something useful from this
retrospective.
Five years in numbers
Since our first Cardano audit in late 2021, we have carried out more than twenty
audits across the ecosystem, covering products such as decentralized exchanges,
stablecoins, lending protocols, oracles, governance rules, and even a
constitution script. Several clients came back for more than one round as their
protocols evolved, and not every engagement looked the same: most were
standalone smart-contract audits, while some others came with extra layers.
Marlowe, for instance, required not only the typical review of the
on-chain portion of the project, but also one of the Isabelle formal
model that came alongside it.
Across our standard smart-contract audits, we reported 276 findings. The
table below breaks them down by severity, using the same five-level scale we
apply in every report we deliver.
Severity
Findings
Critical
30
High
35
Medium
66
Low
95
Lowest
50
Total
276
The first noticeable element in the table is the 30 critical findings and
35 high-severity findings: issues severe enough that, left unaddressed,
would have dire consequences either for the administrators or for the users of
the protocols, if not both. Among those consequences, an attacker could, for
instance, be mistakenly allowed to drain or lock user funds, cause delays in the
life cycle of the product, or even fully empty pools of assets belonging to the
contract.
The second is the shape of the distribution: the majority of reported findings
sit at medium severity and below. These are smaller concerns that might seem
harmless in isolation, but can pile up to form real risks.
Severity is only one axis. We also classify every finding by its nature —
what kind of problem it is — which tells us as much about a codebase as the
severity does. We use five categories:
Vulnerabilities — directly exploitable flaws that an attacker can use to
break the protocol’s safety.
Implementation bugs — code that does not obey its specification, whether
or not the discrepancy turns out to be exploitable.
Design — issues rooted in the protocol’s architecture, which could cause
serious flaws, such as unnecessary hoops in the contract flow or unnecessarily
large transactions.
Unclear specification — missing, ambiguous, or undocumented expected
behaviour, which makes correctness impossible to assess. After all, a piece of
software can only be deemed sound with respect to a complete and unambiguous
specification.
Code quality — lack of readability, maintainability, or usage of good
coding practices. Theoretically low-impact issues, which could however hide or
even be responsible for deeper flaws.
Nature
Critical
High
Medium
Low
Lowest
Total
Vulnerabilities
24
12
6
2
0
44
Implementation bugs
5
5
10
8
0
28
Design
0
6
18
16
3
43
Unclear specification
1
11
16
31
11
70
Code quality
0
1
16
38
36
91
Total
30
35
66
95
50
276
The two axes line up almost diagonally, which makes sense. Vulnerabilities are
comparatively rare but concentrated at the top: 24 of our 30 critical findings
are outright exploitable flaws. At the other end, code quality is our single
largest category and sits almost entirely at low severity — each issue minor on
its own, but collectively a good measure of how much care a codebase has had. In
between, unclear specification stands out as the second-largest bucket, and it
is a theme we keep returning to: a surprising number of concrete bugs ultimately
trace back to behaviour that was poorly specified, or simply left out of the
specification altogether.
The diagonal shape, however, is a tendency and not a law; the off-diagonal cases
are what make the nature axis worth having. Notice the single occurrence of a
critical finding falling into the unclear specification category. Because the
contract in question relied on the order of its outputs to make decisions, an
attacker could manipulate the output order to duplicate an entry and vote with
more weight than they held. The only safeguard happened to be a legacy field
that, almost by accident, kept the positions distinguishable, and nothing in the
spec said it had that meaning; an unclear specification finding by nature, yet
critical in severity. At the opposite corner, two audits surfaced
vulnerabilities rated at low severity. One such occurrence was linked to the
existence of a peer with high privilege on the contract (the operator), who
could drain the protocol’s reserves at will. However dangerous this may be, the
development team already knew of it and had accepted it as an explicit design
trade-off, because the operator was assumed trusted (a pattern we see very
often, and systematically point out).
Severity measures impact in context; nature measures the kind of flaw and how to
prevent it. The two are correlated — hence the diagonal — but they answer
different questions, and where they diverge tends to be insightful.
Not every audit ends with a long findings list. Our review of CTez, the
only non-Cardano audit we ran, turned up nothing out of the ordinary — which,
although a one-off, and quite surprising, is a valid audit outcome
nonetheless. Marlowe sits at the other end of the spectrum: between
the contract review and the audit of its Isabelle formal model, it
yielded over a hundred findings on its own — which is why we did not include it
in the totals of the tables above. That count, however, says more about the
impressive size of the project than about its safety: the majority of those
findings sat at medium severity or below.
Are smart contracts getting safer?
As we’ve already mentioned, a great deal has changed on the Cardano side over
these five years. The chain has moved from era to era, each one adding
capabilities while pushing further towards built-in safety. Plutus has gone
through several versions, gaining new primitives along the way, and a variety of
new languages have appeared, offering a far more pleasant developer experience
while producing smaller, cheaper Plutus Core. Our own expertise, understanding,
and tooling have grown accordingly. What we have not said yet is that the
developers writing these contracts have improved too: the code we see is
steadily wider in scope, cleaner, and safer, written with more attention to
detail and care for the metrics that matter on Cardano, like transaction and
UTxO sizes.
All of this has changed both the quantity and the nature of the findings we
uncover.
Take the quantity first. There will be no table this time, and for a good
reason. To measure the trend properly, we would want, for each audit, the ratio
between its findings (ideally weighted by severity) and some measure of the
codebase it covered, such as its line count. An exact figure for that ratio is
out of reach: line counts are not comparable across the different languages we
audit, all having different degrees of verbosity, and we do not keep client code
once an engagement is over, so we would not even be able to dig up that number.
But across every engagement, the trend is clear: slowly but surely, that ratio
has come down over the years.
The nature of the findings has shifted too, and to see how, it helps to
understand the model the chain is built on. Cardano runs on the extended UTxO
(eUTxO) model, which differs from the account-based architecture familiar from
most other blockchains. Rather than mutating a global, account-based state, a
transaction consumes unspent outputs (UTxOs) and produces new ones, each
carrying a value and a piece of data called a datum, while on-chain scripts —
the smart contracts themselves — decide whether a transaction is allowed to
spend a given output. Ownership, then, is not an entry in a global ledger but a
question of which UTxOs sit at a given address. When that address corresponds to
the hash of a script, thus belonging to it, “ownership” really means that the
script’s logic runs — and gets to approve or reject the transaction — every time
one of its UTxOs is consumed.
We see fewer and fewer vulnerabilities that are inherent to this model and that
the community has since internalized, such as the well-known double
satisfaction attack, which used to be a common find: when
several instances of the same script run in one transaction, each could be
tricked into treating a single output as its own, not realizing the others saw
it the same way. Some other well-known vulnerabilities have disappeared outright
as the platform matured, like the chicken-and-egg situation of minting tokens
and sending them to a script in a single transaction — where the two scripts
could not each be aware of the other’s hash — finally resolved by multipurpose
scripts.
These classes did not simply fade on their own — we codified many of them.
Attacks we once reconstructed by hand for every engagement, such as datum
hijacking, double satisfaction, or token duplication, are
now expressed once and explored systematically using
cooked-validators, in the shape of tweaks scheduled along a
transaction trace, branching to various possible outcomes. Each new class we ran
into taught us something we folded back into the framework whenever possible, so
the next audit could start from a stronger baseline. The findings shaped the
tooling, and the tooling, in turn, increased our audit efficiency.
What remains tends to be of a different kind: mistakes specific to a contract’s
own domain logic, subtle issues introduced by aggressive optimization — decoding
builtin data by hand, or threading everything through continuation-passing style
to shrink the compiled output — and, above all, design flaws. However much
experience a team accumulates, something always seems to get in the way of an
optimal design — if such a thing even exists. It might be disagreements between
team members, turnover in the people involved, the sheer length of a project
with its successive rewrites and redesigns, budget constraints, or new
requirements surfacing halfway through development. Unfortunately, design issues
are also arguably the hardest and most costly to fix, since doing so often means
rewriting significant parts of a project — which is why they are left
unaddressed more often than other kinds of findings. We hope these findings
still prove useful — for future versions of the same project, for new projects,
or simply for sharpening the community’s collective understanding of how to
approach smart-contract design on Cardano.
Are we there yet?
While the story so far points to a clear improvement in the overall safety of
smart contracts, the journey is far from over. A new Cardano era, Dijkstra, lies
ahead, bringing new capabilities — and, almost certainly, new classes of
vulnerability that have yet to be uncovered. Design issues remain common across
freshly written contracts, and developers are still human, after all, and so
prone to the occasional small, unexpected mistake…
Or are they? No retrospective written today can quite dodge the subject — every
other article seems to mention AI, and, as you can see, this one is no
exception. With the recent explosion in both the use and the capability of LLMs,
smart contracts can increasingly be written with their help. It is tempting to
imagine that they will eventually reshape how contracts are written and improve
their safety to the point where external review becomes unnecessary. We are not
there yet. Our most recent audits still turn up genuine mistakes, and LLMs,
however capable, make mistakes of their own — all the more so given how complex
and expressive the eUTxO model can be.
LLMs are reshaping the way we address audits too. They make us more efficient at
building and maintaining our tooling, and at writing audit code. Although we
value confidentiality and never feed them client code, we can point them at a
contract’s interface — blind to the actual code of the on-chain scripts — to
draft endpoints and traces within cooked-validators and
enjoy a quicker mapping to its most advanced features. That pairing makes our
audits faster and, often, better at surfacing the deep corner-case bugs that
might otherwise have slipped through. So the technology cuts both ways: it may
help developers build safer contracts, and at the same time it sharpens the
audits that check them. Which of these effects will win out, we can’t yet say —
but the trickiest findings, such as the design ones we keep running into, or the
ones fully related to specific business logic, are exactly the kind no amount of
automation is likely to catch reliably. On an eUTxO-based blockchain like
Cardano, where elegant and expressive design meets intricacy and subtlety, we
don’t see third-party review going away any time soon.
Conclusion
This is where the retrospective ends, and what five years and more than twenty
audits have looked like from our side. We started out reviewing contracts by
hand and gradually built cooked-validators into a framework
that lets us schedule attacks along a transaction trace, exploring various
possible outcomes; we wrote tools that lasted and tools that did not, learning
from both; and we watched the ecosystem broaden, with higher-level languages
like Plutarch and Aiken joining Plutus, while Cardano grew
new capabilities underneath us. Along the way we caught a lot of bugs, missed a
few, and got better.
The encouraging part is that things are improving. Whole classes of
Cardano-specific vulnerabilities are fading as the platform and its developers
mature, and the contracts we review are cleaner than they used to be. That does
not make audits any less useful; it just changes what they catch. The next era,
Dijkstra, will bring new capabilities, and with them new ways for things to go
wrong. If these five years have taught us anything, it is that every step up in
abstraction and capability opens new room for mistakes — and catching those
mistakes early is exactly what an audit is for — an endeavour we expect to carry
on for another five years at least.
By stateful we mean that, when altering a skeleton, the tweak has read
access to the current blockchain state — the set of available UTxOs and the
values they carry, for instance — so its modifications can depend on that
context. It cannot, however, modify the state itself; only the transaction
skeleton it is applied to. ↩
Sooo yes it’s true, I’ve been integrating LLMs and agentic coding tools in my
Haskell coding since the beginning of this year for a lot of my projects, both
personal and professional. I do all of my programming in Haskell, a language
with a very expressive type system that encourages “type-driven
development”.
Working with LLMs on writing Haskell is a very unique experience; a lot of
similarities and patterns exist with “normal” agentic coding, but I think the
hot flame of LLMs meeting the cool stone of Haskell yields a lot of wholly
unique optimal paths, workflow quirks, and failure modes.
This post will focus on understanding what I call constraint-evading
behavior in LLMs as it relates to writing Haskell effectively with LLM
collaboration.
Consider this Part 1 of a series. This post is about how to spot and
understand a very common failure mode of LLMs once you actually are
writing “type-driven” Haskell.
The Ideal Case: Haskell and LLMs
Now, my personal opinion and wishful hope is that Haskell should be
to agentic software engineering what Lean is in agentic research mathematics: a
framework for LLMs to self-construct the scaffolding they need to guide
themselves to their correct goal.
I don’t believe that “correctness at generation-time” is a plausible goal,
not today in 2026, and probably not any time soon. Motion towards correctness is
asymptotic. You might get close enough sometimes, but the long tail of
correctness is…long. Even the most full-vibed frontier-model projects have large suites of tests that exist to
guide the agent. Agentic coding in five years will not be “spit out the correct
program”, it will be “set up the best scaffolding that guides the
implementation”.
Remember: the point of types in Haskell isn’t to catch bad code. It’s to
direct how you write and structure your code, guide you down the most productive
paths, help you concretely iterate on what design you want, and make the actual
code-writing time a smooth flow process.
Each part of this is antithetical to how LLMs are trained to use types and
write code.
To this end, I try to structure my codebases with the correct scaffolding to
help augment the intuition of path exploration: AI has to search which paths are
the “most promising”, so I structure my code-base to quickly kill off paths that
are most likely to be dead-ends or lead to unmaintainable code, and to channel
AI exploration along more promising paths. The greatest tools are types,
compilers, warnings, hints.
A lot of these are the same things we teach to human coders, and are not very
different than what I write about regularly:
Parse,
Don’t Validate: set up your types to make invalid states unrepresentable. AI
will, by nature, NOT do this, even the latest frontier models. They usually slip
into defensive programming (adding isNull checks everywhere,
boolean checks for constructors like isNothing /
isObject / isArray instead of just pattern matching,
filtering lists for duplicates instead of using Data.Set, adding
precondition assertions, etc.), and it’s usually up to the human driver to stop
and consciously create data types that model the domain correctly, structurally
prevent invalid states, enforce pre- and post-conditions via structurally
verified types and not boolean checks, avoid boolean blindness, etc.
Either that, or dedicate an entire session or planning step to clarifying
these domain requirements in their types.
Use warnings and linters enthusiastically:
-Werror=incomplete-patterns of course, and the default
-Werror usually covers a lot of AI failure modes (leaving in dead
code, leaving in arguments in functions for no reason and killing opportunities
for abstraction).
Add robust test suites for things that cannot be tested within the types,
but also explicitly laying out integration tests: something which LLMs seem
pretty allergic to without proper prompting.
From my own empirical observations, all of these things are against the
nature of how even frontier LLM models operate, embedded deeply from being
trained on terabytes of untyped Python and React slop.
So, I’ve been gathering a list of what I call constraint-evading
behavior: when the requirements and constraints are explicitly and
unambiguously stated, but LLM nature desperately tries to circumvent
them because they cannot keep up a sustained fight against their deepest base
impulses.
Decisions that require scrutiny
There is a class of failure modes where AI makes a risky decision that a
human might reasonably make in the rare case that it is justified. But,
usually, it will be making this decision because it’s the “simplest approach”. It cannot separate questionable
decisions based on reasoned justification and questionable decisions based on
flawed heuristics like “simplicity” or effort.
We have the general failure modes like this that people mention for all
programming languages:
“This test doesn’t pass, so let’s disable it”
“Let’s feed this test junk data so it will
pass.”
“Let’s have this function detect if it’s in a test
environment and behave differently if it is.”
But here are some that I feel are specific to working with LLMs in
Haskell.
Disabling Warnings for Escape
Hatches
A lot of the “type safety” of Haskell can be bypassed trivially by disabling
warnings, and a lot of the “escape hatches” within the language are disabled via
linting rules (Prelude.error, unsafeCoerce, etc.)
LLMs will often add warning suppressors that straight-up disable warning or
lint checks.
“Let me add -Wno-incomplete-patterns to this
file so that it can compile, because this pattern is inaccessible anyway in
normal operation”
“Let me add HLINT ignore to this build so
that I can bypass the hlint rule forbidding
Prelude.error”
It’s pretty straightforward to add post-edit hooks to forbid edits of this
pattern…but I think this is a good platonic example of what I mean by
“constraint-evading behavior”.
Maybe sometimes you should be disabling warnings in your files.
Maybe sometimes you should be using Prelude.error. A human
might look at the situation at hand and think, “this is one of those
rare cases where Prelude.error is correct”, or “this is one of
those rare cases where that warning is incorrect.”
But should you trust an LLM to make that judgment call? Fuck no. 99% of the
time, it is only doing this as the easy way out. Yes, every once in a while it
will discover a legitimate reason, but has not properly weighted
P(legitimate | attempted). Most of the attempts will be as hacks,
and it will be more than happy to follow through with an attempt if it truly is
the “simplest way”.
Understanding something as constraint-evading behavior doesn’t mean “ban this
behavior”. It is meant to help highlight situations where 99% of the time, it’s
the LLM taking the easy or fast way out instead of the correct one. In these
cases, it’s imperative that a human is what is adding the warning
silencing or hlint ignore.
“The simplest approach is…” is the worst thing you
ever want to see in a thought trace, because it’s a sure guaranteed sign that
they are about to spew the most ridiculous and awful code you’ve ever seen.
Breaking down the matrix:
P(not legitimate && not attempted): Correct avoidance
of problematic behavior
P(legitimate && not attempted): The noble struggle. The
rare case that you are really justified in this normally risky behavior, but out
of misguided principle you do not. This is a bias and failure mode more likely
to be hit by humans. Or at least one human (that’s me).
P(not legitimate && attempted): The failure mode where
an LLM will choose this out of a flawed heuristic like simplicity or
effort.
P(legitimate && attempted): The rare case that you are
justified in normally risky behavior, and the LLM was correct in attempting
it.
As this matrix moves towards more favorable marginals, my stance here will
slowly change. But for now, the numbers I roughly see encourage continued
vigilance.
Ignoring types in planned code
Let’s say you plan your perfect types that match your domain exactly,
forbidding all invalid states, perfectly monotonic parsers. Then you go to
execute that plan. Unfortunately, LLMs will not hesitate to throw away your
carefully designed types.
“The plan says to use NonEmpty Int as an
argument, but that would require changing too much. The simplest approach is to
just have it take [Int] and check for empty lists”
“We planned to use this existing enum, but it doesn’t have
a branch we need. Instead of adding a branch, we’ll have it take
String instead.”
“Instead of a structured data type, let’s just use
stringly encoded lists or records with separators we can parse out.”
“We have to call fooFunc, which returns an
Int, so let’s have our function return an Int instead
of a Natural like our original plan”.
“The plan requires adding a field to this record, but to
keep things simple, let’s just take a Data.Aeson.Object instead so
we can return whatever fields we want.”
“The plan was to have this function be
Binary a =>, but this type we defined doesn’t have a
Binary instance yet, so we’ll just use Show a =>
instead. It’s the simplest approach.”
This is especially frustrating because often these plans and types were
chosen to enforce some domain invariant or guide the proper and correct
development, but LLMs will almost never hesitate before throwing away all of the
planned type safety.
These are all reasonable things that a human might reconsider during
the process of following out a plan. Maybe we originally wanted to use
NonEmpty Int, but upon closer examination, we realized it does have
to be an [Int]. This is the natural process of iterating on a
design, as you discover more truths about the domain.
But, that call should be a discussed one, not an implicit one…it took thought
to make the original plan, so it should take thought to change the plan. Most of
the time AI makes these decisions, it isn’t out of discovered truths about the
domain, but rather because of needless heuristics to minimize effort, or a
misunderstanding of the intent and design of the original plan (especially if
after a compaction). Things that should be discussed explicitly, not done
implicitly. Outside of planning mode, LLMs aren’t seeking out the truth of the
domain, they’re seeking out the path of least resistance.
Note that this is different than weakening types in existing
functions. That’s a failure mode I rarely see in practice. Instead, when running
into a wall with the type of existing code, there’s another failure mode that’s
much more common…
Structural Type Abuse
Sometimes AI will optimize preserving existing types (especially
across package boundaries) instead of changing them.
String Stuffing
I like to call this “string stuffing”. We like to make nice semantic types
that match our domain and only allow the creation of meaningful values…but LLMs
absolutely love to find ways to twist these to save time. Strings, in
particular, are vulnerable because most Haskell types have Show
instances.
And we have to add a new handler for a new request type. Maybe this new
handler has a new type of error. Instead of adding a new structural error, LLMs
will find great joy in cleverly abusing the structure to invalidate the
domain.
“Invalid groups are not a valid ErrorEvent.
The simplest solution is to put the error in UnknownUser, which can
take a group name.”
And yes, this depravity knows no bounds. You would be surprised by the
creative ways AI will discover to stuff your strings. These are all things I
have personally witnessed in frontier models.
This type of failure mode is probably more egregious than the others
in that it is very rare that this is ever the intended behavior. The entire
reason we picked an ADT to describe our type is so that we can structurally
match on them later, treat them semantically, etc., and string stuffing to abuse
our structure has pretty much zero legitimate use-cases other than quickly
hacking a printf debug session. However, it truly is often “the simplest solution”.
The main way I deal with this is to be very very careful of putting abusable
fields like String, A.Value, Int,
SomeException in my data types…just a single field or branch that
has an abusable field, AI will find it, and you will feel very
stupid for missing it. But hey, the whole point of using properly structured
values was to avoid stuffing things into String too, right? The fix
for this is a fix that helps human coders, too.
“I need to specify the affiliations of the authors in this
report. The simplest solution is to add this to the list of
reportAuthors after the authors.”
AI will also stuff sentinel values everywhere: instead of changing the type
to take Maybe Day, it might add ModifiedJulianDay 0
for missing days.
There’s also the dual, where the AI will be happy to use existing
record fields in overloaded ways instead of adding a new field.
Let’s say you need to add a new feature or code path that requires a new
target for a baz service.
“I need to get a new target…instead of adding a new field
to Targets, let’s re-use barTarget. That’s the
cleanest approach.”
It will optimize keeping existing types instead of extending them to match
your domain as your domain expands, especially if those existing types cross a
library boundary.
I believe there are three heuristics at play that drive this behavior.
The heuristic to avoid extra risk in modifying upstream types across
library boundaries with heavy dependencies. This might be very risky behavior in
an untyped language like Python, where each type change might introduce new
regressions that are not immediately obvious. So, avoiding upstream type changes
avoids potential regression.
The heuristic to avoid extra work in modifying upstream types
across library boundaries and compilation units. In Haskell, however, upstream
type changes force you to address each possible regression point
downstream. So changing an upstream type will require you to address every place
it is used, which can be time-consuming and avoided by LLMs, especially if
compilation is expensive, or multiple new typeclass instances might need to be
added.
I have literally seen LLMs say “Adding this field would
require adding typeclass instances on several other types, which would be a huge
change. The simplest approach would be…”
However, updates require work by design: API changes should
require lots of thought, and the compiler enforcing that is the whole point.
The big irony here is that these updates and changes are largely mechanical
in nature, and are exactly the boilerplatey task that LLMs are optimally good
for. These are the reasonable one-shots. So it’s kind of funny when you let an
agentic coder take on a “self-directed” mode, it refuses to use “itself” in the
way that a real human would for these smaller tasks.
The heuristic to avoid extra risk in touching data types that might
already be used in prod code or databases. Config files that might have to be
updated to new schemas, inter-op with existing services that might not be easily
deployed in sync, working with data at rest…all of these are real risks you have
to manage when changing data types. In practice, a lot of our type changes will
not be relevant to any of these concerns, but it is understandable that
the LLM would develop an instinct to blanket-avoid them.
These are also the types of failure modes that are most difficult to catch
during code review. Diff views will analyze that code has changed, so code that
didn’t change is especially difficult for human monkey brains to spot,
with no green or red bright highlighting. You must be especially vigilant to
catch code that did not change but should have.
Resisting New Types
Sometimes AI does modify existing sum types, but doesn’t quite adapt
existing code correctly.
For example, if your domain has a specific meaningful universe:
We might want to start supporting countries alongside US states. An LLM might
recognize that the domain needs to be expanded, but it might expand it
flatly:
dataRegion=Canada|Mexico|Alaska|Arkansas|ArizonaprocessRegion ::Region->IO ()processRegion = \caseCanada->...Mexico->... st -> processState stprocessState ::Region->IO ()processState = \caseCanada->pure ()Mexico->pure ()Alaska->...-- actual logic
The real solution would be to have all your pattern matches strictly reduce
the space of what they cover (and be monotonically decreasing), and to be
suspicious of “ignored case matches” that have dummy values like
pure ():
dataRegion=Canada|Mexico|USStateStatedataState=Alaska|Arkansas|Arizona|...processRegion ::Region->IO ()processRegion = \caseCanada->...Mexico->...USState st -> processState stprocessState ::State->IO ()processState = \caseAlaska->...-- actual logic
All things that are code smells in normal human code (not necessarily wrong,
but invite further scrutiny), but are maybe amplified in the age of LLMs because
of a mis-tuned heuristic on not defining new types and instead trying to re-use
or abuse existing types.
So, if there is some pressure against modifying types, there might
be an even greater pressure against adding types.
Meditations
None of these behaviors are blanket-wrong, but they usually signal that the
LLM is under stress or duress and attempting to find ways to take the easy or
“low-effort” path over the correct one. All of them are worth human intervention
and guidance as soon as possible, at least until the day where
P(legitimate | attempted) approaches 1.
Will there be a day when LLMs can generate the correct types to match the
domain, and resist their tendency to “defensive-program” their way into
correctness? Maybe. But I have rarely ever had Opus 4.8 crank out a sufficient
domain model for any non-trivial product. And, when I do reach a plan I find
sufficient, a few compactions later and all of the original motivations seem to
get washed out.
There might be a way to uber-prompt all of these issues away, but I feel that
effectively using LLMs isn’t necessarily something you can address from the
prompt level: it’s something that demands constant vigilance and care. I’ve
found automated hooks (like forbidding warning-disabling, detecting hlint
bypassing…ask claude for help writing these lol) also help me flag areas that
need attention immediately.
Who knows, maybe all of these things will be solved within a year. But I
still think of software development as something that’s worth scrutinizing for
anything of importance. As failure modes like these become less common…the long
tail of correctness, I predict, will remain long.
Anyway, that’s it for this topic, but if I find the time I’ll
continue on with some other topics I’ve been thinking about during my Haskell
and LLM adventures:
Effective ways to plan out Haskell code and approaches, ways to encourage
the best possible types
Structuring your libraries mechanically for the best build-and-test rapid
development cycle
Starting and maintaining full “vibe-coded” Haskell projects (when
correctness is not critical) and the advantages over untyped vibes.
Let me know if there are any you’d like to see first, or if there are other
aspects of Haskell LLM usage you might like me to address!
And hey, since we’re here, why not train your agentic friend to take these
ideas to heart?
<https://blog.jle.im/entry/llms-and-haskell-1-constraint-evading-behavior.html>
Read this post and make me a Claude Code skill that reviews a Haskell diff for
the constraint-evading compromises it describes: suppressed warnings, string
and field stuffing, and weakened types that differ from any recorded plans.
Some of these hide in code that did not change but should have, so the skill
should start from the functions that changed and evaluate how they use or abuse
the types involved, but also spot type changes that look suspicious.
Recently Martijn Bastiaan, QBayLogic’s COO,
invited Well-Typed to come to QBayLogic HQ and give a one-day workshop on
falsify, the new property based
testing library that I developed for the Haskell Symposium back in 2023
(paper,
presentation).
QBayLogic is the company behind
Clash: a purely functional language for hardware
design. In case you haven’t heard of it, Clash translates Haskell to VHDL or
Verilog, which can in turn be translated to actual hardware. It literally uses
ghc as its frontend so you
have essentially the full power of Haskell available. It’s a great project, I
recommend checking it out.
The falsify library takes its main inspiration from the Python
Hypothesis library, though it is not a direct
translation: it takes the same core idea (“parse, don’t generate”) but
reinterprets it in a way that better suits the Haskell way of thinking: more
axiomatic approach, support for generating infinite data types (including
functions), etc. For more information on falsify, see also the original blog
post that announced it, falsify: Hypothesis-inspired shrinking for
Haskell.
At QBayLogic HQ we spent the morning developing
mini-falsify from scratch, so
that we could focus on the main ideas without getting bogged down in the details
of the full library; similar in spirit to for example
TinyServant,
or perhaps Stephen Diehl’s Typechecker
Zoo. In the afternoon we hacked on
falsify and its application within the clash ecosystem. Partly as a result
of that work and partly as a result of me taking this opportunity to do some
long overdue maintenance on falsify, there is now a new falsify release:
falsify-0.4.0. In the
remainder of this blog post we give a brief overview of the main changes.
New feature: Context
Most of this falsify release is just cleanup, but there is one important new
feature, spear-headed by Peter Lebbing and Martijn Bastiaan from QBayLogic. The
Property monad now has an important new function, called
getContext:
getContext ::PropertyContext
The most important information that the context of a property provides is
how many tests we are running for each property (how hard are we trying to
falsify this property), and which iteration this particular attempt is.
This can be quite useful, for example when you want to start by looking at small
test cases and then slowly broaden the scope. Just as a trivial example,
consider the property that “no number is equal to 5”. If we use
prim
to generate the number to test, producing an arbitrary Word64, the chances
that we will find the one counter-example (5) in that enormous search space
are essentially non-existent:
demo ::Property ()demo =do x <- gen Gen.prim assert $ P.ne .$ ("forbidden", 5).$ ("x", x ::Word64)
However, we could start with a small range and slowly grow that range; falsify
now also offers a convenience function, defined in terms of getContext, for
this specific purpose:
sized,
so-named because it is somewhat similar in spirit to sized in
QuickCheck:
sized ::forall e a. (ProperFraction-> a) ->Property a
A
ProperFraction
is in the half-open interval [0,1); that is, between 0 (inclusive) and 1
(exclusive). We can use this to refine our demo property:
demo2b ::Property ()demo2b =do l <- sized $ ProperFraction.scaleIntegral 100 x <- gen $ Gen.inRange $ Range.inclusive (0, l) assert $ P.ne .$ ("forbidden", 5).$ ("x", x ::Word64)
This property is easily falsified.
As an aside, I would be somewhat cautious in using this approach. While it is
sometimes unavoidable, in general I would recommend generating test case of
arbitrary size and then shrinking them down; this is often more likely to
actually find counter-examples. If there are specific edge cases that you want
to hit, write a generator that covers those edge cases specifically, and perhaps
use labelling
(label
and co) to check that those edge cases are indeed covered. However, as this demo
shows, for some properties explicitly searching for small domains first can be
very helpful.
Cleanup
The most important change in this release is a cleanup of the code base:
There is now a separate tasty-falsify
package that provides integration with the tasty test framework; falsify itself now provides Test.Falsify.Driver.
The module hierarchy has been significantly cleaned up. For example, there are now a bunch of new Data.Falsify.* modules for specific datatypes; previously some of these were exported by Test.Falsify.Gen instead; now that module only contains the generators for those datatypes.
Deprecated functions have been removed
Some functions have been renamed and some type aliases have been replaced by newtype definitions for increased API clarity.
All Haddock warnings have been addressed.
For a full list of changes, please refer to the changelog.
Conclusions
Shrinking can be handled manually, in the style of
QuickCheck; or
automatically, in the style of
hedgehog or in the style of
falsify. If you are willing to
put effort into writing good shrinkers for all your types, then QuickCheck is
still your best bet.
If you consider the cost of manually writing shrinkers too large, then within
the Haskell ecosystem you have a choice between hedgehog and falsify. The
former has the considerable advantage that it’s been around for quite a long
time and is a very polished library. The downside is that every time you use
monadic bind you introduce a cut-point, which can often result in poor quality
shrinking. With Hypothesis showing the way,
falsify solves that cut-point problem, though even with falsify it still
matters how you write your generators: shrinking is never truly free.
Moreover, falsify should be considered an experimental library: it’s nowhere
near as battle-tested as Hedgehog, never mind QuickCheck. That said, thanks to
the interest of QBayLogic falsify is now a little more mature, so thank you
QBayLogic!
For clients who are looking for professional support for the use of falsify,
or indeed any other Haskell library, don’t hesitate to contact us at
info@well-typed.com.
Topiary is a uniform formatter designed to support multiple languages through a single, consistent interface. By relying on Tree-sitter grammars and formatting queries, it easily adapts to new languages. But what happens when a language contains other languages inside it?
I’m excited to share how Language Injections work in Topiary, a new feature allowing Topiary to format “forests” of syntax trees embedded within one another. This unlocks support for formatting embedded languages like OCaml snippets inside OCamllex files or code blocks within Markdown documents.
The Problem: Embedded Languages
Many file formats and programming languages permit embedding entirely different syntaxes within them.
Take Markdown, for instance. A Markdown document can contain fenced code blocks for arbitrary languages. You can even nest these embeddings! Consider a Markdown document containing some OCamllex (a lexer generator for OCaml).
Before formatting, the nested syntax might look a bit messy:
Example lexer:
```ocamllex
rule main =parse
|_ {print_string "Hello, ";print_endline
"World!"}```
After passing this Markdown file through Topiary, the Markdown, OCamllex, and even the OCaml code nested inside the OCamllex, are all cleanly formatted:
Example lexer:
```ocamllex
rule main = parse
| _ {
print_string "Hello, ";
print_endline
"World!"
}```
Historically, formatters struggle with this. They either leave the embedded code alone, implement ad-hoc parsing for specific combinations, or risk breaking the inner syntax. I wanted Topiary to handle this reliably.
The Solution: Language Injections
The idea of “language injections” comes from the Tree-sitter ecosystem, where it has become a de facto standard for handling embedded languages. Editors like Neovim, Helix, and Zed all use Tree-sitter’s injection mechanism to provide accurate syntax highlighting, code navigation, and other language-aware features inside embedded documents. The technique works by running a secondary Tree-sitter parser over a region of the host document, producing a separate syntax tree for the injected language. Which regions to inject, and which language to use, is specified through injections.scm query files that ship alongside grammars.
Topiary now adopts this same mechanism for formatting. By delegating formatting tasks to another language’s formatter mid-flight, Topiary can correctly format inner code chunks while still respecting the layout of the host language.
I recently merged the foundational Topiary PRs along with two language integrations to demonstrate this capability in action:
OCamllex / OCaml Injections: Topiary can now format the OCaml semantic actions embedded inside OCamllex files.
Markdown Injections: Topiary can dynamically format code fences in Markdown documents according to their language identifier.
Deep Dive: How It Works Under the Hood
If you’re new to Topiary, the short version is that it formats code by parsing it with Tree-sitter grammars and then applying declarative formatting rules written as Tree-sitter queries. For a fuller introduction, see the announcement blog post or the Topiary Book.
The injection process relies on Topiary’s existing Tree-sitter foundation and extends the atomization model. At a high level, Topiary parses the host document, extracts the injected spans, delegates their formatting to the inner language’s formatter, and stitches the results back together. Here are the technical details of how I achieved this.
1. Discovering Injections with Queries
If a host language definition includes an injections.scm query file, Topiary runs those queries against the host tree prior to standard formatting (a query can be thought of as a pattern-matching function that takes an AST of the host document and produces a list of matching spans). The queries dictate which spans of the host AST contain foreign code.
Topiary supports two methods of resolving the injected language:
Static Resolution:
For languages where the embedded language is always known (e.g., OCamllex always embeds OCaml), the injection query uses an #injection_language! predicate against the relevant AST node (in the tree-sitter-ocamllex grammar, the code block node is handily named ocaml):
Dynamic Resolution:
For formats like Markdown where the inner language isn’t known ahead of time, Topiary dynamically captures the language identifier from the AST (e.g., the info string of a code fence) using the @injection.language capture (a capture simply assigns a name to a matched node in the query):
Once the injection queries identify the embedded spans, Topiary proceeds with the host language formatting. Crucially, Topiary treats the captured @injection.content nodes as leaves during query matching.1 This means the host formatter doesn’t traverse inside them.
After the host document is tokenized into atoms (the basic building blocks of the layout engine, representing either raw text fragments or formatting directives like spaces and line breaks), Topiary iterates through the captured injection spans:
Each injected span is formatted independently by its corresponding inner language formatter.
The inner formatters execute as if they are producing text starting from column zero.
The resulting formatted string replaces the corresponding host leaf in the atom stream.
There is no “re-parse” phase after the injected text is rewritten. The host renderer retains control over the indentation; when it finally prints the substituted atom, it applies the appropriate indentation string from the host context.
Because step 1 invokes the full formatting pipeline, injections are naturally recursive. If the injected language itself defines an injections.scm file, its injections will be discovered and formatted in turn. For example, formatting the nested Markdown/OCamllex/OCaml snippet from the top of this post doesn’t require any additional effort: Topiary will format all three layers without any special orchestration. Markdown delegates to OCamllex, which delegates to OCaml, each through the exact same code path!
3. Performance
The most expensive part of formatting an injected language is compiling its grammar and parsing its query files. If Topiary recompiled the queries for every single injected span, formatting a document with 100 embedded snippets would be prohibitively slow.
To solve this, Topiary needs a way to compile the grammar once and share it across all matching spans. This ensures the total cost is sublinear in the number of injections.
3.1. The LanguageResolver Hook
To achieve this caching dynamically, I introduced the LanguageResolver type alias to the core formatter API. Topiary is not just a CLI; it is also a library. Because the core library doesn’t know how languages are configured or where grammars live on disk, I delegate the responsibility of resolving languages to the frontend via this hook:
dyn Fn(&str): The resolver is a trait object, specifically, a callable value that takes a language name (e.g., "ocaml" or "rust") as a string slice. Using a trait object here means the core formatter doesn’t need to know which concrete function performs the resolution; it just calls whatever the CLI (or any other frontend) hands it.
-> FormatterResult<Option<Arc<Language>>>: The return type is a Result wrapping an Option:
Ok(Some(language)) means the language was found and loaded. Topiary proceeds to format the injected span with that language’s grammar and queries.
Ok(None) is a soft failure: the language isn’t configured or isn’t supported. Topiary skips formatting for that span gracefully, leaving it untouched.
Err(...) is a hard failure (e.g., a query file failed to parse). Topiary aborts the entire formatting run and returns an error without modifying the file.
Arc<Language>: The resolved Language is wrapped in an atomically reference-counted pointer. Topiary compiles the Tree-sitter grammars and queries into a Language for formatting; Arc enables caching. The CLI can compile the Language on the first call and return cloned Arc handles on subsequent calls, ensuring the compilation overhead is shared across all injected spans of the same language.
+ 'a: A lifetime bound tying the resolver to a borrow scope. In practice, this gives the resolver the flexibility to temporarily reference external data (such as the CLI’s configuration cache) without requiring permanent ownership of it.
This design decouples Topiary’s core formatting logic from any particular frontend. The core library doesn’t know how caching works; it just calls the resolver and acts on the result.
Limitations
These changes do a great job in the majority of use cases, but still have a few known limitations.
Firstly, because injected spans are formatted independently, the current injection model is stateless. It cannot express layout decisions that require measuring or choosing between softline layouts across the host/injected boundary. For example, consider a Markdown document like this:
Let's first define a greeting function that takes a string reference:
```rust
fngreet(name:&str){```
Now we complete the function by printing the greeting to standard output:
```rust
println!("Hello, {name}!");}```
Here a Rust function definition is split across two code fences. Since each fence is its own injection, Topiary formats them independently. The first block sees an unclosed brace, and the second sees an indented statement followed by a closing brace with no matching opener. Neither fragment is valid Rust on its own, so the formatter can’t make sensible layout decisions about them. In practice, because the inner language fails to parse, Topiary will abort the entire formatting run and return an error.
However, if you pass the --tolerate-parsing-errors flag, Topiary will do its best. Because each fragment is formatted independently as if starting from column zero, the indentation on the second fragment gets completely stripped:
Let's first define a greeting function that takes a string reference:
```rust
fngreet(name:&str){```
Now we complete the function by printing the greeting to standard output:
```rust
println!("Hello, {name}!");}```
Overall, I believe this is a perfectly acceptable trade-off: purity and statelessness provide an elegant and robust formatting pipeline, which is not worth giving up for gaining the ability to handle fragmented constructs, which are less common.
Another known limitation is that, while Tree-sitter grammars are usually flexible enough to parse individual definitions or expressions, if a grammar strictly expects a full source file and nothing less, it won’t be able to parse short injected snippets correctly. Just like with fragmented code blocks, this will result in a parsing error unless --tolerate-parsing-errors is used. Fortunately, most grammars (e.g. Java, C#, or Rust) are remarkably flexible and do not exhibit this problem.
Conclusion
By leveraging language injection queries and extending the core formatting API, Topiary can now cleanly format embedded code snippets without breaking the host language layout. With language injections, Topiary is no longer just formatting single syntax trees: it’s formatting forests.
This feature will be included in an upcoming release. In the meantime, if you’re working with OCamllex (for which Topiary is, to my knowledge, the only complete formatter!) or writing technical Markdown documents, you can try it out today.
To run it on an OCamllex file using Nix, for example, you can use:
nix run github:topiary/topiary --fmt myfile.mll
Usually embedded language spans are already parsed as leaves by the
tree-sitter grammar itself. But they don’t have to, so enforcing that
Topiary only sees leaves no matter what the grammar says is safer. ↩
I always say, inside every Haskeller there are two wolves, living on opposite
ends of the Haskell Fancy Code Spectrum. Are you going to write “simple
Haskell”, using basic GHC 2010 tools and writing universal Haskell that every
introductory course offers, trying to keep the code as immediately
understandable and accessible? Or are you going to pile in all of the Haskell
type system and evaluation tricks you can find and turn on all the extensions,
and go full fancy?
In my Seven
Levels of Type Safety post, I described different extremes of type safety
and fancy code. I talked about how writing effective code was finding the
correct compromise for the level of communication and safety you need.
But this is not that kind of blog post. This is the kind of blog post where
we celebrate terrifying type-safety, facetious fanciness, and masochistic
meta-analysis. This series is about what happens when we dare to go full fancy.
Let’s write code that is so inscrutable, so painful and torturous to write, yet
so undeniably useful that you can’t help but try to throw it into every
single thing you write and will feel a gnawing emptiness in your soul until you
do.
As our example, let’s write a type-safe method to specify your program as a
series of states, with triggered transitions between them: a type-safe state
machine graph using a type-safe lambda calculus. We want to specify this in a
way that we can write once and then:
be interpretable in a type-safe way within Haskell.
be inspectable with visualizable control flow.
be compilable to multiple actual back-ends, letting you run the same
function under multiple implementations.
This exact thing is something I’ve needed and used multiple times now in
projects. I want to specify one program graph within Haskell, but in a way that
can compile both in C and javascript while also being visualizable and
interactively explorable.
Once you go down this road, everything you ever write will feel woefully
unsafe and limited. And everything you want to write will be hopelessly
inscrutable by normal humans and borderline unusable. But such is the curse we
all bear. Turn around now, you have been warned.
This post will build up the embedded typed expression language. Part 2 will
use that expression language to define typed state machines with embedded
predicates and visualize them, and Part 3 will compile those machines to
different languages and verify they execute identically, with some live
demos.
All of the code here is available
online, and if you check out the repo and run nix develop you
should be able to load it all in ghci:
$ cd code-samples/typed-sm-lc$ nix develop$ ghcighci> :load ExprStage1.hs
The Lambda Calculus
Let’s derive a way to express an algorithm or expression in Haskell that can
be reified and analyzed within Haskell, and eventually be a form we can compile
to different backends, interpret in Haskell, or generate Graphviz visualizations
for.
The strings in ELambda introduce variables, and
EVar refers to the bound variable. As you can see, this is…pretty
untyped. We could easily write something that is meaningless:
Of course, GHC can typecheck our code if we literally write
\x -> x + 3 and reject 1 && 2. But we
aren’t trying to build opaque Haskell code here, we’re trying to represent our
expression as an ADT that we can analyze within the language.
If we want a record projection and one labeled choice with a case analysis
over it:
Now, for the entire point of Expr, we can write a function to
pretty-print it, using the prettyprinter
library:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage1.hs#L253-L299ppPrim ::Prim->PP.Doc annppPrim = \casePInt n -> PP.pretty nPBoolTrue->"true"PBoolFalse->"false"PString s -> PP.pretty (show s)ppOp ::Op->PP.Doc annppOp = \caseOPlus->"+"OTimes->"*"OLte->"<="OAnd->"&&"ppExpr ::Bool->Expr->PP.Doc annppExpr paren = \caseEPrim p -> ppPrim pEVar v -> PP.pretty vELambda n body -> wrap $"\\"<> PP.pretty n <+>"->"<+> ppExpr False bodyEApply f x -> wrap $ ppExpr True f <+> ppExpr True xEOp o x y -> wrap $ ppExpr True x <+> ppOp o <+> ppExpr True yERecord xs -> PP.encloseSep "{ "" }"", "$ [PP.pretty k <+>"="<+> ppExpr False v | (k, v) <- M.toList xs]EAccess e k -> ppExpr True e <>"."<> PP.pretty kEChoice tag x -> wrap $ PP.pretty tag <+> ppExpr True xECase x hs -> wrap $ PP.sep [ "case"<+> ppExpr False x <+>"of" , PP.encloseSep "{ "" }""; "$ [ PP.pretty tag <+> PP.pretty n <+>"->"<+> ppExpr False body| (tag, (n, body)) <- M.toList hs ] ]where wrap| paren = PP.parens|otherwise=idprettyExpr ::Expr->PP.Doc annprettyExpr = ppExpr False
ghci> prettyExpr fifteen(\x -> x *3) 5ghci> prettyExpr badTypeExample1&&2ghci> prettyExpr recordExample{ label ="found", value =7 }.value +1ghci> prettyExpr sumExamplecaseFound7of { Found value -> value +1; Missing message ->0 }
Now, we can write a quick typechecker for this using a greedy type-checking
algorithm (written
out here), which is a fun exercise, but it’s beyond the point of this post.
For our purposes, we’re going to write the in-Haskell evaluator, which is one
sure-fire evidential/constructive way to prove an expression was valid
after-the-fact.
So…how can you “evaluate” this to 15, within Haskell? What would the type
even be? The best we can do at this point is make the entire thing monadic by
returning Maybe or Either, and split out the
expressions we write from the values we can actually evaluate to:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage1.hs#L228-L330dataEValue=EVIntInt|EVBoolBool|EVStringString|EVFun (EValue->MaybeEValue)|EVRecord (MapStringEValue)|EVChoiceStringEValueeval ::MapStringEValue->Expr->MaybeEValueeval env = \caseEPrim p -> evalPrim pEVar v -> M.lookup v envELambda n body ->pure (EVFun (\x -> eval (M.insert n x env) body))EApply f x -> eval env f >>= \caseEVFun f' -> eval env x >>= f' _ ->NothingEOp o x y ->do u <- eval env x v <- eval env ycase (u, v) of (EVInt a, EVInt b) ->case o ofOPlus->pure (EVInt (a + b))OTimes->pure (EVInt (a * b))OLte->pure (EVBool (a <= b))OAnd->Nothing (EVBool a, EVBool b) ->case o ofOAnd->pure (EVBool (a && b)) _ ->Nothing _ ->NothingERecord xs ->EVRecord<$>traverse (eval env) xsEAccess e k ->doEVRecord xs <- eval env e M.lookup k xsEChoice tag x ->EVChoice tag <$> eval env xECase x hs ->doEVChoice tag payload <- eval env x (n, body) <- M.lookup tag hs eval (M.insert n payload env) body
ghci> for_ (eval M.empty plusThree) \caseEVFun f ->print (f (EVInt4)) _ ->putStrLn"not a function"Just (EVInt7)
This kind of works if you remember to thread everything through
Maybe (or Either) or what have you. But this is not
ideal. You should be able to know, at compile-time, that your Expr
is valid. After all, you want to be able to create one “valid”
Expr, and run it at every context. It’s useless to you if every
single time you used an Expr, you had to manually handle the
Nothing case. Your diagram generator, your Haskell runner, your
code generator, will always be in Either even though you know your
Expr is valid, via tests or something. We want GHC to reject badly
typed expressions, so we never need to unwrap or handle a Nothing
or Left!
No, no, this is not okay and not acceptable. We should be able to verify in
the types if an Expr is valid.
Type-Indexed Expressions
Just Add the Index
The next step you’ll see in posts online is to add a phantom index type to
Expr:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage2.hs#L19-L49typedataTy=TInt|TBool|TString|Ty:->TydataPrim ::Ty->TypewherePInt ::Int->PrimTIntPBool ::Bool->PrimTBoolPString ::String->PrimTStringdataOp ::Ty->Ty->Ty->TypewhereOPlus ::OpTIntTIntTIntOTimes ::OpTIntTIntTIntOLte ::OpTIntTIntTBoolOAnd ::OpTBoolTBoolTBooldataExpr ::Ty->TypewhereEPrim ::Prim t ->Expr tEVar ::STy t ->String->Expr tELambda ::STy a ->String->Expr b ->Expr (a :-> b)EApply ::Expr (a :-> b) ->Expr a ->Expr bEOp ::Op a b c ->Expr a ->Expr b ->Expr c
(We’ll explain each part of this declaration eventually)
A phantom type is a type parameter that doesn’t represent any actual
value “contained” inside the data type, but just serves to “tag” or
distinguish values for the compiler to reject or unify things in useful
ways.
This introduces several new language features, so bear with me as I break
them down.
First, we use -XTypeData to define a data kind: Ty
is a kind with types TInt :: Ty, TBool :: Ty, etc. And
in Expr t, we have an expression tagged with t, which
describes the result type. (In fact, all data types are automatically
promoted to the type level with -XDataKinds. You might see the
'Nothing quote prefix syntax in cases where it’s ambiguous if
you’re talking about the data constructor or the type constructor, like
'[] and '(,))
For example, because we have EPrim :: Prim t -> Expr t, and
PInt 3 :: Prim TInt, we have
EPrim (PInt 3) :: Expr TInt: a primitive 3 is an expression
describing an integer.
And because
EOp :: Op a b c -> Expr a -> Expr b -> Expr c, and
OLte :: Op TInt TInt TBool, we have
EOpOLte ::ExprTInt->ExprTInt->ExprTBool
So we can write an operation on two Exprs that typecheck how
we’d expect:
Because of Ty, we can also make a new indexed data type with
phantoms of “fully resolved” values:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage2.hs#L115-L119dataEValue ::Ty->TypewhereEVInt ::Int->EValueTIntEVBool ::Bool->EValueTBoolEVString ::String->EValueTStringEVFun :: (EValue a ->Maybe (EValue b)) ->EValue (a :-> b)
This is what we want to eventually eval into, as we can
guarantee ourselves to get a value of the correct type based on the
Ty:
eValueToInt ::EValueTInt->InteValueToInt = \caseEVInt x -> x
And GHC will verify this as a total pattern match because EVInt
is the only possible way to create an EValue TInt.
Singletons and Existentials
We’ll keep our bound variables stored as an ambient map of variable names to
their evaluated values for now. But, to do this, we need to turn the
heterogeneous EValue t into the homogeneous
Map String SomeValue by wrapping the type variable as an
existential type.
You might notice we have a singleton
for our Ty type, STy, that pops up in multiple
situations.
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage2.hs#L27-L31dataSTy ::Ty->TypewhereSTInt ::STyTIntSTBool ::STyTBoolSTString ::STyTStringSTFun ::STy a ->STy b ->STy (a :-> b)
Firstly, it might help to recognize the general pattern where
STy (the singleton) appears. It usually pops up whenever
we have existentially scoped variables, like in
data SomeValue = forall t. SomeValue (STy t) (EValue t). In this
case, the t is completely lost to the outside world, and
STy t is used to allow us to recover a runtime witness to what
t was, after pattern matching on STy. This is the
dependent sum pattern, and is similar to how Typeable is
used in Data.Dynamic.
In our case, because variables are still stored ambiently in the environment
and validated at runtime, we do need singletons to implement
eval . The type Expr t only specifies the type of the
result, but the type information of the ambient variables is not available. So,
you can write EVar STInt "myVar" :: Expr TInt, but:
myVar might not be a variable in scope at all, so
eval will fail at runtime
myVar might be in scope, but might be a TString
and not a TInt
The first case is easy enough to deal with (M.lookup returns
Nothing), but the second one is a little more subtle. Let’s say we
do have a SomeValue under our key myVar…how
do we make sure it has the correct type?
Runtime Type Equality
We can do ad-hoc pattern matching on EValue, but that won’t get
us too far. Mostly because some of the EValue constructors actually
don’t have enough information for us to validate their actual type (try it!
EVFun will give you a lot of trouble). So, what we can do is write
a function that takes two STy at runtime and unifies them
conditionally if they are the same. We’ll write a function
sameTy :: STy a -> STy b -> Maybe (a :~: b), where
data (:~:) :: k -> k ->TypewhereRefl :: a :~: a
pattern matching on a value of type a :~: b will reveal
that a and b are the same type variable, because the
only way to construct it is with Refl :: a :~: a.
With that, we can write sameTy:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage2.hs#L133-L143sameTy ::STy a ->STy b ->Maybe (a :~: b)sameTy = \caseSTInt-> \caseSTInt->JustRefl; _ ->NothingSTBool-> \caseSTBool->JustRefl; _ ->NothingSTString-> \caseSTString->JustRefl; _ ->NothingSTFun a b -> \caseSTFun c d ->doRefl<- sameTy a cRefl<- sameTy b dJustRefl _ ->Nothing
There’s a typeclass in base (or rather, a “kindclass”),
TestEquality, that encapsulates this pattern:
classTestEquality f where testEquality :: f a -> f b ->Maybe (a :~: b)
We’re now at a higher fanciness level than before. But you might see the
problem here: EVar STInt "x". x might not be defined,
and it also might not have the correct type. Soooo yes, we still have issues
here.
But now at least we can write eval:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage2.hs#L145-L176eval ::MapStringSomeValue->Expr t ->Maybe (EValue t)eval env = \caseEPrim (PInt n) ->pure (EVInt n)EPrim (PBool b) ->pure (EVBool b)EPrim (PString s) ->pure (EVString s)EVar t v ->doSomeValue t' v' <- M.lookup v envRefl<- sameTy t t'pure v'ELambda ta n body ->pure$EVFun$ \x -> eval (M.insert n (SomeValue ta x) env) bodyEApply f x ->doEVFun g <- eval env f x' <- eval env x g x'EOp o x y ->case o ofOPlus->doEVInt a <- eval env xEVInt b <- eval env ypure (EVInt (a + b))OTimes->doEVInt a <- eval env xEVInt b <- eval env ypure (EVInt (a * b))OLte->doEVInt a <- eval env xEVInt b <- eval env ypure (EVBool (a <= b))OAnd->doEVBool a <- eval env xEVBool b <- eval env ypure (EVBool (a && b))
This does seem to work:
ghci> for_ (eval M.empty fifteen) \caseEVInt x ->print x15
Our system also allows us to produce closures and functions as values:
ghci> for_ (eval M.empty plusThree) \caseEVFun f -> for_ (f (EVInt4)) \caseEVInt x ->print x -- compiler-verified to always be EVInt7
We have a type-safe eval now that will create a value of the
type we want. But we still have the same errors when looking at variables:
variables can still not be defined, or be defined as the wrong type.
Pretty-Printing
One nice consequence of this type-index method is that if you choose to
consume them into an untyped target, you can do it more or less in the same way
as the non-indexed untyped data.
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage2.hs#L87-L113ppPrim ::Prim t ->PP.Doc annppPrim = \casePInt n -> PP.pretty nPBool b ->if b then"true"else"false"PString s -> PP.pretty (show s)ppOp ::Op a b c ->PP.Doc annppOp = \caseOPlus->"+"OTimes->"*"OLte->"<="OAnd->"&&"ppExpr ::Bool->Expr t ->PP.Doc annppExpr paren = \caseEPrim p -> ppPrim pEVar _ v -> PP.pretty vELambda _ n body -> wrap $"\\"<> PP.pretty n <+>"->"<+> ppExpr False bodyEApply f x -> wrap $ ppExpr True f <+> ppExpr True xEOp o x y -> wrap $ ppExpr True x <+> ppOp o <+> ppExpr True ywhere wrap| paren = PP.parens|otherwise=idprettyExpr ::Expr t ->PP.Doc annprettyExpr = ppExpr False
And they render the same way:
ghci> prettyExpr fifteen(\x -> x *3) 5ghci> prettyExpr plusThree\x -> x +3ghci> prettyExpr badVariable(\x -> x +3) true
Still Not Fully Verified
Implicit in the previous section was the admission of failure: this system
lets us use indexed types to help propagate unification (the result types of
OLte, OAnd, OPlus, etc.), but it can’t
prevent all ill-defined programs from compiling.
The issue is EVar: its type
EVar :: STy t -> String -> Expr t lets us bind any
variable name as any type, and it’ll still typecheck. Even with the
help of everything we have, we can just straight-up declare a reference to an
unbound variable
That’s because EVar can freely take any STy without
any restriction, and no association with the binder name, so there’s no way for
GHC to stop us.
So, again, we cannot create a fully type-checked Expr.
We still have to deal with most of the same errors. This is noble, but
clearly not good enough. We have to go deeper.
Typed Records and Sums
A quick detour: you might have noticed that this past implementation dropped
records and sums. Before we move on, let’s go ahead and add those. Introducing
records and sums at the same time as type-indexed Expr is a bit
too much of a jump to fit into a single section.
Let’s add sums and records, which can use pretty similar mechanisms (via
duality) for implementation.
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L31-L79typedataTy=TInt|TBool|TString|TRecord [(Symbol, Ty)]|TSum [(Symbol, Ty)]|Ty:->TydataSTy ::Ty->TypewhereSTInt ::STyTIntSTBool ::STyTBoolSTString ::STyTStringSTRecord ::RecSTyField as ->STy (TRecord as)STSum ::RecSTyField as ->STy (TSum as)STFun ::STy a ->STy b ->STy (a :-> b)
Ty now includes TRecord [(Symbol, Ty)] and
TSum [(Symbol, Ty)], which represent the field names and
constructor payloads (Symbol being a type-level string). So, for
example, TRecord ["value" ::: TInt, "label" ::: TString] would be
the type of a record with ordered fields value and
label of integers and strings, respectively.
TSum ["Found" ::: TInt, "Missing" ::: TString] would be the type of
a sum between Found containing an integer and Missing
containing a string. Note we take a page out of vinyl by defining the type
alias (:::) = '(,) to make things syntactically nicer.
Record Access
We need the fields and types at the type level because we have to answer what
the Expr phantom type of field access is. If we had an
x :: Expr (TRecord ["value" ::: TInt, "label" ::: TString]), we
want the type of x.value to be Expr TInt.
To do this, we need to have a value in our Expr for
field access that can “point” at a specific field in the type. One way to do
that is to take a field of type Index:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L47-L49dataIndex :: [k] -> k ->TypewhereIZ ::Index (x : xs) xIS ::Index xs x ->Index (y : xs) x
You can read this as: “IZ is an index to the head of the
type-level list, and IS n is an index to the n-th item of the
tail”. So, IS IZ is an index into the second element,
IS (IS IZ) is an index into the third, etc.
If we have ["value" ::: TInt, "label" ::: TString], then we have
values:
IZ ::Index ["value":::TInt, "label":::TString] ("value":::TInt)ISIZ ::Index ["value":::TInt, "label":::TString] ("label":::TString)
Note that the way this is constructed, it’s impossible for
IS (IS IZ) :: Index ["value" ::: TInt, "label" ::: TString] _ to
typecheck as anything!
In this way, we have a well-typed field accessor syntax, which takes an
Expr of a record of fields and indexes it to get an
Expr of the type at that index:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L105-L105EAccess ::KnownSymbol l =>Expr (TRecord as) ->Index as (l ::: a) ->Expr a
To create an Expr of a record, we can use
Rec from vinyl or
NP from sop-core: a
heterogeneous list indexed by a type-level list.
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L43-L45dataRec :: (k ->Type) -> [k] ->TypewhereRNil ::Rec f '[] (:&) :: f x ->Rec f xs ->Rec f (x : xs)
If you haven’t seen Rec before, basically
Rec f [a,b,c] is a tuple of f a, f b, and
f c. For example:
Keeping Rec f as instead of a direct heterogeneous list of
as lets us store more interesting things than just
Type-kinded things. For example, since our lists here are lists of
(Symbol, Ty), we can create a container to hold fields:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L109-L110dataExprField :: (Symbol, Ty) ->TypewhereEField ::KnownSymbol l =>Expr a ->ExprField (l ::: a)
The field constructor keeps a KnownSymbol l constraint, so the
type-level label is still available later when we need to render it:
So, we can create
Expr (TRecord ["value" ::: TInt, "label" ::: TString]) by taking a
Rec:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L104-L104ERecord ::RecExprField as ->Expr (TRecord as)
We can make this a little more ergonomic by using
-XRequiredTypeArguments (as of GHC 9.10) to get rid of the
-XTypeApplication ugliness:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L115-L150eField ::forall l ->KnownSymbol l =>Expr a ->ExprField (l ::: a)eField l =EField@lmakeRecordExample ::Expr (TRecord ["value":::TInt, "label":::TString])makeRecordExample =ERecord ( eField "value" (EPrim (PInt7)):& eField "label" (EPrim (PString"found")):&RNil )recordExample ::ExprTIntrecordExample =EOpOPlus (EAccess@"value" makeRecordExample IZ) (EPrim (PInt1))
Sum Injection and Case Analysis
We also need a type-level list witness for sum types, because we
need to be able to implement the correct continuations for pattern matches: How
do we know what thing to handle in each pattern match, unless the sum
type has that information in its type?
Luckily due to the magic of duality, we can use the same tools, for the most
part! We can inject into a sum with an Index, let’s say for a
Expr (TSum ["Found" ::: TInt, "Missing" ::: TString]): sum type
with Found containing an integer and Missing
containing a string:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L106-L106EChoice ::KnownSymbol l =>Index as (l ::: a) ->Expr a ->Expr (TSum as)
And we can re-use Rec to define a type that can handle
a ["Found" ::: TInt, "Missing" ::: TString] sum:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L112-L133dataExprHandler ::Ty-> (Symbol, Ty) ->TypewhereEHandler ::KnownSymbol l =>STy a ->String->Expr b ->ExprHandler b (l ::: a)eHandler ::forall l ->KnownSymbol l =>STy a ->String->Expr b ->ExprHandler b (l ::: a)eHandler l =EHandler@l
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L107-L107ECase ::Expr (TSum as) ->Rec (ExprHandler b) as ->Expr b
Note that we’re still using string binders, so there’s still an element of
unsafety here… we say that the variable name is "value" and that it
is a TInt, but when we later refer to the variable with
EVar STInt "value", it isn’t type-checked that later references use
the same type. The compiler would be just as happy with
EVar STString "value".
Here’s an example demonstrating both failure modes: the first handler
references a variable that doesn’t exist, and the second handler references a
variable that does exist as an incorrect type! How unfortunate.
One complication is that we need to update the TestEquality
instance for STy. The record and sum labels are type-level
Symbols, so we compare those with the sameSymbol (kind
of like testEquality for any KnownSymbol instance) and
then compare the payload types recursively.
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L284-L324instanceTestEqualitySTywhere testEquality = sameTyinstanceTestEqualitySTyFieldwhere testEquality = sameFieldsameTy ::STy a ->STy b ->Maybe (a :~: b)sameTy = \caseSTInt-> \caseSTInt->JustRefl; _ ->NothingSTBool-> \caseSTBool->JustRefl; _ ->NothingSTString-> \caseSTString->JustRefl; _ ->NothingSTRecord as -> \caseSTRecord bs ->doRefl<- sameFields as bsJustRefl _ ->NothingSTSum as -> \caseSTSum bs ->doRefl<- sameFields as bsJustRefl _ ->NothingSTFun a b -> \caseSTFun c d ->doRefl<- sameTy a cRefl<- sameTy b dJustRefl _ ->NothingsameFields ::RecSTyField xs ->RecSTyField ys ->Maybe (xs :~: ys)sameFields RNilRNil=JustReflsameFields (x :& xs) (y :& ys) =doRefl<- sameField x yRefl<- sameFields xs ysJustReflsameFields _ _ =NothingsameField ::STyField x ->STyField y ->Maybe (x :~: y)sameField (STyField@l tx) (STyField@m ty) =doRefl<- sameSymbol (Proxy@l) (Proxy@m)Refl<- sameTy tx tyJustRefl
The Full Eval
Before we write the final eval, let’s practice using
Index and Rec together. If we have an index
Index as a that picks out a value a in
as, then we can pick out the f a from a
Rec f as:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L60-L63indexRec ::Index xs x ->Rec f xs -> f xindexRec = \caseIZ-> \(x :& _) -> xIS i -> \(_ :& xs) -> indexRec i xs
We also can recursively iterate a function over each item, assuming the
function forall x. f x -> g x: that is, we can turn a
Rec f as into a Rec g as assuming our function is
polymorphic over each x, and only depends on the shape of
f.
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L65-L67traverseRec ::Applicative m => (forall x. f x -> m (g x)) ->Rec f xs -> m (Rec g xs)traverseRec _ RNil=pureRNiltraverseRec f (x :& xs) = (:&) <$> f x <*> traverseRec f xs
With that, we can write our full eval.
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L326-L369eval ::MapStringSomeValue->Expr t ->Maybe (EValue t)eval env = \caseEPrim (PInt n) ->pure (EVInt n)EPrim (PBool b) ->pure (EVBool b)EPrim (PString s) ->pure (EVString s)EVar t v ->doSomeValue t' v' <- M.lookup v envRefl<- sameTy t t'pure v'ELambda ta n body ->pure$EVFun$ \x -> eval (M.insert n (SomeValue ta x) env) bodyEApply f x ->doEVFun g <- eval env f x' <- eval env x g x'EOp o x y ->case o ofOPlus->doEVInt a <- eval env xEVInt b <- eval env ypure (EVInt (a + b))OTimes->doEVInt a <- eval env xEVInt b <- eval env ypure (EVInt (a * b))OLte->doEVInt a <- eval env xEVInt b <- eval env ypure (EVBool (a <= b))OAnd->doEVBool a <- eval env xEVBool b <- eval env ypure (EVBool (a && b))ERecord xs ->EVRecord<$> traverseRec (evalField env) xsEAccess e i ->doEVRecord xs <- eval env ecase indexRec i xs ofEVField v ->pure vEChoice i x ->EVSum i <$> eval env xECase x hs ->doEVSum i v <- eval env xcase indexRec i hs ofEHandler t n body -> eval (M.insert n (SomeValue t v) env) body
Ergonomics of Records and Sums
Note that we could also choose to implement records and sums using row types
indexed by the name of the field itself, instead of an ordered list of tuples.
This would have the advantage of making, for example,
{ value :: Int, label :: String } the same type as
{ label :: String, value :: Int }. However, I personally prefer the
style of building things inductively (:&/RNil and
IS/IZ), it makes the type errors and constructions a
lot easier to work with and reason with.
However, we can get a little bit of the best of both worlds by using
typeclasses to auto-insert the Index witnesses into a list.
First, we can write a typeclass that searches a type-level list of fields and
produces the right Index:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L51-L58classListIx (l ::Symbol) (xs :: [(Symbol, Ty)]) (a ::Ty) | l xs -> a where listIx ::Index xs (l ::: a)instanceListIx l (l ::: a : xs) a where listIx =IZinstance{-# OVERLAPPABLE #-}ListIx l xs a =>ListIx l (m ::: b : xs) a where listIx =IS (listIx @l)
The FunDep l xs -> a lets us use this like a function: for a
label l and a list xs, we should be able to uniquely
determine the a type it singles out, if it exists. So we have an
instance of
ListIx "value" ["value" ::: TInt, "label" ::: TString] TInt, where
the label "value" and the list uniquely determines the result type
TInt.
We can now have a helper function that we can call like
eAccess "value" using GHC 9.10’s
RequiredTypeArguments:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L118-L123eAccess ::forall (l ::Symbol) -> (KnownSymbol l, ListIx l as a) =>Expr (TRecord as) ->Expr aeAccess l e =EAccess@l e (listIx @l)
That lets us write the field name directly, and the compiler will generate
the Index automatically for us:
Here,
eAccess "value" :: Expr (TRecord ["value" ::: TInt, "label" ::: TString]) -> Expr TInt,
its result type uniquely determined by the types in the record fields.
The same trick works for sum injections, so we can write the constructor name
directly:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L125-L179eChoice ::forall (l ::Symbol) -> (KnownSymbol l, ListIx l as a) =>Expr a ->Expr (TSum as)eChoice l e =EChoice@l (listIx @l) enamedChoiceExample ::Expr (TSum ["Found":::TInt, "Missing":::TString])namedChoiceExample = eChoice "Missing" (EPrim (PString"not here"))
Here,
eChoice "Missing" :: Expr TString -> Expr (TSum ["Found" ::: TInt, "Missing" ::: TString]),
because the constructor name "Missing" uniquely determines the
payload type inside that sum.
Pretty-Printing Records and Sums
To pretty-print, we finally use that KnownSymbol constraint
we’ve been tracking this entire time. We can use
symbolVal :: KnownSymbol s => p s -> String to get the string
value from the type-level string.
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage3.hs#L237-L282ppPrim ::Prim t ->PP.Doc annppPrim = \casePInt n -> PP.pretty nPBool b ->if b then"true"else"false"PString s -> PP.pretty (show s)ppOp ::Op a b c ->PP.Doc annppOp = \caseOPlus->"+"OTimes->"*"OLte->"<="OAnd->"&&"ppFields ::RecExprField xs -> [PP.Doc ann]ppFields RNil= []ppFields (EField@l x :& xs) = (PP.pretty (symbolVal (Proxy@l)) <+>"="<+> ppExpr False x) : ppFields xsppHandlers ::Rec (ExprHandler b) xs -> [PP.Doc ann]ppHandlers RNil= []ppHandlers (EHandler@l _ n body :& xs) = (PP.pretty (symbolVal (Proxy@l)) <+> PP.pretty n <+>"->"<+> ppExpr False body) : ppHandlers xsppExpr ::Bool->Expr t ->PP.Doc annppExpr paren = \caseEPrim p -> ppPrim pEVar _ v -> PP.pretty vELambda _ n body -> wrap $"\\"<> PP.pretty n <+>"->"<+> ppExpr False bodyEApply f x -> wrap $ ppExpr True f <+> ppExpr True xEOp o x y -> wrap $ ppExpr True x <+> ppOp o <+> ppExpr True yERecord xs -> PP.encloseSep "{ "" }"", " (ppFields xs)EAccess@l e _ -> ppExpr True e <>"."<> PP.pretty (symbolVal (Proxy@l))EChoice@l _ x -> wrap $ PP.pretty (symbolVal (Proxy@l)) <+> ppExpr True xECase x hs -> wrap $ PP.sep [ "case"<+> ppExpr False x <+>"of" , PP.encloseSep "{ "" }""; " (ppHandlers hs) ]where wrap| paren = PP.parens|otherwise=idprettyExpr ::Expr t ->PP.Doc annprettyExpr = ppExpr False
Capturing Variables
In order to have EVar be type-safe, the environment itself needs
to be a part of the Expr type, and you should only be able to use
EVar if the Expr enforces it. ELambda
would, therefore, introduce the new variable to the environment.
So a value of type Expr ["x" ::: TInt, "y" ::: TBool] t is an
expression with free variables x of type Int and
y of type Bool.
Surprise! That small detour to add records and sums to our language actually
ended up being a smooth precursor to all of the techniques we will be using to
solve for variable binders.
ELambda would therefore take an Expr with a free
variable and turn it into an Expr of a function type. We also keep
a KnownSymbol constraint for the name so that we can recover the
name when we render the expression.
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage4.hs#L83-L83ELambda ::KnownSymbol n =>Expr (n ::: a ': vs) b ->Expr vs (a :-> b)
EVar then becomes exactly like EAccess! We “index”
into the environment of the Expr vs a using Index:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage4.hs#L82-L82EVar ::Index vs (n ::: t) ->Expr vs t
So it is legal to have
EVar IZ :: Expr ["x" ::: TInt, "y" ::: TBool] TInt, and also it is
automatically inferred to be a TInt. But we could not
write EVar IZ :: Expr [] TInt.
And finally, our whole Expr:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage4.hs#L80-L89dataExpr :: [(Symbol, Ty)] ->Ty->TypewhereEPrim ::Prim t ->Expr vs tEVar ::Index vs (n ::: t) ->Expr vs tELambda ::KnownSymbol n =>Expr (n ::: a ': vs) b ->Expr vs (a :-> b)EApply ::Expr vs (a :-> b) ->Expr vs a ->Expr vs bEOp ::Op a b c ->Expr vs a ->Expr vs b ->Expr vs cERecord ::Rec (ExprField vs) as ->Expr vs (TRecord as)EAccess ::KnownSymbol l =>Expr vs (TRecord as) ->Index as (l ::: a) ->Expr vs aEChoice ::KnownSymbol l =>Index as (l ::: a) ->Expr vs a ->Expr vs (TSum as)ECase ::Expr vs (TSum as) ->Rec (ExprHandler vs b) as ->Expr vs b
Just like with record access, we can use the ListIx class to
write a named helper:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage4.hs#L47-L120classListIx (l ::Symbol) (xs :: [(Symbol, Ty)]) (a ::Ty) | l xs -> a where listIx ::Index xs (l ::: a)instanceListIx l (l ::: a ': xs) a where listIx =IZinstance{-# OVERLAPPABLE #-}ListIx l xs a =>ListIx l (m ::: b ': xs) a where listIx =IS (listIx @l)eVar ::forall n ->ListIx n vs a =>Expr vs aeVar n =EVar (listIx @n)
Adding in our other -XRequiredTypeArguments helpers:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage4.hs#L91-L117eLambda ::forall n ->KnownSymbol n =>Expr (n ::: a ': vs) b ->Expr vs (a :-> b)eLambda n =ELambda@neField ::forall l ->KnownSymbol l =>Expr vs a ->ExprField vs (l ::: a)eField l =EField@leAccess ::forall l -> (KnownSymbol l, ListIx l as a) =>Expr vs (TRecord as) ->Expr vs aeAccess l e =EAccess@l e (listIx @l)eChoice ::forall l -> (KnownSymbol l, ListIx l as a) =>Expr vs a ->Expr vs (TSum as)eChoice l e =EChoice@l (listIx @l) eeHandler ::forall n ->forall l -> (KnownSymbol n, KnownSymbol l) =>Expr (n ::: a ': vs) b ->ExprHandler vs b (l ::: a)eHandler n l =EHandler@n @l
And we get something that is truly type-safe: all expressions are
well-typed, and all variables are ensured to be bound!
Expressions that are not well-typed in our domain language are now rejected
by GHC!1
(As an exercise, can understand why those errors are what they are? Why they
all contain ListIx _ '[]? For more ergonomics there is stuff we can
do with TypeError machinery to make the messages a little prettier;
the vinyl library does a lot to make error messages a bit better, like
in HasFieldinstance)
Note that this is sometimes done using straight De Bruijn indices:
Expr :: [Ty] -> Type, so we don’t use any names but just the
direct index, but the point of this exercise is to be borderline
unbearable to write, and not to be actually unbearable to
write.
Eval with a typed environment
To actually write eval now, we need to have a type-safe environment
to store these variables. In order to eval an Expr vs,
we need EValues for each v in vs. So for
Expr ["x" ::: TInt, "y" ::: TBool], we need to store a
TInt and a TBool. We can once again re-purpose
Rec:
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage4.hs#L130-L131dataEValueField :: (Symbol, Ty) ->TypewhereEVField ::EValue a ->EValueField '(l, a)
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage4.hs#L122-L271dataEValue ::Ty->TypewhereEVInt ::Int->EValueTIntEVBool ::Bool->EValueTBoolEVString ::String->EValueTStringEVRecord ::RecEValueField as ->EValue (TRecord as)EVSum ::Index as (l ::: a) ->EValue a ->EValue (TSum as)EVFun :: (EValue a ->EValue b) ->EValue (a :-> b)eval ::RecEValueField vs ->Expr vs t ->EValue teval env = \caseEPrim (PInt n) ->EVInt nEPrim (PBool b) ->EVBool bEPrim (PString s) ->EVString sEVar i ->case indexRec i env ofEVField v -> vELambda body ->EVFun$ \x -> eval (EVField x :& env) bodyEApply f x ->case eval env f ofEVFun g -> g (eval env x)EOp o x y ->case (o, eval env x, eval env y) of (OPlus, EVInt a, EVInt b) ->EVInt (a + b) (OTimes, EVInt a, EVInt b) ->EVInt (a * b) (OLte, EVInt a, EVInt b) ->EVBool (a <= b) (OAnd, EVBool a, EVBool b) ->EVBool (a && b)ERecord xs ->EVRecord$ mapRec (\(EField x) ->EVField (eval env x)) xsEAccess e i ->case eval env e ofEVRecord xs ->case indexRec i xs ofEVField v -> vEChoice i x ->EVSum i (eval env x)ECase x hs ->case eval env x ofEVSum i y ->case indexRec i hs ofEHandler h -> eval (EVField y :& env) h
At least, we are here. A type-safe EDSL where only AST’s that can be validly
evaluated are legal to represent in Haskell. At least, we can embrace the
freedom of not having to carefully construct your terms. You can relax now. The
compiler and the types have your back.
Even
EVFun :: (EValue a -> EValue b) -> EValue (a :-> b) has a
total EValue a -> EValue b, making the closure evaluation also
fully total.
ghci>case eval RNil plusThree ofEVFun f ->case f (EVInt4) ofEVInt x ->print x7
Pretty-Printing Scoped
Expressions
Now to get to the entire utility of this abstraction: inspecting and
consuming the structure. For our new structure, we took out the string name from
EVar in lieu of an index. This is intentional, so that we keep the
“responsibility” of storing the string name at the ELambda
constructor and not have EVar redundantly store it.
However, this means that for pretty-printing, we will need to track the
variable names in the environment as we descend into lambdas. This is done very
similar to how it was done in eval, but instead of tracking and
indexing out the evaluated EValues, we track and index out the
string names of each variable instead.
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage4.hs#L182-L183dataNameField :: (Symbol, Ty) ->TypewhereNameField ::KnownSymbol l =>NameField (l ::: a)
ghci>:t NameField@"hello":&NameField@"world":&RNilRecNameField ["hello"::: a, "world"::: b]
-- source: https://github.com/mstksg/inCode/tree/master/code-samples/typed-sm-lc/ExprStage4.hs#L185-L245ppPrim ::Prim t ->PP.Doc annppPrim = \casePInt n -> PP.pretty nPBool b ->if b then"true"else"false"PString s -> PP.pretty (show s)ppOp ::Op a b c ->PP.Doc annppOp = \caseOPlus->"+"OTimes->"*"OLte->"<="OAnd->"&&"ppFields ::RecNameField vs ->Rec (ExprField vs) xs -> [PP.Doc ann]ppFields _ RNil= []ppFields names (EField@l x :& xs) = (PP.pretty (symbolVal (Proxy@l)) <+>"="<+> ppExpr names False x) : ppFields names xsppHandlers ::RecNameField vs ->Rec (ExprHandler vs b) xs -> [PP.Doc ann]ppHandlers _ RNil= []ppHandlers names (EHandler@n @l body :& xs) = ( PP.pretty (symbolVal (Proxy@l))<+> PP.pretty (symbolVal (Proxy@n))<+>"->"<+> ppExpr (NameField@n :& names) False body ): ppHandlers names xsppExpr ::RecNameField vs ->Bool->Expr vs t ->PP.Doc annppExpr names paren = \caseEPrim p -> ppPrim pEVar i ->case indexRec i names ofNameField@n -> PP.pretty (symbolVal (Proxy@n))ELambda@n body -> wrap $"\\"<> PP.pretty (symbolVal (Proxy@n))<+>"->"<+> ppExpr (NameField@n :& names) False bodyEApply f x -> wrap $ ppExpr names True f <+> ppExpr names True xEOp o x y -> wrap $ ppExpr names True x <+> ppOp o <+> ppExpr names True yERecord xs -> PP.encloseSep "{ "" }"", " (ppFields names xs)EAccess@l e _ -> ppExpr names True e <>"."<> PP.pretty (symbolVal (Proxy@l))EChoice@l _ x -> wrap $ PP.pretty (symbolVal (Proxy@l)) <+> ppExpr names True xECase x hs -> wrap $ PP.sep [ "case"<+> ppExpr names False x <+>"of" , PP.encloseSep "{ "" }""; " (ppHandlers names hs) ]where wrap| paren = PP.parens|otherwise=idprettyExpr ::Expr '[] t ->PP.Doc annprettyExpr = ppExpr RNilFalse
ghci> prettyExpr fifteen(\x -> x *3) 5
The Next Step
Well, we set out with a simple goal: an expression language that lives within
Haskell that we can inspect and interrogate within the language, but where it
was impossible to construct (in Haskell) a term that did not type-check (in the
domain language).
But, honestly, why does it matter that our expression language has to reject
invalid domain-level terms at the Haskell level? Why couldn’t we go the way of
other expression DSLs in Haskell, where we settle with “untyped” terms being
expressible in Haskell, and validated using a separate typeCheck
function to validate terms at runtime?
I don’t know. But really, maybe this is another way of taking the old Parse,
Don’t Validate adage to the extreme. Why allow yourself to construct invalid
terms of your domain inside Haskell? Why use a “validate” function
(isValid) when you can just make invalid terms impossible to
construct?
But…no. There is no other choice. We CANNOT allow invalid domain terms to be
constructible. If we have the ability to do better, we MUST. With great power
comes great responsibility. And if we compromise here, how can we trust
ourselves not to compromise when it really matters?
One must imagine the stubborn typer happy.
In Part 2, we’ll use our new EDSL to specify visualizable state machines and
programs within Haskell that are type-checked to be correct, and what it looks
like to actually use these within Haskell. And in Part 3, we’ll start
“compiling” them to different language targets and different backends, with the
assurance that our generated programs are all synchronized and
self-consistent.
A Note on AI Coding
I guess I’m going to have to start mentioning this in every post.
But, I really do feel like this “extreme type safety” approach is more
critical than ever, in the age of agentic coding and LLM. I’ve been using LLMs
in my daily coding for many months now at this point, and one common pattern
I’ve noticed: when I start with a design with very clear, very strict types,
LLMs excel. They make much fewer errors, and the type system provides more
immediate feedback on their progress, without needing hundreds of defensive
x != null-style guard pollution.
Once I can express what I want in the language of extreme “invalid states
unrepresentable” types, LLM agents no longer feel like agents of chaotic
spaghetti extruding unmaintainable code. Instead, it feels like…seeding a
crystal and watching it grow into a beautiful, shimmering lattice. It feels like
the language they yearn to speak. And the more expressive your types, the more
beautiful the crystalline structure in the end.
Honestly, humans might have problems writing and using this code, but LLMs
definitely don’t, if properly scaffolded! Since early 2026, at least, for me.
We’ll explore a bit more about this once we have more to work with in Part
2.
Special Thanks
I am very humbled to be supported by an amazing community, who make it
possible for me to devote time to researching and writing these posts. Very
special thanks to my supporter at the “Amazing” level on patreon, Josh Vera! :)
Excluding _|_ in Haskell-land (recursion or
undefined), unfortunately.↩︎
I've spent about 100 hours of work over the past month to make sure
git-annex can build without dependencies that contain LLM generated code.
At least so far.
Needing to review a program's whole dependency tree on an ongoing basis is
apparently what programming has come to?
I've found some real stinkers. Large LLM generated changes being reverted
in the next release without any explanation. An incoherent 1489 line
commit message with 10,000 lines of changes to a 26,000 LOC code base.
A LLM prompt to copy code from another project that seems to have only
avoided being copyright infringement due to luck.
I now have additional information about the quality of dependencies
which will surely influence future decisions. As far as I
can see, that's the only positive benefit of this work.
I realize that I am probably trying to hold back the tide at this point.
That appears to be why Software Freedom Conservancy
punted,
and I doubt that the FSF will do any better.
As these dominos fall, I am reconsidering my participation in these
communities. But I continue my work and support my users.
It may seem easy to prompt a LLM with
Add fourmolu config and restyled
neat
format a module
And commit the result and call yourself a 10xer.
But please consider the broader impact of your actions.
(In the above case, that project lost my further collaboration on it.)
Update June 2026 This blog is about drawing finite regions of the infinite non-periodic tessellations of Roger Penrose’s kite and dart tiles. It was first written before developing a Haskell package (PenroseKiteDart) now available on Hackage. More info and developments can be seen at the end. I have made small updates to this blog to keep it compatible with later developments (and it is no longer a complete literate Haskell file).
Introduction
As part of a collaboration with Stephen Huggett, working on some mathematical properties of Penrose tilings, I recognised the need for quick renderings of tilings. I thought Haskell diagrams would be helpful here, and that turned out to be an excellent choice. Two dimensional vectors were well-suited to describing tiling operations and these are included as part of the diagrams package.
The Haskell below uses the Haskell Diagrams package to draw tilings with kites and darts. It also implements compChoices and decompPatch which are used for constructing tilings (explained below).
Firstly, these 5 lines are needed in Haskell to use the diagrams package:
{-# LANGUAGE NoMonomorphismRestriction #-}{-# LANGUAGE FlexibleContexts #-}{-# LANGUAGE TypeFamilies #-}importDiagrams.PreludeimportDiagrams.Backend.SVG.CmdLine
and we will also import a module for half tiles (explained later)
importHalfTile
Legal tilings
These are the kite and dart tiles.
Figure: Kite and Dart
The red line marking here on the right hand copies, is purely to illustrate rules about how tiles can be put together for legal (non-periodic) tilings. Obviously edges can only be put together when they have the same length. If all the tiles are marked with red lines as illustrated on the right, the vertices where tiles meet must all have a red line or none must have a red line at that vertex. This prevents us from forming a simple rombus by placing a kite top at the base of a dart and thus enabling periodic tilings.
All edges are powers of the golden section which we write as phi.
phi::Doublephi=(1.0+sqrt5.0)/2.0
So if the shorter edges are unit length, then the longer edges have length phi. We also have the interesting property of the golden section that and so
, and .
All angles in the figures are multiples of tt which is 36 deg or 1/10 turn. We use ttangle to express such angles (e.g 180 degrees is ttangle 5).
In order to implement compChoices and decompPatch, we need to work with half tiles. We now define these in the separately imported module HalfTile with constructors for Left Dart, Right Dart, Left Kite, Right Kite
dataHalfTilerep-- defined in HalfTile module=LDrep|RDrep|LKrep|RKrep
where rep is a type variable allowing for different representations. However, here, we want to use a more specific type which we call Piece.
typePiece=HalfTile[V2Double]
Here the half tiles have a simple 2D vector representation to provide orientation and scale. (Update 2026: Originally a single vector was used to represent the join edge of a half tile but now we use a list of two vectors instead).
The list of two two-dimensional vectors represents two edges of the half tile (excluding the join edge where half tiles come together). The origin for a dart is the tip, and the origin for a kite is the acute angle tip (marked in the figure with a red dot).
These are the only 4 pieces we use. They are defined oriented with the join along the x axis. In the figure they are rotated 90 degrees so the join edges are vertical (and they have also been labelled).
Perhaps confusingly, we regard left and right of a dart differently from left and right of a kite when viewed from the origin. The diagram shows the right dart before the left dart and the left kite before the right kite. Thus in a complete tile, going clockwise round the origin the right dart comes before the left dart, but the left kite comes before the right kite.
When it comes to drawing pieces, for the simplest case, we just want to draw the two tile edges of each piece (and not the join edge). These are the drawnEdges of the tile. We can also use the joinVector of the tile (which is simply the vector sum of the drawn edges) The drawnEdges are ordered starting from the origin of each piece.
For an alternative fill operation on whole tiles, we calculated a list of the 4 tile edges of a completed half-tile piece clockwise from the origin of the tile. This allows colour filling a whole tile, but it does rely on prior transformations preserving angles as it uses angles in the definition (not shown).
wholeTileEdges::Piece->[V2Double]
To fill whole tiles with colours, darts with dcol and kites with kcol we can now use leftFillPieceDK. This uses only the left pieces to identify the whole tile and ignores right pieces so that a tile is not filled twice and uses wholeTileEdges.
So we can scale and rotate a piece by an angle (positive rotations are in the anticlockwise direction) but we cannote translate until they are located (as Patches).
Here mapLoc applies a function to the piece in a located piece – producing a located diagram in this case, and viewLoc returns the pair of point and diagram from a located diagram. Finally position forms a single diagram from the list of pairs of points and diagrams. (The use of a class definition here anticipates more instances in later developments).
Patches are automatically inferred to be transformable, so we can also scale a patch, translate a patch by a vector, and rotate a patch by an angle (for example).
This figure shows some example patches, drawn with draw The first is a star and the second is a sun.
Figure: Tile Patches
The tools so far for creating patches may seem limited (and do not help with ensuring legal tilings), but there is an even bigger problem.
Correct Tilings
Unfortunately, correct tilings – that is, tilings which can be extended to infinity – are not as simple as just legal tilings. It is not enough to have a legal tiling, because an apparent (legal) choice of placing one tile can have non-local consequences, causing a conflict with a choice made far away in a patch of tiles, resulting in a patch which cannot be extended. This suggests that constructing correct patches is far from trivial.
The infinite number of possible infinite tilings do have some remarkable properties. Any finite patch from one of them, will occur in all the others (infinitely many times) and within a relatively small radius of any point in an infinite tiling. (For details of this see links at the end).
This is why we need a different approach to constructing larger patches. There are two significant processes used for creating patches, namely inflate (also called compose) and decompose.
To understand these processes, take a look at the following figure.
Figure: Experiment
Here the small pieces have been drawn in an unusual way. The edges have been drawn with dashed lines, but long edges of kites have been emphasised with a solid line and the join edges of darts marked with a red line. From this you may be able to make out a patch of larger scale kites and darts. This is an inflated patch arising from the smaller scale patch. Conversely, the larger kites and darts decompose to the smaller scale ones.
Decomposition
Since the rule for decomposition is uniquely determined, we can express it as a simple function on patches.
where the function decompPiece acts on located pieces and produces a list of the smaller located pieces contained in the piece. For example, a larger right dart will produce both a smaller right dart and a smaller left kite. Decomposing a located piece also takes care of the location, scale and rotation of the new pieces. (Revised version 2026 avoiding use of angles).
This is illustrated in the following figure for the cases of a right dart and a right kite.
Figure: Decomposition and Composition
The symmetric diagrams for left pieces are easy to work out from these, so they are not illustrated.
With the decompPatch operation we can start with a simple correct patch, and decompose repeatedly to get more and more detailed patches. (Each decomposition scales the tiles down by a factor of but we can rescale at any time.)
This figure illustrates how each piece decomposes with 4 decomposition steps below each one.
The earlier figure illustrating larger kites and darts emphasised from the smaller ones is also suns!!6 but this time pieces are drawn with experiment.
experimentFig=drawWithexperiment(suns!!6)#lwthinexperiment::Piece->DiagramBexperimentpc=emphpc<>(drawRoundPiecepc#dashingN[0.002,0.002]0#lwultraThin)whereemphpc=casepcof(LDv)->(strokeLine.fromOffsets)[v]#lcred-- emphasise join edge of darts in red(RDv)->(strokeLine.fromOffsets)[v]#lcred(LKv)->(strokeLine.fromOffsets)[rotate(ttangle1)v]-- emphasise long edge for kites(RKv)->(strokeLine.fromOffsets)[rotate(ttangle9)v]
Compose Choices
You might expect composition (also called inflation) to be a kind of inverse to decomposition, but it is a bit more complicated than that. With our current representation of pieces, we can only compose single pieces. This amounts to embedding the piece into a larger piece that matches how the larger piece decomposes. There is thus a choice at each composition step as to which of several possibilities we select as the larger half-tile. We represent this choice as a list of alternatives. This list should not be confused with a Patch. It only makes sense to select one of the alternatives giving a new single piece.
The earlier diagram illustrating how decompositions are calculated also shows the two choices for embedding a right dart into either a right kite or a larger right dart. There will be two symmetric choices for a left dart, and three choices for left and right kites.
Once again we work with located pieces to ensure the resulting larger piece contains the original in its original position in a decomposition. (Revised version 2026 avoiding use of angles).
As the result is a list of alternatives, we need to select one to do further inflations. We can express all the alternatives after n steps as compNChoices n where
This figure illustrates 5 consecutive choices for composing a left dart to produce a left kite. On the left, the finishing piece is shown with the starting piece embedded, and on the right the 5-fold decomposition of the result is shown.
Finally, at the end of this haskell program we choose which figure to draw as output.
fig::DiagramBfig=leftFilledSun6main=mainWithfig
That’s it. But, What about composing whole patches?, I hear you ask. Unfortunately we need to answer questions like what pieces are adjacent to a piece in a patch and whether there is a corresponding other half for a piece. These cannot be done with our simple vector representations. We would need some form of planar graph representation, which is much more involved. That is another story which can be found in these subsequent blogs.
Graphs, Kites and Darts intoduced Tgraphs. This gave more details of implementation and results of early explorations. (The class Forcible was introduced subsequently).
Empires and SuperForce – these new operations were based on observing properties of boundaries of forced Tgraphs.
There is also a very interesting article by Roger Penrose himself: Penrose R Tilings and quasi-crystals; a non-local growth problem? in Aperiodicity and Order 2, edited by Jarich M, Academic Press, 1989.
More information about the diagrams package can be found from the home page Haskell diagrams
In this episode of the Haskell Interlude, we are joined by Sylvain Henry, one of the all-time top contributors to GHC. He tells us about his work on GHC, the bignum library, modularization, and the secret to becoming a top contributor!
So, evidently I failed to fulfill my ambition to blog regularly about the contents of my planned book on Patterns in Functional Programming. But I have been making progress. I had the privilege of another sabbatical 2024-2025, in which I managed to draft the entire book.
It’s in short chapters, following the example set by Dexter Kozen in his lovely books: the idea is that each chapter is roughly one lecture’s worth of material. I had 40 to 50 chapter ideas, and 40 to 50 weeks in my sabbatical, so diligently stuck to one chapter per week for a year—if this week’s chapter wasn’t finished by the end of the week, it was put aside anyway in order to move on to the next chapter the following week.
That did mean that although I ended the year with a draft of the entire book, it did have many gaps and to-dos remaining. Reality hit at the end of my sabbatical in October 2025, and it has taken me the best part of another year around actual responsibilities to fill in most of the gaps and knock off most of the to-dos. I have also had the benefit of a number of readers (thank you, everyone!), with many helpful comments to implement.
But I have just yesterday sent a complete polished version to the publisher, Cambridge University Press, for the input of a professional copyeditor. I still have to construct an index, write solutions for the many exercises (which may appear separately from the book itself, in order to keep the length and hence the cost down), and make my own proof-reading pass. I am hoping to complete those tasks and implement the copyeditor’s eventual corrections over the summer, and that the book will finally appear before the end of 2026.
(Updated June 2026 for PenroseKiteDart version 1.10)
PenroseKiteDart is a Haskell package with tools to experiment with finite tilings of Penrose’s Kites and Darts. It uses the Haskell Diagrams package for drawing tilings. As well as providing drawing tools, this package introduces tile graphs (Tgraphs) for describing finite tilings. (I would like to thank Stephen Huggett for suggesting planar graphs as a way to reperesent the tilings).
This document summarises the design and use of the PenroseKiteDart package.
PenroseKiteDart package is now available on Hackage.
In figure 1 we show a dart and a kite. All angles are multiples of (a tenth of a full turn). If the shorter edges are of length 1, then the longer edges are of length , where is the golden ratio.
Figure 1: The Dart and Kite Tiles
Aperiodic Infinite Tilings
What is interesting about these tiles is:
It is possible to tile the entire plane with kites and darts in an aperiodic way.
Such a tiling is non-periodic and does not contain arbitrarily large periodic regions or patches.
The possibility of aperiodic tilings with kites and darts was discovered by Sir Roger Penrose in 1974. There are other shapes with this property, including a chiral aperiodic monotile discovered in 2023 by Smith, Myers, Kaplan, Goodman-Strauss. (See the Penrose Tiling Wikipedia page for the history of aperiodic tilings)
This package is entirely concerned with Penrose’s kite and dart tilings also known as P2 tilings.
Legal Tilings
In figure 2 we add a temporary green line marking purely to illustrate a rule for making legal tilings. The purpose of the rule is to exclude the possibility of periodic tilings.
If all tiles are marked as shown, then whenever tiles come together at a point, they must all be marked or must all be unmarked at that meeting point. So, for example, each long edge of a kite can be placed legally on only one of the two long edges of a dart. The kite wing vertex (which is marked) has to go next to the dart tip vertex (which is marked) and cannot go next to the dart wing vertex (which is unmarked) for a legal tiling.
Figure 2: Marked Dart and Kite
Correct Tilings
Unfortunately, having a finite legal tiling is not enough to guarantee you can continue the tiling without getting stuck. Finite legal tilings which can be continued to cover the entire plane are called correct and the others (which are doomed to get stuck) are called incorrect. This means that decomposition and forcing (described later) become important tools for constructing correct finite tilings.
2. Using the PenroseKiteDart Package
You will need the Haskell Diagrams package (See Haskell Diagrams) as well as this package (PenroseKiteDart). When these are installed, you can produce diagrams with a Main.hs module. This should import a chosen backend for diagrams such as the default (SVG) along with Diagrams.Prelude.
Note that the token B is used in the diagrams package to represent the chosen backend for output. So a diagram has type Diagram B. In this case B is bound to SVG by the import of the SVG backend. When the compiled module is executed it will generate an SVG file. (See Haskell Diagrams for more details on producing diagrams and using alternative backends).
3. Overview of Types and Operations
Half-Tiles
In order to implement operations on tilings (decompose in particular), we work with half-tiles. These are illustrated in figure 3 and labelled RD (right dart), LD (left dart), LK (left kite), RK (right kite). The join edges where left and right halves come together are shown with dotted lines, leaving one short edge and one long edge on each half-tile (excluding the join edge). We have shown a red dot at the vertex we regard as the origin of each half-tile (the tip of a half-dart and the base of a half-kite).
The labels are actually data constructors introduced with type operator HalfTile which has an argument type (rep) to allow for more than one representation of the half-tiles.
dataHalfTilerep=LDrep-- Left Dart|RDrep-- Right Dart|LKrep-- Left Kite|RKrep-- Right Kitederiving(Show,Eq)
Tgraphs
We introduce tile graphs (Tgraphs) which provide a simple planar graph representation for finite patches of tiles. For Tgraphs we first specialise HalfTile with a triple of vertices (positive integers) to make a TileFace such as RD(1,2,3), where the vertices go clockwise round the half-tile triangle starting with the origin.
typeTileFace=HalfTile(Vertex,Vertex,Vertex)typeVertex=Int-- must be positive
The function
makeTgraph::[TileFace]->Tgraph
then constructs a Tgraph from a TileFace list after checking the TileFaces satisfy certain properties (described below). We also have
faces::Tgraph->[TileFace]
to retrieve the TileFace list from a Tgraph.
As an example, the fool (short for fool’s kite and also called an ace in the literature) consists of two kites and a dart (= 4 half-kites and 2 half-darts):
fool::Tgraphfool=makeTgraph[RD(1,2,3),LD(1,3,4)-- right and left dart,LK(5,3,2),RK(5,2,7)-- left and right kite,RK(5,4,3),LK(5,6,4)-- right and left kite]
To produce a diagram, we simply draw the Tgraph
foolFigure::DiagramBfoolFigure=drawfool
which will produce the diagram on the left in figure 4.
Alternatively,
foolFigure::DiagramBfoolFigure=labelleddrawjfool
will produce the diagram on the right in figure 4 (showing vertex labels and dashed join edges).
Figure 4: Diagram of fool without labels and join edges (left), and with (right)
When any (non-empty) Tgraph is drawn, a default orientation and scale are chosen based on the lowest numbered join edge. This is aligned on the positive x-axis with length 1 (for darts) or length (for kites).
Tgraph Properties
Tgraphs are actually implemented as
newtypeTgraph=Tgraph[TileFace]deriving(Show)
but the data constructor Tgraph is not exported to avoid accidentally by-passing checks for the required properties. The properties checked by makeTgraph ensure the Tgraph represents a legal tiling as a planar graph with positive vertex numbers, and that the collection of half-tile faces are both connected and have no crossing boundaries (see note below). Finally, there is a check to ensure two or more distinct vertex numbers are not used to represent the same vertex of the graph (a touching vertex check). An error is raised if there is a problem.
Note: If the TileFaces are faces of a planar graph there will also be exterior (untiled) regions, and in graph theory these would also be called faces of the graph. To avoid confusion, we will refer to these only as exterior regions, and unless otherwise stated, face will mean a TileFace. We can then define the boundary of a list of TileFaces as the edges of the exterior regions. There is a crossing boundary if the boundary crosses itself at a vertex. We exclude crossing boundaries from Tgraphs because they prevent us from calculating relative positions of tiles locally and create touching vertex problems.
For convenience, in addition to makeTgraph, we also have
The first of these (performing no checks) is useful when you know the required properties hold. The second performs the same checks as makeTgraph except that it omits the touching vertex check. This could be used, for example, when making a Tgraph from a sub-collection of TileFaces of another Tgraph.
Main Tiling Operations
There are three key operations on finite tilings, namely
Decomposition (also called deflation) works by splitting each half-tile into either 2 or 3 new (smaller scale) half-tiles, to produce a new tiling. The fact that this is possible, is used to establish the existence of infinite aperiodic tilings with kites and darts. Since our Tgraphs have abstracted away from scale, the result of decomposing a Tgraph is just another Tgraph. However if we wish to compare before and after with a drawing, the latter should be scaled by a factor times the scale of the former, to reflect the change in scale.
Figure 5: fool (left) and decompose fool (right)
We can, of course, iterate decompose to produce an infinite list of finer and finer decompositions of a Tgraph
Force works by adding any TileFaces on the boundary edges of a Tgraph which are forced. That is, where there is only one legal choice of TileFace addition consistent with the seven possible vertex types. Such additions are continued until either (i) there are no more forced cases, in which case a final (forced) Tgraph is returned, or (ii) the process finds the tiling is stuck, in which case an error is raised indicating an incorrect tiling. [In the latter case, the argument to force must have been an incorrect tiling, because the forced additions cannot produce an incorrect tiling starting from a correct tiling.]
An example is shown in figure 6. When forced, the Tgraph on the left produces the result on the right. The original is highlighted in red in the result to show what has been added.
Figure 6: A Tgraph (left) and its forced result (right) with the original shown red
Compose
Composition (also called inflation) is an opposite to decompose but this has complications for finite tilings, so it is not simply an inverse. (See Graphs,Kites and Darts and Theorems for more discussion of the problems). Figure 7 shows a Tgraph (left) with the result of composing (right) where we have also shown (in pale green) the faces of the original that are not included in the composition – the remainder faces.
Figure 7: A Tgraph (left) and its (part) composed result (right) with the remainder faces shown pale green
Under some circumstances composing can fail to produce a Tgraph because there are crossing boundaries in the resulting TileFaces. However, we have established that
If g is a forced Tgraph, then compose g is defined and it is also a forced Tgraph.
Try Results
It is convenient to use types of the form Try a for results where we know there can be a failure. For example, compose can fail if the result does not pass the connected and no crossing boundary check, and force can fail if its argument is an incorrect Tgraph. In situations when you would like to continue some computation rather than raise an error when there is a failure, use a try version of a function.
We define Try as a synonym for Either ShowS (which is a monad) in module Tgraph.Try.
type Try a = Either ShowS a
(Note ShowS is String -> String). Successful results have the form Right r (for some correct result r) and failure results have the form Left (s<>) (where s is a String describing the problem as a failure report).
The function
runTry::Trya->arunTry=eithererrorid
will retrieve a correct result but raise an error for failure cases. This means we can always derive an error raising version from a try version of a function by composing with runTry.
force=runTry.tryForcecompose=runTry.tryCompose
Elementary Tgraph and TileFace Operations
The module Tgraph.Prelude defines elementary operations on Tgraphs relating vertices, directed edges, and faces. We describe a few of them here.
When we need to refer to particular vertices of a TileFace we use
originV::TileFace->Vertex-- the first vertex - red dot in figure 2oppV::TileFace->Vertex-- the vertex at the opposite end of the join edge from the originwingV::TileFace->Vertex-- the vertex not on the join edge
A directed edge is represented as a pair of vertices.
typeDedge=(Vertex,Vertex)
So (a,b) is regarded as a directed edge from a to b.
When we need to refer to particular edges of a TileFace we use
joinE::TileFace->Dedge-- shown dotted in figure 2shortE::TileFace->Dedge-- the non-join short edgelongE::TileFace->Dedge-- the non-join long edge
which are all directed clockwise round the TileFace. In contrast, joinOfTile is always directed away from the origin vertex, so is not clockwise for right darts or for left kites:
In the special case that a list of directed edges is symmetrically closed [(b,a) is in the list whenever (a,b) is in the list] we can think of this as an edge list rather than just a directed edge list.
For example,
internalEdges::Tgraph->[Dedge]
produces an edge list, whereas
boundary::Tgraph->[Dedge]
produces single directions. Each directed edge in the resulting boundary will have a TileFace on the left and an exterior region on the right. The function
dedges::Tgraph->[Dedge]
produces all the directed edges obtained by going clockwise round each TileFace so not every edge in the list has an inverse in the list.
Note 1: There is now a class HasFaces (introduced in version 1.4) which includes instances for both Tgraph and [TileFace] and others. This allows some generalisations. For example
Note 2: There is now a class HasGraph (introduced in version 1.8) which includes instances for Tgraph as well as other types used in forcing. This allows some other generalisations. For example
Behind the scenes, when a Tgraph is drawn, each TileFace is converted to a Piece. A Piece is another specialisation of HalfTile. Since version 1.10 this uses two two-dimensional vectors to represent the drawn edges of the half-tile starting from the origin, fixing its scale and orientation. The whole Tgraph then becomes a list of located Pieces called a Patch.
Piece drawing functions can use the drawnEdges (two vectors) and the joinVector (which is just the sum of the drawn edge vectors). In particular (in the TileLib module) we have
where the first draws the drawnEdges (the non-join edges) of a Piece, the second does the same but adds a faint dashed line for the join edge, and the third takes two colours – one for darts and one for kites, which are used to fill the piece as well as using drawPiece.
Patch is an instance of class Transformable so a Patch can be scaled, rotated, and translated for example.
Vertex Patches
It is useful to have an intermediate form between Tgraphs and Patches, that contains information about both the location of vertices (as 2D points), and the abstract TileFaces. This allows us to introduce labelled drawing functions (to show the vertex labels) which we then extend to Tgraphs. We call the intermediate form a VPatch (short for Vertex Patch).
calculates vertex locations using a default orientation and scale.
VPatch is made an instance of class Transformable so a VPatch can also be scaled, translated, and rotated for example.
One essential use of this intermediate form is to be able to draw a Tgraph with labels, transformed but without the labels themselves being transformed. We can simply convert the Tgraph to a VPatch, and transform that before drawing with labels.
labelleddraw(rotatesomeAngle(makeVPg))
We can also align a VPatch using vertex labels.
alignXaxis::(Vertex,Vertex)->VPatch->VPatch
So if g is a Tgraph with vertex labels a and b we can align it on the x-axis with a at the origin and b on the positive x-axis (after converting to a VPatch), instead of accepting the default orientation.
labelleddraw(alignXaxis(a,b)(makeVPg))
Another use of VPatches is to share the vertex location map when drawing only subsets of the faces (see Overlaid examples in the next section).
4. Drawing in More Detail
Class Drawable
There is a class Drawable with instances Tgraph, VPatch, Patch. When the token B is in scope standing for a fixed backend then we can assume
draw::Drawablea=>a->DiagramB-- draws non-join edgesdrawj::Drawablea=>a->DiagramB-- as with draw but also draws dashed join edgesfillDK::Drawablea=>ColourDouble->ColourDouble->a->DiagramB-- fills with colours
where fillDK clr1 clr2 will fill darts with colour clr1 and kites with colour clr2 as well as drawing non-join edges.
These are the main drawing tools. However they are actually defined for any suitable backend b so have more general types.
(Update Sept 2024) From version 1.1 onwards of PenroseKiteDart, these are
Class DrawableLabelled is defined with instances Tgraph and VPatch, but Patch is not an instance (because this does not retain vertex label information).
So labelColourSize c m modifies a Patch drawing function to add labels (of colour c and size measure m). Measure is defined in Diagrams.Prelude with pre-defined measures tiny, verySmall, small, normal, large, veryLarge, huge. For most of our diagrams of Tgraphs, we use red labels and we also find small is a good default size choice, so we define
and then labelled draw, labelled drawj, labelled (fillDK clr1 clr2) can all be used on both Tgraphs and VPatches as well as (for example) labelSize tiny draw, or labelCoulourSize blue normal drawj.
Further drawing functions
There are a few extra drawing functions built on top of the above ones. The function smart is a modifier to add dashed join edges only when they occur on the boundary of a Tgraph
smart::HasGrapha=>(VPatch->DiagramB)->a->DiagramB
So smart vpdraw g will draw dashed join edges on the boundary of g before applying the drawing function vpdraw to the VPatch for g. For example the following all draw dashed join edges only on the boundary for a Tgraph g
Here, smartOn g vpdraw vp uses the given vp for drawing boundary joins and drawing faces of g (with vpdraw) rather than converting g to a new VPatch. This assumes vp has locations for vertices in g.
Overlaid examples (location map sharing)
The function
drawForce::Tgraph->DiagramB
will (smart) draw a Tgraph g in red overlaid (using <>) on the result of force g as in figure 6. Similarly
drawPCompose::Tgraph->DiagramB
applied to a Tgraph g will draw the result of a partial composition of g as in figure 7. That is a drawing of compose g but overlaid with a drawing of the remainder faces of g shown in pale green.
Both these functions make use of sharing a vertex location map to get correct alignments of overlaid diagrams. In the case of drawForce g, we know that a VPatch for force g will contain all the vertex locations for g since force only adds to a Tgraph (when it succeeds). So when constructing the diagram for g we can use the VPatch created for force g instead of starting afresh. Similarly for drawPCompose g the VPatch for g contains locations for all the vertices of compose g so compose g is drawn using the VPatch for g instead of starting afresh.
The location map sharing is done with
subFaces::HasFacesa=>a->VPatch->VPatch
so that subFaces fcs vp is a VPatch with the same vertex locations as vp, but replacing the faces of vp with fcs. [Of course, this can go wrong if the new faces have vertices not in the domain of the vertex location map so this needs to be used with care. Any errors would only be discovered when a diagram is created.]
For cases where labels are only going to be drawn for certain faces, we need a version of subFaces which also gets rid of vertex locations that are not relevant to the faces. For this situation we have
restrictTo::HasFacesa=>a->VPatch->VPatch
which filters out un-needed vertex locations from the vertex location map. Unlike subFaces, restrictTo checks for missing vertex locations, so restrictTo fcs vp raises an error if a vertex in fcs is missing from the keys of the vertex location map of vp.
5. Forcing in More Detail
The force rules
The rules used by our force algorithm are local and derived from the fact that there are seven possible (internal) vertex types as depicted in figure 8.
Figure 8: Seven vertex types
Our rules are shown in figure 9 (omitting mirror symmetric versions). In each case the TileFace shown yellow needs to be added in the presence of the other TileFaces shown.
Figure 9: Rules for forcing
Main Forcing Operations
To make forcing efficient we convert a Tgraph to a BoundaryState to keep track of boundary information of the Tgraph, and then calculate a ForceState which combines the BoundaryState with a record of awaiting boundary edge updates (an update map), and an UpdateGenerator. Then each face addition is carried out on a ForceState, converting back when all the face additions are complete. It makes sense to apply force (and related functions) to a Tgraph, a BoundaryState, or a ForceState, so we define a class Forcible with instances Tgraph, BoundaryState, and ForceState.
The first will raise an error if a stuck tiling is encountered. The second uses a Try result which produces a Left report for failures and a Right a for successful result a.
There are several other operations related to forcing including
The first two force (up to) a given number of steps (=face additions) and the other four add a half dart/kite on a given boundary edge.
Update Generators
An update generator is used to calculate which boundary edges can have a certain update. There is an update generator for each force rule, but also a combined (all update) generator. The force operations mentioned above all use the default all update generator (defaultAllUGen) but there are more general (with) versions that can be passed an update generator of choice. For example
where wholeTileUpdates is an update generator that just finds boundary join edges to complete whole tiles.
In fact UpdateGenerators are functions that take a BoundaryState and a focus (list of boundary directed edges) to produce an update map. Each Update is calculated as either a SafeUpdate (where two of the new face edges are on the existing boundary and no new vertex is needed) or an UnsafeUpdate (where only one edge of the new face is on the boundary and a new vertex needs to be created for a new face).
Completing (executing) an UnsafeUpdate requires a touching vertex check to ensure that the new vertex does not clash with an existing boundary vertex. Using an existing (touching) vertex would create a crossing boundary so such an update has to be blocked.
Forcible Class Operations
The Forcible class operations are higher order and designed to allow for easy additions of further generic operations. They take care of conversions between Tgraphs, BoundaryStates and ForceStates. The first two are designed to create functions that return the same Forcible type as the input.
For example, given any f:: ForceState -> Try ForceState , then f can be generalised to work on any Forcible using tryFSOp f. This is used to define both tryForce and tryStepForce.
Similarly given any f:: BoundaryState -> Try BoundaryChange , then f can be generalised to work on any Forcible using tryChangeBoundary f. This is used to define tryAddHalfDart and tryAddHalfKite.
Note that the type BoundaryChange contains a resulting BoundaryState, the single TileFace that has been added, a list of edges removed from the boundary (of the BoundaryState prior to the face addition), and a list of the (3 or 4) boundary edges affected around the change that require checking or re-checking for updates.
The class function tryInitFS will create an initial ForceState for any Forcible. If the Forcible is already a ForceState it will do nothing. Otherwise it will calculate updates for the whole boundary using defaultAllUGen.
The update generator is assumed to be defaultAllUGen but this can be changed using
Note that (force . force) does the same as force, but we might want to chain other force related steps in a calculation.
For example, consider the following combination which, after decomposing a Tgraph, forces, then adds a half dart on a given boundary edge (d) and then forces again.
Since decompose produces a Tgraph, the instances of force and addHalfDart d will have type Tgraph -> Tgraph so each of these operations, will begin and end with conversions between Tgraph and ForceState. We would do better to avoid these wasted intermediate conversions working only with ForceStates and keeping only those necessary conversions at the beginning and end of the whole sequence.
This can be done using tryFSOp. To see this, let us first re-express the forcing sequence using the Try monad, so
force.addHalfDartd.force
becomes
tryForce<=<tryAddHalfDartd<=<tryForce
Note that (<=<) is the Kliesli arrow which replaces composition for Monads (defined in Control.Monad). (We could also have expressed this right to left sequence with a left to right version tryForce >=> tryAddHalfDart d >=> tryForce). The definition of combo becomes
The sequence actually has type Forcible a => a -> Try a but when passed to tryFSOp it specialises to type ForceState -> Try ForseState. This ensures the sequence works on a ForceState and any conversions are confined to the beginning and end of the sequence, avoiding unnecessary intermediate conversions.
A limitation of forcing
To avoid creating touching vertices (or crossing boundaries) a BoundaryState keeps track of locations of boundary vertices. At around 35,000 face additions in a single force operation the calculated positions of boundary vertices can become too inaccurate to prevent touching vertex problems. In such cases it is better to use
These work by recalculating all vertex positions at 20,000 step intervals to get more accurate boundary vertex positions. For example, 6 decompositions of the kingGraph has 2,906 faces. Applying force to this should result in 53,574 faces but will go wrong before it reaches that. This can be fixed by calculating either
recalibratingForce(decompositionskingGraph!!6)
or using an extra force before the decompositions
force(decompositions(forcekingGraph)!!6)
In the latter case, the final force only needs to add 17,864 faces to the 35,710 produced by decompositions (force kingGraph) !!6.
6. Advanced Operations
Guided comparison of Tgraphs
Asking if two Tgraphs are equivalent (the same apart from choice of vertex numbers) is a an np-hard problem. However, we do have an efficient guided way of comparing Tgraphs. In the module Tgraph.Rellabelling we have
sameGraph::(Tgraph,Dedge)->(Tgraph,Dedge)->Bool
The expression sameGraph (g1,d1) (g2,d2) asks if g2 can be relabelled to match g1 assuming that the directed edge d2 in g2 is identified with d1 in g1. Hence the comparison is guided by the assumption that d2 corresponds to d1.
where tryRelabelToMatch (g1,d1) (g2,d2) will either fail with a Left report if a mismatch is found when relabelling g2 to match g1 or will succeed with Right g3 where g3 is a relabelled version of g2. The successful result g3 will match g1 in a maximal tile-connected collection of faces containing the face with edge d1 and have vertices disjoint from those of g1 elsewhere. The comparison tries to grow a suitable relabelling by comparing faces one at a time starting from the face with edge d1 in g1 and the face with edge d2 in g2. (This relies on the fact that Tgraphs are connected with no crossing boundaries, and hence tile-connected.)
which tries to find the union of two Tgraphs guided by a directed edge identification. However, there is an extra complexity arising from the fact that Tgraphs might overlap in more than one tile-connected region. After calculating one overlapping region, the full union uses some geometry (calculating vertex locations) to detect further overlaps.
which will find common regions of overlapping faces of two Tgraphs guided by a directed edge identification. The resulting common faces will be a sub-collection of faces from the first Tgraph. These are returned as a list as they may not be a connected collection of faces and therefore not necessarily a Tgraph.
Empires and SuperForce
In Empires and SuperForce we discussed forced boundary coverings which were used to implement both a superForce operation
superForce::Forciblea=>a->Forceda
and operations to calculate empires.
We will not repeat the descriptions here other than to note that
forcedBoundaryECovering::Tgraph->[ForcedTgraph]
finds boundary edge coverings after forcing a Tgraph. That is, forcedBoundaryECovering g will first force g, then (if it succeeds) finds a collection of (forced) extensions to force g such that
each extension has the whole boundary of force g as internal edges.
each possible addition to a boundary edge of force g (kite or dart) has been included in the collection.
(possible here means – not leading to a stuck Tgraph when forced.) There is also
forcedBoundaryVCovering::Tgraph->[ForcedTgraph]
which does the same except that the extensions have all boundary vertices internal rather than just the boundary edges. In both cases the result is a list of explicitly forced Tgraphs (discussed next).
Combinations and Explicitly Forced
We introduced a new type Forced (in v 1.3) to enable a forcible to be explictily labelled as being forced. For example
This allows us to restrict certain functions which expect a forced argument by making this explicit.
composeF::HasGrapha=>Forceda->ForcedTgraph
The definition makes use of theorems established in Graphs,Kites and Darts and Theorems that composing a forced Tgraph does not require a check (for connectedness and no crossing boundaries) and the result is also forced. This can then be used to define efficient combinations such as
compForce::(Forciblea,HasGrapha)=>a->ForcedTgraph-- compose after forcingcompForce=composeF.forceFallCompForce::(Forciblea,HasGrapha)=>a->[ForcedTgraph]-- iterated (compose after force) while not emptyTgraphmaxCompForce::(Forciblea,HasGrapha)=>a->ForcedTgraph-- last item in allCompForce (or emptyTgraph)
Note that BoundaryState, ForceState as well as Tgraph and Forced versions of these are all instances of class HasGraph.
has proven useful in experimentation as well as in producing artwork with darts and kites. The idea is to keep a record of sub-collections of faces of a Tgraph when doing both force operations and decompositions. A list of the sub-collections forms the tracked list associated with the Tgraph. We make TrackedTgraph an instance of class Forcible by having force operations only affect the Tgraph and not the tracked list. The significant idea is the implementation of
decomposeTracked::TrackedTgraph->TrackedTgraph
Decomposition of a Tgraph involves introducing a new vertex for each long edge and each kite join. These are then used to construct the decomposed faces. For decomposeTracked we do the same for the Tgraph, but when it comes to the tracked collections, we decompose them re-using the same new vertex numbers calculated for the edges in the Tgraph. This keeps a consistent numbering between the Tgraph and tracked faces, so each item in the tracked list remains a sub-collection of faces in the Tgraph.
is used to draw a TrackedTgraph. It uses a list of functions to draw VPatches. The first drawing function is applied to a VPatch for any untracked faces. Subsequent functions are applied to VPatches for the tracked list in order. Each diagram is beneath later ones in the list, with the diagram for the untracked faces at the bottom. The VPatches used are all restrictions of a single VPatch for the Tgraph, so will be consistent in vertex locations. When labels are used, there is also a drawTrackedTgraphRotating and drawTrackedTgraphAligning for rotating or aligning the VPatch prior to applying the drawing functions.
Note that the result of calculating empires (see Empires and SuperForce ) is represented as a TrackedTgraph. The result is actually the common faces of a forced boundary covering, but a particular element of the covering (the first one) is chosen as the background Tgraph with the common faces as a tracked sub-collection of faces. Hence we have
Diagrams for Penrose Tiles – the first blog introduced drawing Pieces and Patches (without using Tgraphs) and provided a version of decomposing for Patches (decompPatch).
Graphs, Kites and Darts intoduced Tgraphs. This gave more details of implementation and results of early explorations. (The class Forcible was introduced subsequently).
Empires and SuperForce – these new operations were based on observing properties of boundaries of forced Tgraphs.
tl;dr: This is a fully commentated, from-scratch proof of the
Fundamental Theorem of Arithmetic in
Agda,
intended for those who already know a bit of Agda but might benefit
from reading and working through a larger example. See the
Introduction and the Table of Contents below for more details.
So I decided to spend about an hour trying to prove it in Agda,
to gauge the level of the project. At the end of an hour, I had
learned two things: (1) proving the Fundamental Theorem of Arithmetic is not an appropriate project for
my students (who had only had a few weeks’ practice with Agda); (2) I
was not going to be able to stop until I finished the proof myself!
Over the next week or so, I finished the proof completely from
scratch—without using anything from the standard library, and without
looking up any reference material. I based it only on my experience
in Agda, knowledge of the relevant proofs on an informal level, and
Agda techniques I’ve picked up along the way (from e.g. Conor McBride, Jacques
Carette, colleagues at Penn, and elsewhere).
I decided to publish the proof, with extra commentary, in the hopes
that it can be useful as an intermediate-level reference. That is,
perhaps you’ve learned some basic Agda (if not, I suggest this
tutorial to
start)
and have some basic familiarity
with the Curry-Howard correspondence, but would benefit from seeing an
example of a fully worked out, medium-sized proof.Another good
source of information along these lines is this post by Jesper
Cockx.
The resulting blog
post is extremely long, but I make no apologies for that—if you want
an entertaining 5-minute read, you should look elsewhere!
Below is a table of contents. Depending on your background, you may
of course choose to skip some sections. For example, if you have
already had a good deal of practice dealing with basic natural number
arithmetic, equality, and inequality in Agda, you might wish to skip
over those sections.
(Half of) The Fundamental Theorem of Arithmetic (Constructively)
The Fundamental Theorem of
Arithmetic
(FTA for short) states that any natural number \(n \geq 1\) can be
written as a product of zero or more primes, and moreover that this
product is unique up to permutation.
For now, we are only going to prove the existence part (I may write
another blog post with the uniqueness proof later). Since a constructive
existence proof is really an algorithm for constructing the thing that
is claimed to exist, this can also be seen as a formally verified
factorization program: put any number in, get a prime factorization
out. Writing a prime factorization program is not hard, of course;
it’s the formal verification part that is interesting!
Stop! Before reading on, if you want to get the most out of this tutorial, I
strongly recommend downloading the version with
holes
and trying to complete as many of the proofs as you can before reading mine!
Preliminaries
We will often make use of A and B to stand for arbitrary
sets/types, so we use a variable declaration to tell Agda that it
should implicitly quantify them whenever they show up as free
variables. That way we don’t have to write {A B : Set} → ... all the time.
variable A B :Set
Basic logic
Since we’re building this completely from scratch, we start with some
types to represent basic logical building blocks (via the
Curry-Howard correspondence). First, the “top” type ⊤ to stand for truth, i.e. a proposition with trivial evidence:
data ⊤ :Setwhere tt : ⊤
tt is declared to be the one and only value of type ⊤.
Note that some things we define here—such as ⊤—will have the same
names as they do in the Agda standard library. However, many things
won’t, since I either didn’t know the standard name and made up my
own, or (in a few cases) did know the standard name but didn’t like
it, and made up my own anyway.
Next, the “bottom” type ⊥ with no constructors, representing
falsity, along with a corresponding elimination principle, absurd. The elimination
principle says that anything follows from ⊥ (“ex falso
quodlibet”), and is implemented using Agda’s absurd pattern,
written (). If Agda can tell that there are no possible
constructors which could give rise to a value of a certain type, we
can pattern-match on it with (), and are absolved of providing a right-hand side for the definition in that case.
data ⊥ :Setwhereabsurd : ⊥ → Aabsurd ()
We can now define negation as an implication to ⊥.
¬ :Set→Set¬ P = P → ⊥
Dependent pairs are next: a pair of values where the type of the
second component can depend on the value of the first. That is, a value of type Σ A B is a value a of type
A paired with a value of type B a. Via Curry-Howard, this is used
to represent existential quantification: a (constructive) proof of
\(\exists a : A.\; B(a)\) is a value \(a\) of type \(A\) (the witness)
paired with a proof that \(a\) has property \(B\) (i.e. a value of type \(B(a)\)).
infixr1_,_data Σ (A :Set)(B : A →Set):Setwhere_,_:(a : A)→ B a → Σ A B
We also define a projection function (we only end up needing fst;
defining snd is left as an exercise for the readerThe definition
of snd is trivial; writing down its type is a worthwhile
exercise.
), along with a type of non-dependent pairs, corresponding
to logical conjunction (and).
fst :∀{A B}→ Σ A B → Afst (a , _)= ainfixr3_×__×_:(A B :Set)→SetA × B = Σ A (λ_→ B)
Finally, we define a disjoint (tagged) union type corresponding to
logical disjunction (or).
infixr2_⊎_data_⊎_(A B :Set):Setwhere inj₁ : A → A ⊎ B inj₂ : B → A ⊎ B
Equality
Next, we write down the standard equality (aka identity, aka path) type, with a single
constructor refl that witnesses when its two arguments are
identical.It still seems somewhat magical to me that this seemingly
too-simple definition encapsulates everything we want in an equality
relation (well, almost everything).
We also define a convenient
synonym for inequality.
infix4_≡_data_≡_(a : A): A →Setwhere refl : a ≡ a_≢_: A → A →Setx ≢ y = ¬ (x ≡ y)
Besides reflexivity, equality enjoys various properties that we will
need: symmetry, transitivity, and congruence (i.e., we can apply any
function to both sides of an equation).
sym :{x y : A}→ x ≡ y → y ≡ xsym refl = refltrans :{x y z : A}→ x ≡ y → y ≡ z → x ≡ ztrans refl y≡z = y≡zcong :(f : A → B)→{x y : A}→ x ≡ y → f x ≡ f ycong _ refl = refl
Since we will spend a good amount of time reasoning about equality, it
is worthwhile building up some machinery for writing more readable
equality proofs. Instead of writing, say,
trans p (trans q (trans (sym r) s))
we will be able to instead write equality proofs like so:
begin
v ≡[ p ⟩≡
w ≡[ q ⟩≡
x ≡⟨ r ]≡
y ≡[ s ⟩≡
z ∎
The intention is that this proof shows v ≡ z, by first using p to
show that v ≡ w, then q to show w ≡ x, and so on. This notation
is one of my favorite applications of Agda’s mixfix operator
syntax,
and has several benefits:
We can avoid nested parentheses when chaining uses of transitivity.
We can automatically apply symmetry by using a left-pointing
instead of right-pointing operator.
We get to explicitly mention (and have Agda check for us) all the
intermediate values, making it easier to write the proof
incrementally, and much easier for humans to read.
This is one of the places where I deliberately chose different
operator names than the standard library, which uses _≡⟨_⟩_ and
_≡⟨_⟨_. The operator names I decided to use are inspired by Conor
McBride. I just like the way they look better.
infix1 begin_begin_:{x y : A}→ x ≡ y → x ≡ ybegin x≡y = x≡yinfixr2_≡[_⟩≡__≡[_⟩≡_:(x : A)→{y z : A}→(x ≡ y)→(y ≡ z)→(x ≡ z)_ ≡[ x≡y ⟩≡ y≡z = trans x≡y y≡zinfixr2_≡⟨_]≡__≡⟨_]≡_:(x : A)→{y z : A}→(y ≡ x)→(y ≡ z)→(x ≡ z)_ ≡⟨ y≡x ]≡ y≡z = trans (sym y≡x) y≡zinfixr5_∎_∎ :(x : A)→ x ≡ x_ ∎ = refl
Finally, a few Applicative-like operators for more conveniently
writing common forms of congruence. For example, instead of writing
cong f x≡y, we can write f $≡ x≡y; or to use congruence on both
arguments of a two-place function at once, we can write f $≡ x≡y ≡$≡ z≡w. (These operators were also inspired by Conor.)
infixl4_$≡__$≡_:(f : A → B)→{x y : A}→ x ≡ y → f x ≡ f yf $≡ x≡y = cong f x≡yinfixl4_≡$__≡$_:{f g : A → B}→ f ≡ g →(x : A)→ f x ≡ g xf≡g ≡$ x = cong (λ h → h x) f≡ginfixl4_≡$≡__≡$≡_:{f g : A → B}→ f ≡ g →{x y : A}→ x ≡ y → f x ≡ g yf≡g ≡$≡ x≡y = trans (f≡g ≡$ _)(_ $≡ x≡y)
Natural numbers
Of course, we will need a type to represent the natural numbers. We
can also tell Agda that our natural number type should correspond to its
built-in notion of natural numbers, so we can use numeric literals
like 2 : ℕ instead of having to write suc (suc zero).
data ℕ :Setwhere zero : ℕ suc : ℕ → ℕ{-# BUILTIN NATURAL ℕ #-}
No confusion
For our natural number type—and often, for any algebraic data type—we
need to know that the constructors are
disjoint, meaning that different constructors always generate
different values (so it’s a contradiction to have an equality
between values built with different constructors); and
injective, meaning that if we have an equality between values built
with the same constructor, we can decompose it into equalities between
the components.
We can prove both of these simultaneously using a property called “no
confusion”. This property and its name is well-known in the
literature; for example, see
McBridge or Cornes +
Terrasse.
For natural numbers m and n, the type NoConf m n should be
thought of as the type of evidence that m ≡ n, based on looking at
the top-level constructors of m and n. If m and n have different
constructors, then no evidence can possibly show that they are equal,
so NoConf m n = ⊥ in that case. If m and n are both zero, then
they are evidently equal, so NoConf 0 0 = ⊤. Otherwise, if m and
n are both successors, NoConf m n reduces to a proof of equality
between their predecessors.
NoConf : ℕ → ℕ →SetNoConf zero zero = ⊤NoConf zero (suc n)= ⊥NoConf (suc m) zero = ⊥NoConf (suc m)(suc n)= m ≡ n
Now we can prove the no confusion lemma for our natural number type,
which says that NoConf m n always holds whenever m ≡ n.
Since m ≡ n, we only have to deal with the cases when m and n
are both zero or both a successor—but this also justifies assigning a
type of ⊥ to the cases when the constructors do not match. noConf
can therefore be used to strip suc from both sides of an equation,
or to derive a contradiction when we have an equation between
non-matching constructors.
noConf :{m n : ℕ}→ m ≡ n → NoConf m nnoConf {zero} refl = ttnoConf {suc m} refl = refl
As an aside, this definition of the no confusion property uses a
technique I like: defining a type starting with a capital letter,
then defining a term that returns that type starting with a
lowercase letter. This pattern will come up again later. Sometimes
we define named types in this way just for convenience, say, to be
able to refer to the type multiple times in a concise way; or, as in
the above case, sometimes the type is actually defined via some
nontrivial computation.
Decidable equality
We can now show how to decide equality of natural numbers. We first
define a simple type representing decidability in general: Dec P represents
either a proof of P, or a proof of ¬ P.You may be aware that the
law of excluded middle, which
says that \(P \lor \neg P\) for all propositions \(P\), is rejected in constructive logic. However, even
though \(P \lor \neg P\) does not hold for all\(P\), it can still hold
for certain specific propositions. Propositions \(P\) for which \(P \lor
\neg P\) holds constructively are called decidable.
data Dec (P :Set):Setwhere yes : P → Dec P no : ¬ P → Dec P
We can then prove that for any natural numbers x and y, we can decide
whether x ≡ y. Notice the several different uses of the no
confusion lemma: two to handle impossible situations, and one to strip
suc off both sides of an equality.
_≟_:(x y : ℕ)→ Dec (x ≡ y)zero ≟ zero = yes reflzero ≟ suc y = no noConfsuc x ≟ zero = no noConfsuc x ≟ suc y with x ≟ y...| yes x≡y = yes (suc $≡ x≡y)...| no x≢y = no (λ sx≡sy → x≢y (noConf sx≡sy))
Addition
We next turn to defining addition (by pattern-matching on the
left-hand argument), along with several properties of
addition we will need: zero is a right identity for addition; we can
pull out a suc from the right-hand argument; and addition is
commutative, associative, and left-cancellable.
infixl6_+__+_: ℕ → ℕ → ℕzero + y = ysuc x + y = suc (x + y)_+0 :(n : ℕ)→(n + 0 ≡ n)zero +0 = refl(suc n) +0 = suc $≡ (n +0)_+suc_:(x y : ℕ)→(x + suc y) ≡ suc (x + y)zero +suc y = refl(suc x) +suc y = suc $≡ (x +suc y)+-comm :(x y : ℕ)→ x + y ≡ y + x+-comm zero y = sym (y +0)+-comm (suc x) y = trans (suc $≡ (+-comm x y))(sym (y +suc x))+-assoc :(x y z : ℕ)→(x + y) + z ≡ x + (y + z)+-assoc zero y z = refl+-assoc (suc x) y z = suc $≡ (+-assoc x y z)+-cancelˡ :(x y z : ℕ)→ x + y ≡ x + z → y ≡ z+-cancelˡ zero y z x+y≡x+z = x+y≡x+z+-cancelˡ (suc x) y z x+y≡x+z = +-cancelˡ x y z (noConf x+y≡x+z)
Multiplication
Multiplication is next: we start by defining the multiplication
operation (by pattern-matching on the left-hand argument) and proving
a few lemmas about multiplying by known arguments on the right. The
proof of *suc is the most involved proof we have seen yet, but it
ultimately just comes down to algebra, and we can make good use of our
notation for writing chained equality proofs.
infixl7_*__*_: ℕ → ℕ → ℕzero * y = zerosuc x * y = y + x * y_*0 :(n : ℕ)→(n * 0 ≡ 0)zero *0 = refl(suc n) *0 = n *0_*1 :(n : ℕ)→(n * 1 ≡ n)0 *1 = refl(suc n) *1 = suc $≡ (n *1)_*suc_:(x y : ℕ)→(x * suc y ≡ x + x * y)zero *suc y = refl(suc x) *suc y = suc $≡ ( begin y + x * suc y ≡[ (y +_) $≡ (x *suc y) ⟩≡ y + (x + x * y) ≡⟨ +-assoc y x (x * y) ]≡(y + x) + x * y ≡[ _+_ $≡ +-comm y x ≡$ x * y ⟩≡(x + y) + x * y ≡[ +-assoc x __ ⟩≡ x + (y + x * y) ∎)
We prove some standard properties of multiplication: commutativity,
distributivity over addition, associativity. Again, the proofs mostly
consist of a whole bunch of algebra, using the special notation for
building chained equality proofs.
*-comm :(x y : ℕ)→ x * y ≡ y * x*-comm zero y = sym (y *0)*-comm (suc x) y = begin y + x * y ≡[ y +_ $≡ *-comm x y ⟩≡ y + y * x ≡⟨ y *suc x ]≡ y * suc x ∎*-distribˡ :(x y z : ℕ)→ x * (y + z) ≡ x * y + x * z*-distribˡ zero y z = refl*-distribˡ (suc x) y z = begin y + z + x * (y + z) ≡[ (y + z) +_ $≡ *-distribˡ x y z ⟩≡ y + z + (x * y + x * z) ≡[ +-assoc y __ ⟩≡ y + (z + (x * y + x * z)) ≡⟨ y +_ $≡ +-assoc z __ ]≡ y + ((z + x * y) + x * z) ≡[ y +_ $≡ (_+_ $≡ +-comm z _ ≡$ x * z) ⟩≡ y + ((x * y + z) + x * z) ≡[ y +_ $≡ +-assoc (x * y)__ ⟩≡ y + (x * y + (z + x * z)) ≡⟨ +-assoc y __ ]≡ y + x * y + (z + x * z) ∎*-distribʳ :(x y z : ℕ)→(x + y) * z ≡ x * z + y * z*-distribʳ x y z = begin(x + y) * z ≡[ *-comm (x + y)_ ⟩≡ z * (x + y) ≡[ *-distribˡ z __ ⟩≡ z * x + z * y ≡[ _+_ $≡ *-comm z _ ≡$≡ *-comm z _ ⟩≡ x * z + y * z ∎*-assoc :(x y z : ℕ)→(x * y) * z ≡ x * (y * z)*-assoc zero y z = refl*-assoc (suc x) y z = begin(y + x * y) * z ≡[ *-distribʳ y __ ⟩≡ y * z + (x * y) * z ≡[ y * z +_ $≡ *-assoc x __ ⟩≡ y * z + x * (y * z) ∎
Finally, we prove that multiplication is left-cancellative. This
proof is somewhat tricky—in the case that x, y, and z are all
successors, we need to use the induction hypothesis (i.e. a
recursive call to *-cancelˡ) on x and the
predecessors of y and z, using the fact that + is
left-cancellative to construct the required input equality.
*-cancelˡ :(x y z : ℕ)→(0 ≢ x)→ x * y ≡ x * z → y ≡ z*-cancelˡ zero y z x≢0 xy≡xz = absurd (x≢0 refl)*-cancelˡ (suc x) zero zero x≢0 xy≡xz = refl*-cancelˡ (suc x) zero (suc z) x≢0 xy≡xz = absurd (noConf (trans (sym (x *0)) xy≡xz))*-cancelˡ (suc x)(suc y) zero x≢0 xy≡xz = absurd (noConf (trans xy≡xz (x *0)))*-cancelˡ (suc x)(suc y)(suc z) x≢0 xy≡xz = suc $≡( *-cancelˡ (suc x) y z x≢0( +-cancelˡ (suc x)(suc x * y)(suc x * z)( begin suc x + suc x * y ≡⟨ (suc x) *suc y ]≡ suc x * suc y ≡[ xy≡xz ⟩≡ suc x * suc z ≡[ (suc x) *suc z ⟩≡ suc x + suc x * z ∎)))
Inequality
Next, we give a standard definition of the “less than or equal to”
relation on natural numbers. Note that the structure of a proof of \(x \leq
y\) exactly matches the structure of \(x\) itself.
data_≤_: ℕ → ℕ →Setwhere zle :{n : ℕ}→ zero ≤ n sle :{m n : ℕ}→ m ≤ n → suc m ≤ suc n
We also prove some standard properties of \(\leq\): it is reflexive and
transitive, and is related to suc in various ways.
≤-refl :{m : ℕ}→ m ≤ m≤-refl {zero}= zle≤-refl {suc m}= sle ≤-refl≤-trans :{x y z : ℕ}→ x ≤ y → y ≤ z → x ≤ z≤-trans zle y≤z = zle≤-trans (sle x≤y)(sle y≤z)= sle (≤-trans x≤y y≤z)≤-sucr :{m n : ℕ}→ m ≤ n → m ≤ suc n≤-sucr zle = zle≤-sucr (sle m≤n)= sle (≤-sucr m≤n)≤-sucl :{m n : ℕ}→ suc m ≤ n → m ≤ n≤-sucl (sle sm≤n)= ≤-sucr sm≤n≤-pred :{x y : ℕ}→ suc x ≤ suc y → x ≤ y≤-pred (sle sx≤sy)= sx≤sy
For convenience, we define \(<\) in terms of \(\leq\), and prove a few
properties: any number is less than its successor, and \(<\) is
transitive and non-reflexive.
_<_: ℕ → ℕ →Setx < y = suc x ≤ y_<suc :(x : ℕ)→ x < suc x_<suc zero = sle zle_<suc (suc x)= sle (x <suc)<-trans :{x y z : ℕ}→ x < y → y < z → x < z<-trans (sle x<y)(sle y<z)= ≤-trans (sle x<y)(≤-sucr y<z)x≮x :{x : ℕ}→ ¬ (x < x)x≮x {zero}=λ()x≮x {suc x}=λ{(sle x<x)→ x≮x x<x}
Relationships among equality and inequality
Of course, equality, \(<\) and \(\leq\) have various relationships that we
will need. First, equality implies \(\leq\).
≡→≤ :{x y : ℕ}→ x ≡ y → x ≤ y≡→≤ refl = ≤-refl
Next, \(x < y\) implies that \(x\) and \(y\) are not related by \(\equiv\) or
\(\geq\). The first lemma in particular—that \(<\) implies \(\not\equiv\)—gets used quite a bit. Note that it can be read in two equivalent
ways: on the surface, it is a way to turn a proof of \(x < y\) into a
proof of \(x \not\equiv y\); but
since \(x \not\equiv y\) is really an abbreviation for \((x \equiv y) \to \bot\), it can be
used to derive a contradiction if we have proofs that \(x < y\) and
also \(x \equiv y\).
<→≢ :{x y : ℕ}→ x < y → x ≢ y<→≢ x<y refl = x≮x x<y<→≱ :{x y : ℕ}→ x < y → ¬ (y ≤ x)<→≱ (sle x<y)(sle y≤x)= <→≱ x<y y≤x
If \(x \leq y\) but they are not equal, then \(x < y\).
≤≢→< :{x y : ℕ}→ x ≤ y → x ≢ y → x < y≤≢→< {y = zero} zle x≢y = absurd (x≢y refl)≤≢→< {y = suc y} zle x≢y = sle zle≤≢→< (sle x≤y) x≢y = sle (≤≢→< x≤y (λ m≡n → x≢y (suc $≡ m≡n)))
We will need a form of transitivity that says if \(x \leq y\) and \(y <
z\), then \(x < z\), as well as the other way around.
≤-<-trans :{x y z : ℕ}→ x ≤ y → y < z → x < z≤-<-trans x≤y (sle y<z)= ≤-trans (sle x≤y)(sle y<z)<-≤-trans :{x y z : ℕ}→ x < y → y ≤ z → x < z<-≤-trans (sle x<y) y≤z = ≤-trans (sle x<y) y≤z
Finally, a very specific lemma we will need: if a number is not
equal to either 0 or 1, then it must be greater than or equal to 2.
The last lemmas we need relate arithmetic operations and inequality.
First, adding and multiplying cannot make anything smaller (unless we
multiply by zero, of course).
≤+ :{x y : ℕ}→ x ≤ (x + y)≤+ {zero}= zle≤+ {suc x}= sle ≤+≤* :{x y : ℕ}→(x ≢ 0)→ y ≤ (x * y)≤* {zero} x≢0 = absurd (x≢0 refl)≤* {suc x} x≢0 = ≤+
As a result, if we know that one thing is equal to a sum or product of
other things, we can conclude something about their relative sizes.
+→≤ :{x y z : ℕ}→ x + y ≡ z → x ≤ z+→≤ refl = ≤++→< :{x y z : ℕ}→0 < y → x + y ≡ z → x < z+→< {x}{suc y}_ x+y≡z = +→≤ (trans (sym (x +suc y)) x+y≡z)*→≤ :{x y z : ℕ}→(y ≢ 0)→ y * x ≡ z → x ≤ z*→≤ {x}{y} y≢0 refl = ≤* y≢0
Divisibility, primes, and composites
With the preliminaries out of the way, we can finally get on with the
meat of the problem—and we finally get to make use of a dependent pair! A constructive proof that a divides
b is a specific natural number witness k, along with a proof that k * a ≡ b.
_∣_: ℕ → ℕ →Seta ∣ b = Σ ℕ (λ k → k * a ≡ b)
Proofs of divisibility are unique—that is, for given \(a\) and \(b\) there
is at most one value of \(k\) such that \(ka = b\). We won’t need this,
but it follows easily from the fact that multiplication is
cancellative. More interesting is the fact that divisibility is
decidable—that is, for given numbers \(a\) and \(b\) we can calculate
either a proof that \(a \mid b\), or a proof that \(\neg (a \mid b)\).
This will play a starring role later on—to factor a number we need to
be able to try potential divisors and find out whether they work—but proving it is not easy!
It will take us several hundred more lines of Agda to get there.
In any case, using this notion of divisibility, we can now define prime and
composite numbers. A number \(n\) is defined to be prime if it is at
least two, and every \(2 \leq d < n\) does not divide \(n\).
Prime : ℕ →SetPrime n =(2 ≤ n) × (∀(d : ℕ)→(d < n)→(2 ≤ d)→ ¬ (d ∣ n))
One could equivalently define primality by saying that any divisor of
\(n\) must be equal to \(1\) or \(n\); I just decided I liked this
formulation better, especially because it directly matches up with the
way we will test a number for primality later.
A composite number is one that has a nontrivial divisor—that is, a number \(d\)
such that \(2 \leq d < n\) and \(d\) divides \(n\).Note
that we could easily prove that if \(n\) is prime then \(n\) is not
composite, and likewise if \(n\) is composite then it is not prime, but
we won’t end up needing these lemmas.
Composite : ℕ →SetComposite n = Σ ℕ (λ d →2 ≤ d × d < n × d ∣ n)
Unlike proofs of divisibility, proofs of Composite n are not
unique. For example, we could prove Composite 12 by showing that
\(2\) is a nontrivial divisor of \(12\), or by showing that \(3\) is.
Although this does not matter from a purely logical point of view, it
matters computationally; in general, we care which specific proof of
Composite n we have.
Nontrivial divisors come in pairs
Before moving on to other things, we will prove a lemma about
composite numbers. If \(n\) is composite, by definition it has a
nontrivial divisor \(a\); but this means it must also have a second
nontrivial divisor \(b\) such that \(ab = n\). This fact seems almost
trivial to us. Indeed, it’s easy to show that if \(n\) has a divisor
\(a\), then it must have another divisor \(b\) such that \(ab = n\). The
tricky part is showing that if \(a\) is a nontrivial divisor, then \(b\)
is also nontrivial. The proof relies on much of the infrastructure
we have built up about natural numbers, multiplication, and
inequality.
First, we define a type representing two factors of a number \(n\): a
pair of proofs that \(n\) is composite (i.e. two nontrivial divisors
of \(n\)), along with a proof that the product of those divisors is \(n\).
Now, we prove that if \(n\) is composite, then it has two nontrivial
factors. We begin by pattern-matching on the proof that \(n\) is
composite, which consists of a divisor \(a\), evidence that \(a\) is
nontrivial (i.e.\(2 \leq a\) and \(a < n\)), and a proof that \(a\) is a
divisor of \(n\), which itself consists of a number \(b\) paired with a
proof that \(ba = n\).
factorsOf :(n : ℕ)→ Composite n → FactorsOf nfactorsOf n (a , 2≤a , a<n , b , ba≡n)=
To construct the proof of FactorsOf n, we need two proofs of
Composite n along with a proof that the product of the two divisors
is \(n\). We already have a proof that \(ba = n\), so we use that, with
\(a\) as the second divisor (replicating the corresponding proof of
Composite n), and \(b\) as the first. Proving that \(b\) is a divisor of
\(n\) is easy: \(a\) is the witness, and proving that \(ab = n\) is easy
since we already know \(ba = n\) and multiplication is commutative. The
only thing left is to prove that \(b\) is nontrivial, i.e. that \(2
\leq b\) and \(b < n\).
((b , 2≤b , b<n , a , trans (*-comm a b) ba≡n) , (a , 2≤a , a<n , b , ba≡n)) , ba≡n
First, we need a lemma that \(0 < n\), which follows because \(0 < 1 < a
< n\) (remember that a proof of \(1 < a\) is actually defined to be the
same thing as a proof of \(2 \leq a\)).
where 0<n :0 < n 0<n = <-trans (sle zle)(<-trans 2≤a a<n)
Next, we tackle \(2 \leq b\), by showing that \(b\) can’t possibly be \(0\)
or \(1\) (using our previous lemma that anything not equal to 0 or 1
must be greater than or equal to 2).
2≤b :2 ≤ b 2≤b = ¬01-is-≥2 b
If \(b\) were \(0\), then \(ba = n\) would imply \(0 = n\), but we know
\(0 < n\), so this is a contradiction.
(λ b≡0 → <→≢ 0<n(begin0 ≡[ refl ⟩≡0 * a ≡⟨ _*_ $≡ b≡0 ≡$ a ]≡ b * a ≡[ ba≡n ⟩≡ n ∎))
If \(b\) were \(1\), then \(ba = n\) would imply \(a = n\), but we know
\(a < n\), so this is also a contradiction.
(λ b≡1 → <→≢ a<n(begin a ≡⟨ a *1 ]≡ a * 1 ≡[ *-comm a 1 ⟩≡1 * a ≡⟨ _*_ $≡ b≡1 ≡$ a ]≡ b * a ≡[ ba≡n ⟩≡ n ∎))
Finally, we prove \(b < n\), by showing \(b \leq n\) and \(b \neq n\).
b<n : b < n b<n = ≤≢→<
\(b \leq n\) since \(ba = n\) and \(a\) is not zero (if \(a\) were zero it
would contradict the fact that \(2 \leq a\)).
(*→≤ (λ a≡0 → <→≢ (<-trans (sle zle) 2≤a)(sym a≡0))(trans (*-comm a b) ba≡n))
\(b \neq n\), since \(b = n\) together with \(ba = n\) would imply \(a = 1\)
(since multiplication is cancellative), but \(2 \leq a\) so it cannot
equal 1.
(λ b≡n → <→≢ 2≤a(sym(*-cancelˡ n a 1(<→≢ 0<n)(begin n * a ≡⟨ _*_ $≡ b≡n ≡$ a ]≡ b * a ≡[ ba≡n ⟩≡ n ≡⟨ n *1 ]≡ n * 1 ∎))))
Division
Let’s start working our way towards proving that divisibility is
decidable. To check whether \(d \mid n\), the usual idea would be to
divide \(n\) by \(d\) and check whether we get a remainder of zero. So we
need to formalize this notion of division with remainder.
Specifically, when we divide \(n\) by \(d\), we expect to get a quotient\(q\)
and a remainder\(r\), such that \(r + qd = n\), and \(0 \leq r < d\).
The first condition, \(r + qd = n\), just defines what we mean by division: \(n\) is
\(q\) times \(d\), plus a remainder of \(r\). The second condition will ensure
that the result is unique. We wouldn’t want to divide \(17\) by \(2\) and
end up with a quotient of \(6\) and a remainder of \(5\); the remainder should be as small as possible.
The DivMod type simply encodes these requirements.
data DivMod (n d q r : ℕ):Setwhere DM :(r + q * d ≡ n)→(r < d)→ DivMod n d q r
We can prove a few lemmas about DivMod. First, whenever we have
DivMod n d q r, then d must be positive, since \(r < d\) and \(r\) is
a natural number.
divMod→0<d :{n d q r : ℕ}→ DivMod n d q r →0 < ddivMod→0<d (DM _ r<d)= ≤-<-trans zle r<d
We can also show that for nonzero \(d\), the remainder is zero if and only if \(d
\mid n\):
mod0→divides :(n d : ℕ){q : ℕ}→ DivMod n d q 0→ d ∣ nmod0→divides n d {q}(DM eq _)= q , eqdivides→mod0 :(n d : ℕ)→(0 < d)→ d ∣ n → Σ ℕ (λ q → DivMod n d q 0)divides→mod0 n d 0<d (q , qd≡n)= q , (DM qd≡n 0<d)
We would also like to show that if the remainder when dividing \(n\) by
\(d\) is not zero, then \(d\) does not divide \(n\). This is almost
the contrapositive of divides→mod0—which would be trivial to
show—but not quite: I said “the” remainder, but actually we don’t yet
know that the quotient and remainder are unique! Perhaps we could get
a remainder of 0 and some other remainder for the same \(n\) and \(d\), by
choosing different quotients?
Of course, quotients and remainders are unique: that is, if \(q_1,
r_1\) and \(q_2, r_2\) both satisfy the properties to be the quotient and
remainder of \(n\) divided by \(d\), then in fact \(q_1 = q_2\) and \(r_1 =
r_2\). But how can we prove this? The usual idea is to look at the
difference \(r_1 - r_2 = dq_1 - dq_2\), which is divisible by \(d\); but
since \(r_1 < d\) and \(r_2 < d\), the only way for the difference \(r_1 -
r_2\) to be divisible by \(d\) is if in fact \(r_1 - r_2 = 0\). From here
we can also derive \(q_1 = q_2\) via algebra.
Subtraction, eh? In order to formalize this, it seems as though we might
need to define the integers… but there is a better way!
Absolute difference
The previous informal argument mentioned the difference \(r_1 - r_2\).
But we could just as easily have talked about \(r_2 - r_1\) instead, and
the same argument would work just as well. This observation shows
that we do not actually care about the (signed) difference between
\(r_1\) and \(r_2\), but only the distance between them. This means we
can just stick to our well-loved natural numbers, and define a commutative
absolute difference function which computes the nonnegative distance
between its two arguments, like so:
∥_-_∥ : ℕ → ℕ → ℕ∥ zero - y ∥ = y∥ suc x - zero ∥ = suc x∥ suc x - suc y ∥ = ∥ x - y ∥
Of course, we will need a lot of small lemmas about the properties of
this operation. We can start by proving that the distance between two
numbers is 0 if and only if they are equal:
diff0 :(x : ℕ)→0 ≡ ∥ x - x ∥diff0 zero = refldiff0 (suc x)= diff0 xdiff0→≡ :{x y : ℕ}→0 ≡ ∥ x - y ∥ → x ≡ ydiff0→≡ {zero}{zero} eq = eqdiff0→≡ {suc x}{suc y} eq = suc $≡ diff0→≡ eq
Next, the distance between any number and 0 is the number itself, and
the distance function is commutative.
∥x-0∥≡x :(x : ℕ)→ ∥ x - 0 ∥ ≡ x∥x-0∥≡x zero = refl∥x-0∥≡x (suc x)= refldiff-comm :{x y : ℕ}→ ∥ x - y ∥ ≡ ∥ y - x ∥diff-comm {zero}{zero}= refldiff-comm {zero}{suc y}= refldiff-comm {suc x}{zero}= refldiff-comm {suc x}{suc y}= diff-comm {x}{y}
A key lemma supporting the argument outlined in the previous section
is that if \(x\) and \(y\) are both less than \(d\), so is their absolute difference.
diff-< :{x y d : ℕ}→ x < d → y < d → ∥ x - y ∥ < ddiff-< {zero}{y} x<d y<d = y<ddiff-< {suc x}{zero} x<d y<d = x<ddiff-< {suc x}{suc y} x<d y<d = diff-< {x}{y}(<-trans (x <suc) x<d)(<-trans (y <suc) y<d)
We can also cancel the same thing being added to both sides, or factor out
the same thing being multiplied on both sides.
diff-cancelˡ :(a b c : ℕ)→ ∥ (a + b) - (a + c) ∥ ≡ ∥ b - c ∥diff-cancelˡ zero b c = refldiff-cancelˡ (suc a) b c = diff-cancelˡ a b cdiff-distribʳ :(x y d : ℕ)→ ∥ x * d - y * d ∥ ≡ ∥ x - y ∥ * ddiff-distribʳ zero y d = refldiff-distribʳ (suc x) zero d = ∥x-0∥≡x (d + x * d)diff-distribʳ (suc x)(suc y) d = begin ∥ (d + x * d) - (d + y * d) ∥ ≡[ diff-cancelˡ d (x * d)(y * d) ⟩≡ ∥ x * d - y * d ∥ ≡[ diff-distribʳ x y d ⟩≡ ∥ x - y ∥ * d ∎
Another key lemma is that if \(w + x = y + z\), then \(\|w - y\| = \|x -
z\|\) (sub₂ below). Personally, I found this quite tricky to prove.
The best approach I found was to first prove the simpler lemma that
\(x + y = z\) implies \(x = \| z - y \|\) (sub₁), which can then be used
in several places in the proof of sub₂.
sub₁ :{x y z : ℕ}→ x + y ≡ z → x ≡ ∥ z - y ∥sub₁ {zero}{y}{z} refl = diff0 ysub₁ {suc x}{zero}{suc z} x+y≡z = begin suc x ≡⟨ suc $≡ x +0 ]≡ suc (x + 0) ≡[ x+y≡z ⟩≡ suc z ∎sub₁ {suc x}{suc y}{suc z} x+y≡z = sub₁ {suc x}{y}{z}(noConf (trans (suc $≡ sym (x +suc y)) x+y≡z))sub₂ :{w x y z : ℕ}→ w + x ≡ y + z → ∥ w - y ∥ ≡ ∥ x - z ∥sub₂ {zero}{x}{y}{z} w+x≡y+z = sub₁ (sym w+x≡y+z)sub₂ {suc w}{x}{zero}{z} w+x≡y+z = trans (sub₁ w+x≡y+z)(diff-comm {z})sub₂ {suc w}{x}{suc y}{z} w+x≡y+z = sub₂ {w}(noConf w+x≡y+z)
Quotient and remainder are unique
We can now return to prove that quotient and remainder are unique.
First, we show that zero is the only multiple of \(d\) which is less than \(d\).
∣<→0 :{d x : ℕ}→ d ∣ x → x < d →0 ≡ x∣<→0 (zero , ad≡x) x<d = ad≡x∣<→0 (suc a , ad≡x) x<d = absurd (<→≱ x<d (+→≤ ad≡x))
And now for the main event: if we have both DivMod n d q₁ r₁ and
DivMod n d q₂ r₂, then in fact the qs and rs must be the same.
divModUnique :{n d q₁ r₁ q₂ r₂ : ℕ}→ DivMod n d q₁ r₁ → DivMod n d q₂ r₂ →(q₁ ≡ q₂) × (r₁ ≡ r₂)divModUnique {n}{d}{q₁}{r₁}{q₂}{r₂} dm@(DM r₁+q₁d≡n r₁<d)(DM r₂+q₂d≡n r₂<d)= q₁≡q₂ , r₁≡r₂where
Since \(r_1 + q_1d = n\) and \(r_2 + q_2d = n\), by transitivity and
symmetry we have \(r_1 + q_1d = r_2 + q_2d\); then by the sub₂ lemma,
\(\|r_1 - r_2\| = \|q_1d - q_2d\|\).
Next, we can show that \(d\) divides the absolute difference \(\|r_1 -
r_2\|\), by factoring it out of \(\|q_1 d - q_2 d\|\).
d∣r₁-r₂ : d ∣ ∥ r₁ - r₂ ∥ d∣r₁-r₂ = ∥ q₁ - q₂ ∥ ,(begin ∥ q₁ - q₂ ∥ * d ≡⟨ diff-distribʳ q₁ q₂ d ]≡ ∥ q₁ * d - q₂ * d ∥ ≡⟨ rem-diff ]≡ ∥ r₁ - r₂ ∥ ∎)
We can then put three lemmas together to conclude \(r_1 = r_2\): first,
since \(r_1\) and \(r_2\) are both less than \(d\), so is their absolute
difference; since \(d\) also divides the absolute difference, the
absolute difference must be zero; and finally, an absolute difference
of zero means \(r_1\) and \(r_2\) must be equal.
From here, proving \(q_1 = q_2\) just requires some algebra.
dq₁≡dq₂ : d * q₁ ≡ d * q₂ dq₁≡dq₂ = +-cancelˡ r₁ (d * q₁)(d * q₂)(begin r₁ + d * q₁ ≡[ r₁ +_ $≡ *-comm d q₁ ⟩≡ r₁ + q₁ * d ≡[ r₁+q₁d≡n ⟩≡ n ≡⟨ r₂+q₂d≡n ]≡ r₂ + q₂ * d ≡[ _+_ $≡ sym r₁≡r₂ ≡$≡ *-comm q₂ d ⟩≡ r₁ + d * q₂ ∎) q₁≡q₂ : q₁ ≡ q₂ q₁≡q₂ = *-cancelˡ d q₁ q₂ (<→≢ (divMod→0<d dm)) dq₁≡dq₂
Finally, we can use uniqueness of quotients and remainders to show the
lemma we wanted about divisibility and remainders: if \(n\) divided by
\(d\) has some nonzero number as remainder, then \(d\) does not divide
\(n\). If \(d\) did divide \(n\), then we know we would get a remainder of
\(0\); but since remainders are unique, we can’t have both a zero and
nonzero remainder.
modS→¬divides :(n d : ℕ){q r : ℕ}→ DivMod n d q (suc r)→ ¬ (d ∣ n)modS→¬divides n d dm d∣n with divides→mod0 n d (divMod→0<d dm) d∣n...| q₂ , dm₂ with divModUnique dm dm₂...| q₁≡q₂ , ()
The division algorithm, take 1
So, given natural numbers \(n\) and \(d\), how do we compute the quotient
and remainder? We can write down a type DivAlg that expresses what
we want: given some \(n\) and \(d\), DivAlg n d represents the result of
the division algorithm, that is, a pair of numbers \((q,r)\) such that
DivMod n d q r holds.
DivAlg : ℕ → ℕ →SetDivAlg n d = Σ (ℕ × ℕ)(λ{(q , r)→ DivMod n d q r })
Then we want a function with a type something like (n d : ℕ) → DivAlg n d (actually this type is not quite correct—can you see why?).
How can we write something with this type?
One simple idea, expressed imperatively, is to start with \(q = 0\).
Now, as long as \(n \geq d\), subtract \(d\) from \(n\) and add one to
\(q\)—if we can find \(q\) and \(r\) such that \(r + qd = n - d\), then \(r +
(q+1)d = n\). Eventually, \(n\) must land in the range \(0 \leq n < d\),
in which case it will be the remainder, and the current value of \(q\)
will be the quotient.
Construct the evidence you would like to pattern-match on
That’s the idea, but turning this into a verified constructive algorithm will take some
work. First, let’s formalize the idea of testing whether \(n\) is less
than \(d\), and decreasing it by \(d\) if not. We’d rather not actually
deal with subtraction, so the idea is to generate either a proof that
\(n < d\), or another number \(n'\) along with a proof that \(n' + d = n\).
We encapsulate this in the following type Cmp:
data Cmp (n d : ℕ):Setwhere LT : n < d → Cmp n d GE :(n′ : ℕ)→(n′ + d ≡ n)→ Cmp n d
Cmp n d represents the result of comparing \(n\) and \(d\), and is
equivalent to having either \(n < d\) or \(n \geq d\), but expressed in a
form that is more directly useful to us. Construct the evidence you
would like to pattern-match on! That is, in general, evidence for a
proposition \(P\) can take many logically equivalent forms, and you
should pick the form that will make your life easiest at the use
site, even if it means you have to work harder to construct it in
the first place. You can write standalone lemmas for constructing
your evidence; but pattern-matching it will happen in the middle of
some bigger proof which ought not to be cluttered by calls to
conversion lemmas.
To construct evidence for Cmp n d, we write the function
decreaseBy? which decides whether we can decrease n by d or not.
Writing this function is a bit more work than writing something
of type (n d : ℕ) → (n < d) ⊎ (d ≤ n), but our work will pay off later!
_decreaseBy?_:(n d : ℕ)→ Cmp n dzero decreaseBy? zero = GE 0 reflzero decreaseBy? suc d = LT (sle zle)suc n decreaseBy? zero = GE (suc n)((suc n) +0)suc n decreaseBy? suc d with n decreaseBy? d...| LT n<d = LT (sle n<d)...| GE n′ n′+d≡n = GE n′ (trans (n′ +suc d)(suc $≡ n′+d≡n))
We can also write a helper function incDivMod which encodes the
observation from before, that if \(r + qd = n-d\), then \(r + (q+1)d =
n\). Of course we don’t actually want to use subtraction, so instead
of writing \(n-d\), we work in terms of an \(n'\) such that \(n' + d = n\).
Proving this requires only some straightforward algebra.
incDivMod :{n′ n d q r : ℕ}→ n′ + d ≡ n → DivMod n′ d q r → DivMod n d (suc q) rincDivMod {n′}{n}{d}{q}{r} n′+d≡n (DM r+qd≡n′ r<d)= DM r+d+qd≡n r<dwhere r+d+qd≡n : r + (d + q * d) ≡ n r+d+qd≡n = begin r + (d + q * d) ≡⟨ +-assoc r __ ]≡(r + d) + q * d ≡[ _+_ $≡ +-comm r _ ≡$ q * d ⟩≡(d + r) + q * d ≡[ +-assoc d __ ⟩≡ d + (r + q * d) ≡[ d +_ $≡ r+qd≡n′ ⟩≡ d + n′ ≡[ +-comm d _ ⟩≡ n′ + d ≡[ n′+d≡n ⟩≡ n ∎
Now it seems like we have everything we need to write the division
algorithm as a recursive algorithm: given \(n\) and \(d\), check whether
\(n\) can be decreased by \(d\) or not. If not, we can return \(q = 0\) and
\(r = n\). Otherwise, recurse on \(n - d\), returning the same remainder
and an incremented quotient from whatever the recursive call returns, using
incDivMod to discharge the proof obligation. It’s just a few lines
of code, right?
module DivModBad where{-# NON_TERMINATING #-} divAlg :(n d : ℕ)→ DivAlg n d divAlg n d with n decreaseBy? d...| LT n<d =(0 , n) , (DM (n +0) n<d)...| GE n′ n′+d≡n with divAlg n′ d...|(q , r) , dm =(suc q , r) , (incDivMod n′+d≡n dm)
Well, as you can see, it is just a few lines of code, but all is not
well: although this function typechecks, Agda can’t tell that it is
terminating! (I added the NON_TERMINATING pragma so I could include
this bad version of divAlg in the code without causing an error.)
The problem is that the recursive call to divAlg is on n′,
which is not a subterm of n, but instead comes from the call to
decreaseBy?. Agda has no way of knowing whether the result from
some random function call is going to end up being smaller than the
original input.
Now, you and I can both see that this function does indeed terminate,
but we just need a way to convince Agda of this fact… right?
…have you spotted the flaw? Remember how I mentioned that the type (n d : ℕ) → DivAlg n d is not quite right? In fact, the above bad
implementation of divMod is not terminating, and Agda is quite right to
be worried! In particular, the function recurses infinitely when given an input of \(d
= 0\), since it will keep subtracting \(0\) from \(n\) forever. This makes sense, of
course: everyone knows you can’t divide by zero because it makes the
universe go into infinite recursion.
The correct type for divAlg is (n d : ℕ) → (0 < d) → DivAlg n d,
but we’re still going to have trouble convincing Agda that our
algorithm is terminating. In order to do so, we need to take a detour
through well-founded induction.
Well-founded induction
Normally, Agda only allows functions that are structurally
recursive—that is, functions which make recursive calls on syntactic
subterms of their inputs. (Agda’s termination checking is a bit more
sophisticated than that, but that’s the basic idea.) However, we can
use basic structural induction to bootstrap our way into more exotic
forms. In particular, we are going to define something called
well-founded induction.Note that I will use the terms “recursion”
and “induction” more or less interchangeably. In some contexts, people
make a distinction between the terms (typically a recursion
principle is a less-dependently-typed version of an induction
principle), but in this context, inductive proofs correspond, via
Curry-Howard, to (suitably restricted) recursive functions, so
“induction” and “recursion” describe the same thing from a logical and
computational viewpoint, respectively.
The idea of well-founded induction starts with the general idea of a
relation. A relation on A is just a function that takes two
values of type A and produces a type, representing evidence that the
two values are related (according to whatever kind of relationship we
have in mind).
Rel :Set→Set₁Rel A = A → A →Set
We have already seen quite a few relations, such as equality, \(<\),
\(\leq\), and divisibility.
Suppose we’re writing a recursive function, with some relation \(\prec\)
in mind, and for a given input \(x\) we’re only allowed to make
recursive calls on values \(y\) such that \(y \prec x\). If \(\prec\) is the
“is a syntactic subterm of” relation, then we get structural
recursion as usual. But what if \(\prec\) is some other relation? What
needs to be true about \(\prec\) for this to make sense? In particular, how
can we be sure that the function won’t get stuck in infinite recursion?
One’s first instinct might be to say that \(y \prec x\) needs to imply that
\(y\) is “smaller than” \(x\) somehow. But what does “smaller than” mean?
And in fact, “smaller than” doesn’t always work: for example, if we are
writing a function over the rational numbers, or even just the
integers, the usual “smaller than” relation does not guarantee our
function will terminate; it’s possible to continue choosing smaller
and smaller rational numbers or integers forever.
The key idea is exactly that this can’t happen: it’s not possible to
have an infinite chain of values where each is related to the
previous. That is, there should be no left-infinite chains \(\dots
\prec y_3 \prec y_2 \prec y_1 \prec x\). Then we are guaranteed that if we keep making
recursive calls on values that are related by \(\prec\) to the previous
value, we will have to stop eventually: after some finite number of calls
we will hit a value with nothing else related to it.
A relation \(\prec\) with this “no left-infinite chains” property is
called well-founded. But how do we encode this idea in Agda?
Accessibility
Instead of thinking negatively (no left-infinite
chains), the key is to think positively: all chains to the left of
every value are finite. Call a value accessible if all chains
leading to it are finite. Another way to say this is that a value is
accessible if every value related to it is also accessible:
data Acc (_≺_: Rel A): A →Setwhere acc :{x : A}→((y : A)→ y ≺ x → Acc _≺_ y)→ Acc _≺_ x
Acc _≺_ x defines what it means for a particular value x to be
accessible with respect to a relation \(\prec\). There is only one
constructor, acc, which requires (y : A) → y ≺ x → Acc _≺_ y—that
is, for every value y of type A, if y is related to x, then
y is accessible. In other words, x is accessible if and only if every y ≺ x
is accessible.
This is definitely tricky to wrap your head around! At this point
you may have two objections:
What about base cases? Shouldn’t we have another constructor which
says x is accessible if nothing is related to it? Actually,
the acc constructor already says that! If nothing is related to
x, then (y : A) → y ≺ x → Acc _≺_ y is trivially true: we can
easily promise anything we want as the output of a function if we
know it can never be called. Every natural number less than zero is a
purple flying weasel.
Doesn’t this just run into the same problem as before with
left-infinite chains? If we consider the “is one less than”
relation on the integers, isn’t \(2\) accessible because \(1\) is
accessible because \(0\) is accessible because \(-1\) is accessible
because … ?
There is something a bit subtle going on here: recursive data types
in Agda (unlike Haskell) are interpreted according to a
least fixed point semantics. Put in plain terms, the only values
of a data type are those which can be built by applications of a
finite number of constructors. So in fact, the “no left-infinite
chain” condition is foundationally built into the way Agda data
types work!
The idea is that we can turn well-founded induction into structural
induction by pattern-matching on Acc proofs! Starting with some
\(x\), any \(y \prec x\) has an accessibility proof which is a subterm of
the accessibility proof for \(x\). This is a bit exotic though: if we
pattern-match on the acc for \(x\) we get a function that yields an
accessibility proof for each \(y \prec x\); calling that function
produces another accessibility proof, which counts as a structural
subterm of the original. This seems a bit strange until you think of
a value of type Acc like an big, arbitrarily-branching tree of
finite depth; each node contains a function which really just stores
all the subtrees. In other words, a function of type (y : A) → y ≺ x → Acc _≺_ y can be thought of as a giant tuple of Acc values, one
for each y \prec x.I am not sure exactly how Agda handles this
internally, but I assume this is well-trodden ground for designers of
proof assistants.
Well-founded induction, defined
Given the definition of accessible elements, we can now give the
definition of a well-founded relation: a relation on A is well-founded if
every value of type A is accessible.
WellFounded : Rel A →SetWellFounded {A}_≺_=(a : A)→ Acc _≺_ a
We can now write down the principle of well-founded induction. This
is also quite tricky to wrap your brain around, so we’ll go through it
slowly. Previously, we were just talking about whether functions terminated or
not; but the reason this is important is that a function might be
calculating a proof. A function which purports to calculate a proof
but sometimes goes into infinite recursion is a charlatan, and not
really a proof at all.
So instead of thinking about termination, let’s switch to thinking
about proofs. Given a proposition \(P\), we want to prove that
\(P\) holds for every value of type \(A\). The idea is that when trying to
prove \(P(y)\) for a particular \(y\), we get to assume that \(P(z)\) holds
(i.e. we get to make recursive calls) for all \(z \prec y\).
Here, then, is the statement of the principle of well-founded induction, with
each argument broken out on a separate line so we can explain them as
we go.
wf-ind :{P : A →Set}
P represents an arbitrary proposition on A; our goal is to show P holds for
every value of type A.
{_≺_: Rel A}→
An arbitrary relation.
WellFounded _≺_→
A proof that ≺ is a well-founded relation.
((y : A)→((z : A)→ z ≺ y → P z)→ P y)→
This is the trickiest
argument to understand. Intuitively, it says “For any
y, if we know P z holds for all z ≺ y, then we can show P y
also holds.”
(x : A)→ P x
The principle of well-founded induction says that all of this is
enough to show that P x holds for allx : A.
So, how do we implement this? If we try something
straightforward, as in wf-ind-bad below—just call ind on x, then call wf-ind recursively
to fill in the proofs for P z—of course it does not work; Agda cannot
tell that this is terminating.As an aside, Agda complains that this definition of wf-ind-bad
is not terminating—which makes sense—but it continues to complain even
when I include the NON_TERMINATING pragma, which I don’t
understand. Perhaps this has been fixed in a more recent version of Agda.
And this makes sense, because we are
not even using the fact that the relation is well-founded at all!
module WFIndBad where
{-# NON_TERMINATING #-}
wf-ind-bad : {P : A → Set} {_≺_ : Rel A} → WellFounded _≺_ → ((y : A) → ((z : A) → z ≺ y → P z) → P y) → (x : A) → P x
wf-ind-bad wf ind x = ind x (λ z Rzx → wf-ind-bad wf ind z)
Instead, the right idea is to use the fact that \(\prec\) is
well-founded to generate an initial proof of accessibility for the
input \(x\), and then pattern match on accessibility proofs alongside
the values as we recurse. Every time we make a recursive call on some
\(y \prec x\), we can just pattern-match on the accessibility proof for
\(x\) to get an accessibility proof for \(y\), so Agda will be able to see
that the whole thing is structurally recursive on the accessibility
proofs.
wf-ind {A}{P}{_≺_} wf ind x = go x (wf x)where go :(x : A)→ Acc _≺_ x → P x go x (acc f)= ind x (λ z z≺x → go z (f z z≺x))
Less-than is well-founded
Now that we have well-founded induction under our belts, let’s show
that the less-than relation on natural numbers is well-founded. This
corresponds to what is often called “strong induction”.Incidentally,
we could probably have gotten away with directly defining a principle
of strong natural number induction, without bothering with the full
generality of well-founded induction, but this way is more fun and
interesting!
To prove that \(<\) is well-founded, we of course must
show that every natural number is accessible under \(<\). However, if
we directly try to prove (m : ℕ) → (Acc _<_) m, we run into a
variant of the exact same problem we have been dealing with: to prove
that \(m\) is accessible we need to know that every\(k < m\) is also
accessible, but again, we cannot show this directly by
recursion/induction, since \(k\) may not be a structural subterm of \(m\).
What we need is the usual trick for proving strong induction from weak
induction: instead of proving that \(P(x)\) holds for all \(x\), we prove
that \((\downarrow P)(x)\) holds for all \(x\), where \(\downarrow P\) is
the “downward closure” of \(P\). That is, \((\downarrow P)(x)\) says that
\(P(x)\) holds for all\(y < x\).Note that another way to define
\((\downarrow P)(x)\) is that \(P(y)\) holds for all \(y \sim x\), where \(\sim\)
is the transitive closure of the predecessor relation. As an
advanced exercise, generalize \(\downarrow_\prec P\) to be defined relative to
the transitive closure of any relation \(\prec\), and then prove that \(\prec\) is well-founded
if and only if its transitive closure is. For more along
these lines, see this very cool (but much more abstract) post on
well-founded induction by Callan McGill.
↓ :(ℕ →Set)→(ℕ →Set)↓ P n =(k : ℕ)→(k < n)→ P k
Now we can prove, for all natural numbers \(m\), that every natural
number up to and including \(m\) is accessible under the \(<\) relation. Zero is accessible
because nothing is less than it; everything up to the successor of \(m\) is accessible because by induction we know everything up to \(m\) is, and anything less than the successor of \(m\) must in fact be \(\leq m\).
<-acc :(m : ℕ)→ ↓ (Acc _<_) m<-acc zero =λ k ()<-acc (suc m) zero (sle le)= acc (λ y ())<-acc (suc m)(suc k)(sle le)= acc (λ y y<sk → <-acc m y (<-≤-trans y<sk le))
Finally, to show that any natural number \(n\) is accessible—i.e. that \(<\)
is well-founded—we can use the fact that all numbers less than the successor of \(n\)
are accessible, and just project out accessibility for \(n\) itself.
<-wf : WellFounded _<_<-wf n = <-acc (suc n) n (n <suc)
The division algorithm
Finally, we can define the division algorithm, via well-founded
induction! The definition is very similar to our first attempt, but we use the principle of well-founded induction with \(<\). Note that neither divAlg nor its helper function go is directly recursive. Instead, go takes an induction hypothesis as an argument, which we call instead, providing an extra proof that the subject of the induction hypothesis is in fact less than the original input. wf-ind takes care of the actual recursion.
divAlg :(n d : ℕ)→(0 < d)→ DivAlg n ddivAlg n d 0<d = wf-ind {P =λ n → DivAlg n d} <-wf go nwhere go :(n : ℕ)→((n′ : ℕ)→ n′ < n → DivAlg n′ d)→ DivAlg n d go n IH with n decreaseBy? d...| LT n<d =(0 , n) , DM (n +0) n<d...| GE n′ n′+d≡n with IH n′ (+→< 0<d n′+d≡n)...|(q , r) , dm =(suc q , r) , incDivMod n′+d≡n dm
Using the division algorithm, we can also finally decide whether one
number divides another: zero divides zero; zero does not divide any
successor since that would imply there is some \(k\) such that \(k\) times
zero is nonzero, which is absurd; and if \(x\) is a successor, we can
apply the division algorithm and check the remainder, applying some
previous lemmas that say what zero and nonzero remainders tells us
about divisibility.
_∣?_:(x y : ℕ)→ Dec (x ∣ y)zero ∣? zero = yes (0 , refl)zero ∣? (suc y)= no λ{(a , eq)→ absurd (noConf (trans (sym (*-comm a zero)) eq))}(suc x) ∣? y with divAlg y (suc x)(sle zle)...|(q , zero) , dm = yes (mod0→divides y (suc x) dm)...|(q , suc r) , dm = no (modS→¬divides y (suc x) dm)
Primality testing
To test a number for primality, we are just going to use
straightforward, naive trial division. The straightforward way of
doing this is by starting at \(2\) and counting up—but this is a
problem, because when pattern-matching on natural numbers we most
naturally count down.
Well, remember—build the evidence you want to pattern-match on! Let’s
develop some machinery for counting up instead of down.
Counting up
The problem starts with \(\leq\): a proof of \(m \leq n\) starts with a
base case representing evidence that \(0 \leq k\), then every
constructor application of sle increments both sides by one.
Pattern-matching on a proof of \(m \leq n\) thus either reveals that \(m
= 0\), or that \(m' \leq n'\) where \(m'\) and \(n'\) are the predecessors of
\(m\) and \(n\): in other words, it facilitates counting down, just
like matching directly on \(m\) would.
However, there is an alternative way to define the \(\leq\) relation,
which we will call \(\leq'\).
We can choose reflexivity of \(\leq'\) as a base case, that is, \(n \leq'
n\) for any \(n\). We can then decrement the left-hand side every time
we apply another constructor. Like so:
data_≤′_: ℕ → ℕ →Setwhere lerefl :{n : ℕ}→ n ≤′ n lesuc :{m n : ℕ}→ suc m ≤′ n → m ≤′ n
Pattern-matching on a proof of \(m \leq' n\) thus facilitates counting
up from \(m\) to \(n\), just like we wanted!
We can also prove a few lemmas about properties of \(\leq'\). For
example, the two axioms that define the usual \(\leq\) can be proved as
lemmas.
≤′-suc :{m n : ℕ}→ m ≤′ n → suc m ≤′ suc n≤′-suc lerefl = lerefl≤′-suc (lesuc m≤′n)= lesuc (≤′-suc m≤′n)0≤′ :(n : ℕ)→0 ≤′ n0≤′ zero = lerefl0≤′ (suc n)= lesuc (≤′-suc (0≤′ n))
We can also prove that \(m \leq n\) implies \(m \leq' n\). (The converse
is true as well, and can be proved as an easy exercise, but we won’t need it.)
≤→≤′ :{m n : ℕ}→ m ≤ n → m ≤′ n≤→≤′ {n = n} zle = 0≤′ n≤→≤′ (sle m≤n)= ≤′-suc (≤→≤′ m≤n)
So, we can pattern-match on a proof of \(m \leq' n\) to count up from \(m\) to
\(n\), but this isn’t quite good enough: while counting, we won’t remember the relationship of the
intermediate values to the original \(m\). We would like to be able to
count up through some interval, from some starting \(a\) to ending \(b\),
knowing all along the way that the values we count are contained in
the interval.
To this end, we can create a data
type i ∈[ a ⋯ b ] which represents a stage in counting from \(a\) to
\(b\). The base case is when \(i = b\); otherwise i ∈[ a ⋯ b ] when \(a
\leq i\) and also suc i ∈[ a ⋯ b ].
data_∈[_⋯_] : ℕ → ℕ → ℕ →Setwhere stop :{a b : ℕ}→ a ≤ b → b ∈[ a ⋯ b ] step :{a i b : ℕ}→ a ≤ i → suc i ∈[ a ⋯ b ] → i ∈[ a ⋯ b ]
We can then write a function which “constructs a loop”—that is,
starting from a proof of \(a \leq' b\), it builds
a value of type a ∈[ a ⋯ b ] which represents a reified loop from
\(a\) to \(b\). By pattern-matching on this value we can successively
increment from \(a\) up to \(b\), with the appropriate guarantees along
the way.
loop :(a b : ℕ)→(a ≤′ b)→ a ∈[ a ⋯ b ]loop a b a≤′b = mid a a b ≤-refl a≤′bwhere mid :(a i b : ℕ)→(a ≤ i)→(i ≤′ b)→ i ∈[ a ⋯ b ] mid a i b a≤i lerefl = stop a≤i mid a i b a≤i (lesuc i≤′b)= step a≤i (mid a (suc i) b (≤-sucr a≤i) i≤′b)
Finally, we need as a simple lemma the fact that if i ∈[ a ⋯ b ]
then in fact \(i \leq b\).
top :{i a b : ℕ}→ i ∈[ a ⋯ b ] → i ≤ btop (stop _)= ≤-refltop (step _ s)= ≤-sucl (top s)
Primality testing by trial division
First, a lemma about downward closure: if we know \((\downarrow P)(m)\),
and we know \(P(m)\), then we know \((\downarrow P)(1 + m)\). In other
words, if \(P\) holds for everything less than \(m\), we can extend it by
one by providing a proof that \(P\) also holds for \(m\).
extend :{P : ℕ →Set}{m : ℕ}→(↓ P) m → P m →(↓ P)(suc m)extend {m = m} soFar Pm j with(j ≟ m)...| yes refl =λ_→ Pm...| no j≢m =λ{(sle j≤m)→ soFar j (≤≢→< j≤m j≢m)}
Now we can define primality testing itself, via trial division. We
loop from \(2\) up to \(n\), testing each number to see if it divides
\(n\), keeping track along the way of the fact that all the numbers less
than our current trial divisor do not divide \(n\). If the next divisor
does divide \(n\), we return it as proof that \(n\) is composite. If it
does not, we extend our accumulating proof of all the numbers that do
not divide \(n\), and proceed to the next. If we reach \(n\), our
accumulated proof tells us that none of the numbers less than \(n\)
divide it, which is proof that \(n\) is prime.
prime? :(n : ℕ)→(2 ≤ n)→ Prime n ⊎ Composite nprime? n 2≤n = trialDiv (loop 2 n (≤→≤′ 2≤n)) noDivisorsUpTo2where NoDivisorsUpTo : ℕ →Set NoDivisorsUpTo = ↓ (λ # →(2 ≤ #)→ ¬ (# ∣ n)) noDivisorsUpTo2 : NoDivisorsUpTo 2 noDivisorsUpTo2 (suc zero)_(sle ())_ noDivisorsUpTo2 (suc (suc j))(sle (sle ()))__ trialDiv :{m : ℕ}→ m ∈[ 2 ⋯ n ] → NoDivisorsUpTo m → Prime n ⊎ Composite n trialDiv (stop 2≤n) soFar = inj₁ (2≤n , soFar) trialDiv {m}(step 2≤m next) soFar with m ∣? n...| yes m∣n = inj₂ (m , 2≤m , top next , m∣n)...| no pf = trialDiv next (extend soFar (λ_→ pf))
Lists
Before we are able to state the Fundamental Theorem of Arithmetic, we need to build up a data type for lists, along with some standard list manipulation functions. First, we define the type of lists and the standard foldr function.
data List (A :Set):Setwhere [] : List A_∷_: A → List A → List Afoldr :(A → B → B)→ B → List A → Bfoldr _&_ z [] = zfoldr _&_ z (x ∷ xs)= x & foldr _&_ z xs
Now we can define concatenation and product via foldr.
_++_: List A → List A → List Axs ++ ys = foldr (_∷_) ys xsproduct : List ℕ → ℕproduct = foldr _*_1
We will need All, which expresses that some predicate holds of all
the elements of a list. In fact, All is manifestly an instance of
foldr as well, but we would need a universe-polymorphic version of foldr for that, so we just write it manually.
All :(P : A →Set)→ List A →SetAll P [] = ⊤All P (x ∷ xs)= P x × All P xs
Now, we just need a couple lemmas about concatenation: first, that if P holds for all the elements in xs and all the elements in ys, then it holds for all the elements in xs ++ ys; and second, that product distributes over concatenation (i.e. it is a homomorphism from the monoid of lists under concatenation to the monoid of natural numbers under multiplication).
All-++ :{P : A →Set}{xs ys : List A}→ All P xs → All P ys → All P (xs ++ ys)All-++ {xs = []} Pxs Pys = PysAll-++ {xs =_ ∷ _}(Px , Pxs) Pys = Px , All-++ Pxs Pysproduct-++ :(xs ys : List ℕ)→ product (xs ++ ys) ≡ product xs * product ysproduct-++ [] ys = sym (_ +0)product-++ (x ∷ xs) ys = trans (x *_ $≡ product-++ xs ys)(sym (*-assoc x (product xs)(product ys)))
The Fundamental Theorem of Arithmetic
Finally, we can put all the pieces together to state and prove (one
half of) the Fundamental Theorem of Arithmetic! FTA n says that for
some positive integer n, we can find a list of natural numbers which
are all prime, and whose product is n.
FTA : ℕ →SetFTA n = Σ (List ℕ)(λ ps → All Prime ps × product ps ≡ n)
To prove that this holds for all positive integers, we can again use
well-founded induction: in the base case, if \(n = 1\), the empty list
suffices. Otherwise, we can decide whether \(n\) is prime. If so, the
singleton list containing \(n\) fits the bill. Otherwise, \(n = ab\) where
\(a\) and \(b\) are both nontrivial divisors of \(n\); by the induction
hypothesis, both can be factored into primes, and the list we want for
\(n\) is simply the concatenation of the factorizations for \(a\) and \(b\).
fta :(n : ℕ)→(0 < n)→ FTA nfta n = wf-ind {P =(λ n →(0 < n)→ FTA n)} <-wf go nwhere go :(n : ℕ)→((n′ : ℕ)→ n′ < n →0 < n′ → FTA n′)→0 < n → FTA n go (suc zero) IH 0<n = [] , (tt , refl) go (suc (suc n)) IH 0<n with prime? (suc (suc n))(sle (sle zle))...| inj₁ P =(suc (suc n) ∷ []) , ((P , tt) , (suc $≡ (suc $≡ (n *1))))...| inj₂ C with factorsOf _ C...|((a , sle _ , a<n , _) , (b , sle _ , b<n , _)) , ab≡nwith IH a a<n (sle zle)| IH b b<n (sle zle)...| ps₁ , Pps₁ , prod₁ | ps₂ , Pps₂ , prod₂ =(ps₁ ++ ps₂) , (All-++ Pps₁ Pps₂) , begin product (ps₁ ++ ps₂) ≡[ product-++ ps₁ ps₂ ⟩≡ product ps₁ * product ps₂ ≡[ _*_ $≡ prod₁ ≡$≡ prod₂ ⟩≡ a * b ≡[ ab≡n ⟩≡ suc (suc n) ∎
Further Directions
That concludes our tour of the Fundamental Theorem of Arithmetic in Agda! If you’ve worked through the whole thing and
completed the proofs mostly on your own, congratulations! If you want
more practice, there are a lot of directions you could take this:
Doing trial division all the way up to \(n\) is silly; we can stop
when we get to \(\sqrt n\). I think this would make for a nice
exercise (in fact, Taneb’s version in the Agda stdlib does this).
Define “\(d\) is a proper divisor of \(n\)” to mean that that \(d \mid n\)
and \(2 \leq d < n\), then show that the “is a proper divisor of” relation
is well-founded, and prove fta via well-founded induction on that
relation instead. This might streamline some parts of the proof;
I’m not sure.
The other half of the FTA says that the prime factorization is
unique up to permutation. To prove this, one has to define
what it means for one list to be a permutation of another, and show that
if there are two different prime factorizations then one must be a
permutation of the other (if \(p\) is a prime from the first
factorization, then it divides the other factorization as well,
which means it must be equal to one of the primes in the other
factorization). I may write up this proof in a follow-up post.
<noscript>Javascript needs to be activated to view comments.</noscript>
The GHC developers are very pleased to announce the availability
of the second release candidate for GHC 9.12.5, ghc-9.12.5-rc2. The first release candidate did not receive an announcement since promptly after publishing it, an urgent issue with distribution of the process library on macos was discovered. This is now fixed.
Binary distributions, source distributions, and documentation are available at
downloads.haskell.org and via GHCup.
GHC 9.12.5 is a bug-fix release fixing many issues of a variety of
severities and scopes, including:
Implement version 2 of the ghc semaphore protocol for GHC’s -jsem jobserver on Linux and other POSIX platforms (#25087, #27253)
several improvements to the pattern match checker (#25926, #27124)
Mind that the very important overhaul of the semaphore-compat library to v2 is included in this release and that the semaphore feature will only start working with cabal-install 3.18 again, which is sadly a breaking but necessary change.
A full accounting of these fixes can be found in the
release notes. As always, GHC’s release status, including planned future
releases, can be found on the GHC Wiki status.
This release candidate will have a two-week testing period. If all goes well
the final release will be available the week of 8 July 2026.
We would like to thank these sponsors and other anonymous contributors
whose on-going financial and in-kind support has facilitated GHC maintenance
and release management over the years. Finally, this release would not have
been possible without the hundreds of open-source contributors whose work
comprise this release.
As always, do give this release a try and open a ticket if you see
anything amiss.
In the previous post, we learned how to get started with managing and building a Haskell project with Nix. In this post, we learn how to easily create statically-linked executables for Haskell projects with Nix.
I recommend going through the previous post, because we are going to start off from where we left last time (ignoring the bonus sections). This is how our project’s directory tree looks at this point:
Main.hs is the default generated main file that prints “Hello, Haskell!�. ftr.cabal is the default generated Cabal file. sources.(json|nix) are generated by Niv to pin Nixpkgs to a particular revision. nixpkgs.nix provides the nixpkgs that we use for building tools and dependencies. package.nix and shell.nix build the package and manage the Nix shell respectively. We are not going to touch any of these files in this post. Let’s get started.
A static build is an executable that is statically-linked against all the libraries it depends on. This is in contrast to a dynamically-linked executable, which contains references to the libraries it depends on, and those libraries are loaded and linked when the executable runs. While dynamic linking has its benefits, the main advantage of static linking is that the executable can be shipped by itself, without needing to ship or install dependency libraries. This makes it quite attractive for deploying backend services. You download and deploy that one binary executable file and you are done! No need to care about installing and maintaining its dependencies.
Many compilers support static builds—Go and Rust being two. Haskell compiler GHC also supports it, but not out-of-the-box. To statically link a Haskell executable, we need to configure GHC itself, and then configure the executable build as well. We also need to configure GHC to link with musl libc. That’s where Nix helps us by smoothing out the process1.
As mentioned, first we need a GHC configured to do static builds. We create a nixpkgs derivation, separate from nixpkgs.nix, that contains the custom configured GHC.
Let’s go over it piece-by-piece. First, we take the arch and ghcVersion parameters, letting us build the package for different architectures (X86-64 and AArch64), and for different GHC versions. We default the ghcVersion to the default GHC in nixpkgs.
The derivation is same as nixpkgs.nix, except we add some overlays. The first overlay adds the custom configured GHC for static builds. We enable certain configurations for that purpose:
enableRelocatedStaticLibs = true
Configures GHC runtime system and core packages to be built with position independent code so that they can be loaded for template Haskell.
enableShared = false
Disables building dynamically-linkable libraries, so they are built only as static archives.
enableDwarf = false
Disables DWARF-based stack traces, because it is unavailable on musl targets.
enableProfiledLibs = false
Disables building profiling enabled libraries.
enableDocs = false
Disables generation of documentation.
enableNativeBignum = true
Makes GHC use pure-Haskell based native bignum backend ghc-bignum instead of GMP, so that the package executables it creates are GPL-free. You may remove this setting if you are okay with GPL executables.
The buildHaskellPackages related lines set the custom GHC as the compiler for Haskell-based tools used in Nix2.
The second overlay makes cabal2nix—the tool used to convert .cabal files into Nix derivations—use the custom GHC. The third overlay disables documentation generation, testing, and profiling of all Haskell libraries built with the custom GHC. We do this to save the build time, assuming that static builds are for release only, and the docs, tests, and profiling are done using a normal GHC.
Building this custom GHC may take anywhere from several minutes to several hours depending on the build machine configuration34. But this is a one-time price to pay, as long as we keep the GHC build around. Next, we configure our package to be built as a statically-linked executable.
package-static.nix also takes arch and ghcVersion as parameters, and passes them to nix/nixpkgs-static-ghc.nix to create the nixpkgs with the custom GHC as described above. This give us pkgsOrig, from which we get the pkgsMusl version. pkgsMusl is same nixpkgs, except every executable in it links to musl libc. We capture this as pkgs, and use it to build our Haskell package.
When linking the executable, we need to link it against static version of all the dependency libraries it depends on. That’s what nix/static-deps.nix file provides us. We’ll look at it in the next section, but for now, we see that it gives us the libffi, zlib, and numactl libraries5.
Finally, we get to the package configuration. It starts the same as package.nix, using cabal2nix to connect the Haskell project to Nix, but then, we provide a list of custom configurations. We disable Haddock docs, hyperlinked source docs, coverage tests, profiling, and shared library build. We enable static executable build and dead code elimination. Then we configure cabal to run builds with multithreading, and add lld to its list of build tools.
Configures GHC to use lld as the linker, which is much faster than the default linker. You can omit these lines to use the default linker. Or you can replace all metions of lld with mold to use the Mold linker, which may be even faster depending on your project.
Enables reductions in binary size by removing dead code6.
--extra-lib-dirs=...
These lines allow GHC to link the output executable against the static version of the mentioned dependency libraries.
Finally, the last function in the pipeline uses UPX to compress the output executable. This generally results in a large reduction in the binary size7.
Now we can actually build the statically-linked exe:
The first and second line above build the exe for the X86-64 and AArch64 architectures with the default GHC version. The third line specifies a different GHC version to build with. Here is the cleaned-up output log for the first command:
Output log
$ nix-build --argstr arch x86_64 package-static.nix
these 2 derivations will be built:
/nix/store/42a291bq7ydvkdy3fdsyj82axrfsi6sy-ftr-0.1.0.0.drv
/nix/store/2c2l50la8291q0jrqlc23bybaxwip8y2-ftr-0.1.0.0-compressed.drv
building '/nix/store/42a291bq7ydvkdy3fdsyj82axrfsi6sy-ftr-0.1.0.0.drv' on 'ssh-ng://builder@linux-builder'...
copying 1 paths...
copying path '/nix/store/l9ls307kzxby72hqj4yl7ri7m8s3b3fk-source' to 'ssh-ng://builder@linux-builder'...
building '/nix/store/42a291bq7ydvkdy3fdsyj82axrfsi6sy-ftr-0.1.0.0.drv'...
Running phase: setupCompilerEnvironmentPhase
Build with /nix/store/717lxds14ra0ndbnin2qhdhh91d3b69g-ghc-musl-native-bignum-9.10.3.
Running phase: unpackPhase
unpacking source archive /nix/store/l9ls307kzxby72hqj4yl7ri7m8s3b3fk-source
source root is source
Running phase: patchPhase
Running phase: compileBuildDriverPhase
setupCompileFlags: -package-db=/nix/var/nix/b/10kwxdphxvyy519y831ryji7fn/b/tmp.EPOEKNjT6Z/setup-package.conf.d -threaded
[1 of 2] Compiling Main ( /nix/store/4mdp8nhyfddh7bllbi7xszz7k9955n79-Setup.hs, /nix/var/nix/b/10kwxdphxvyy519y831ryji7fn/b/tmp.EPOEKNjT6Z/Main.o )
[2 of 2] Linking Setup
Running phase: updateAutotoolsGnuConfigScriptsPhase
Running phase: configurePhase
configureFlags: --verbose --prefix=/nix/store/pb2zay1k8b0vifhx7ghd5j6lbncq4b66-ftr-0.1.0.0 --libdir=$prefix/lib/$compiler/lib --libsubdir=$abi/$libname --with-gcc=gcc --package-db=/nix/var/nix/b/10kwxdphxvyy519y831ryji7fn/b/tmp.EPOEKNjT6Z/package.conf.d --ghc-option=-j4 --ghc-option=+RTS --ghc-option=-A64M --ghc-option=-RTS --disable-library-profiling --disable-profiling --disable-shared --disable-coverage --enable-static --disable-executable-dynamic --disable-tests --disable-benchmarks --enable-library-vanilla --disable-library-for-ghci --enable-split-sections --enable-library-stripping --enable-executable-stripping -O2 --ghc-option=-fPIC --ghc-option=-split-sections --ghc-option=-optl-fuse-ld=lld --ld-option=-fuse-ld=lld --ld-option=-Wl,--gc-sections,--build-id,--icf=all --with-ld=ld.lld --ghc-option=-optl=-static --extra-lib-dirs=/nix/store/yi771fg1dfj1bg618vv5flmisy8zw3hm-libffi-3.5.2/lib --extra-lib-dirs=/nix/store/jk77s356gjn68dcrzpz1m7m5amzxmkw8-zlib-1.3.2-static/lib --extra-lib-dirs=/nix/store/044b10glmg0f3yyijmrwrgv5lsys6x6n-numactl-2.0.18/lib --extra-lib-dirs=/nix/store/m1j2f9b1h6pbq1mq5ibnw4cpp60w5dfi-libffi-3.5.2/lib --extra-include-dirs=/nix/store/j9c1ifaa7vph3zxfbzb55y1frm0vp4xm-musl-iconv-1.2.5/include --extra-lib-dirs=/nix/store/lrrmafbkrpa4f3wxfz6a3sd3dv6xgp7n-numactl-2.0.18/lib
[snip]
Running phase: buildPhase
Preprocessing executable 'ftr' for ftr-0.1.0.0...
Building executable 'ftr' for ftr-0.1.0.0...
[1 of 1] Compiling Main ( app/Main.hs, dist/build/ftr/ftr-tmp/Main.o )
[2 of 2] Linking dist/build/ftr/ftr
Running phase: haddockPhase
Running phase: installPhase
Installing executable ftr in /nix/store/pb2zay1k8b0vifhx7ghd5j6lbncq4b66-ftr-0.1.0.0/bin
Warning: The directory
/nix/store/pb2zay1k8b0vifhx7ghd5j6lbncq4b66-ftr-0.1.0.0/bin is not in the
system search path.
Running phase: fixupPhase
shrinking RPATHs of ELF executables and libraries in /nix/store/pb2zay1k8b0vifhx7ghd5j6lbncq4b66-ftr-0.1.0.0
shrinking /nix/store/pb2zay1k8b0vifhx7ghd5j6lbncq4b66-ftr-0.1.0.0/bin/ftr
patchelf: cannot find section '.dynamic'. The input file is most likely statically linked
checking for references to /nix/var/nix/b/10kwxdphxvyy519y831ryji7fn/b/ in /nix/store/pb2zay1k8b0vifhx7ghd5j6lbncq4b66-ftr-0.1.0.0...
patchelf: cannot find section '.dynamic'. The input file is most likely statically linked
patching script interpreter paths in /nix/store/pb2zay1k8b0vifhx7ghd5j6lbncq4b66-ftr-0.1.0.0
stripping (with command strip and flags -S -p) in /nix/store/pb2zay1k8b0vifhx7ghd5j6lbncq4b66-ftr-0.1.0.0/bin
copying 1 paths...
copying path '/nix/store/pb2zay1k8b0vifhx7ghd5j6lbncq4b66-ftr-0.1.0.0' from 'ssh-ng://builder@linux-builder'...
building '/nix/store/2c2l50la8291q0jrqlc23bybaxwip8y2-ftr-0.1.0.0-compressed.drv' on 'ssh-ng://builder@linux-builder'...
copying 0 paths...
building '/nix/store/2c2l50la8291q0jrqlc23bybaxwip8y2-ftr-0.1.0.0-compressed.drv'...
Running phase: unpackPhase
unpacking source archive /nix/store/pb2zay1k8b0vifhx7ghd5j6lbncq4b66-ftr-0.1.0.0
source root is ftr-0.1.0.0
Running phase: patchPhase
Running phase: updateAutotoolsGnuConfigScriptsPhase
Running phase: configurePhase
no configure script, doing nothing
Running phase: buildPhase
no Makefile or custom buildPhase, doing nothing
Running phase: installPhase
Ultimate Packer for eXecutables
Copyright (C) 1996 - 2026
UPX 5.1.1 Markus Oberhumer, Laszlo Molnar & John Reiser Mar 5th 2026
File size Ratio Format Name
-------------------- ------ ----------- -----------
1356096 -> 525804 38.77% linux/amd64 ftr bin/ftr [linux/amd64, LZMA/1]
Packed 1 file.
Running phase: fixupPhase
shrinking RPATHs of ELF executables and libraries in /nix/store/j8gg1x3vrlb5dc1mh149ys0nih9fvmwk-ftr-0.1.0.0-compressed
shrinking /nix/store/j8gg1x3vrlb5dc1mh149ys0nih9fvmwk-ftr-0.1.0.0-compressed/bin/ftr
patchelf: no section headers. The input file is probably a statically linked, self-decompressing binary
checking for references to /nix/var/nix/b/19b7g2frvcvani3cnj61lsb4fq/b/ in /nix/store/j8gg1x3vrlb5dc1mh149ys0nih9fvmwk-ftr-0.1.0.0-compressed...
patchelf: no section headers. The input file is probably a statically linked, self-decompressing binary
patching script interpreter paths in /nix/store/j8gg1x3vrlb5dc1mh149ys0nih9fvmwk-ftr-0.1.0.0-compressed
stripping (with command strip and flags -S -p) in /nix/store/j8gg1x3vrlb5dc1mh149ys0nih9fvmwk-ftr-0.1.0.0-compressed/bin
copying 1 paths...
copying path '/nix/store/j8gg1x3vrlb5dc1mh149ys0nih9fvmwk-ftr-0.1.0.0-compressed' from 'ssh-ng://builder@linux-builder'...
/nix/store/j8gg1x3vrlb5dc1mh149ys0nih9fvmwk-ftr-0.1.0.0-compressed
The output log mentions:
patchelf: cannot find section ‘.dynamic’. The input file is most likely statically linked
We can also verify for ourselves:
$ file /nix/store/j8gg1x3vrlb5dc1mh149ys0nih9fvmwk-ftr-0.1.0.0-compressed/bin/ftr
/nix/store/j8gg1x3vrlb5dc1mh149ys0nih9fvmwk-ftr-0.1.0.0-compressed/bin/ftr: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), BuildID[sha1]=df53df80d301b4bba2a7634a4169c6291d64ea72, statically linked, no section header
Dynamically-linked Haskell builds contain references to their dependency libraries and GHC that was used to build it. If you use direnv or install a dynamically-linked executable, it creates Nix GC roots for the libraries and GHC, preventing them from being garbage-collected by Nix. But statically-linked builds have no references to anything, as intended. So we need to create GC roots by ourselves to the libraries and the GHC toolchain. This is even more important because building the custom GHC may be an extremely time-consuming affair.
First, we list all dependencies in a separate file:
This file lists the dependency libraries and the GHC toolchain. Notice how we override each library’s config to make it statically-linkable. I’ve included some additional libraries here (gmp6 and ncurses) that are generally used by Haskell projects, but we don’t use them in this project. You may have to add more of such libraries depending on your project’s dependencies.
We already saw how we use this file in package-static.nix. Now, we use it to create Nix GC roots:
package-static-deps.nix simply gathers all dependencies from nix/static-deps.nix and creates a directory with symlinks to them. This brings us to the finale.
One more thing static builds are great for: wrapping them into Docker images. Since they are much smaller than dynamically-linked executables and their dependencies combined, they are better to package as Docker images. Here’s how we do it:
This image also shows how to package extra Nix packages in images, setting up a non-root user to run the executable, and setting up user-owned directories to expose as volumes. We can build the image by running:
This post shows how to configure GHC and Haskell projects to build statically-linked executables that are fully portable and independent. If your Haskell project has any complex requirements, such as custom dependency versions, patched dependencies, custom non-Haskell dependencies etc., this setup may not scale. In such case you can either grow this setup by learning Nix in more depth with the help of the official Haskell with Nix docs and this great tutorial, or switch to using a framework like haskell.nix or haskell-flake. For dealing with complex static builds, static-haskell-nix project may be of help.
If you have any questions or comments, please leave a comment below. If you liked this post, please share it. Thanks for reading!
This is the first part of a miniseries on this year’s Symposium on Principles of Programming Languages, a.k.a. POPL 2026, hosted by Jessica Foster.
In this episode we talk about: symbolic execution monads, what a lazy linear core in Haskell might have in common with Rust, hyperfunctions, the hallway track, and how to deal with rejection.
Today the 17 October 2019 I discussed a very remarkable fixed point theorem discovered by the Ukrainian mathematician Oleksandr Micholayovych Sharkovsky.
We recall that a periodic point of period for a function is a point such that . With this definition, a periodic point of period is also periodic of period for every which is a multiple of . If but for every from 1 to , we say that is the least period of .
Theorem 1. (Sharkovsky’s “little” theorem)Let be an interval and a continuous function. If has a point of least period 3, then it has points of arbitrary least period; in particular, it has a fixed point.
Note that no hypothesis is made on being open or closed, bounded or unbounded.
Our proof of Sharkovsky’s “little” theorem follows the one given in (Sternberg, 2010), and could even be given in a Calculus 1 course: the most advanced result will be the intermediate value theorem.
Lemma 1.Let be a compact interval of the real line and a continuous function. Suppose that for some compact interval it is . Then has a fixed point in .
Proof. Let and be the minimum and the maximum of in , respectively. As , it is and . Choose such that and . Then is nonpositive at and nonnegative at . By the intermediate value theorem applied to , must have a fixed point in the closed and bounded interval (possibly reduced to a single point) delimited by and , which is a subset of .
Lemma 2.In the hypotheses of Lemma 1, let be a closed and bounded interval contained in . Then there exists a closed and bounded subinterval of such that .
Proof. Let . We may suppose , otherwise the statement is trivial. Let be the largest such that . Two cases are possible.
There exists such that . Let be the smallest such , and let . Then surely , but if for some we had either or , then by the intermediate value theorem, for some  we would also have either or , against our choice of and .
for every . Let then be the largest such that , and let . Then for reasons similar to those of the previous point.
Proof of Sharkovsky’s “little” theorem. Let be such that , , and . Up to cycling between these three values and replacing with , we may suppose . Fix a positive integer : we will prove that there exists such that and for every .
Let and be the “left” and “right” side of the closed and bounded interval : then and by the intermediate value theorem. In particular, , and Lemma 1 immediately tells us that has a fixed point in . Also, , so also has a point of period 2 in , again by Lemma 1: call it . This point cannot be a fixed point, because then it would also belong to as , but which has period 3. As we can obviously take , we only need to consider the case .
By Lemma 2, there exists a closed and bounded subinterval of such that . In turn, as , there also exists a closed and bounded subinterval of such that , again by Lemma 2: but then, . By iterating the procedure, we find a sequence of closed and bounded intervals such that, for every , and .
We stop at and recall that : we are still in the situation of Lemma 2, with in the role of . So we choose as a closed and bounded subinterval not of , but of , such that . In turn, as , there exists a closed and bounded subinterval of such that . Following the chain of inclusions we obtain . By Lemma 1, has a fixed point in , which is a periodic point of period for .
Can the least period of for be smaller than ? No, it cannot, for the following reason. If has least period , then so has , and in addition is divisible by . But while for every . Consequently, if has least period , then . But this is impossible, because by construction as , while .
Theorem 1 is a special case of a much more general, and complex, result also due to Sharkovsky. Before stating it, we need to define a special ordering on positive integers.
Definition. The Sharkovsky ordering between positive integers is defined as follows:
Identify the number , with odd integer, with the pair .
Sort the pairs with in lexicographic order.
That is: first, list all the odd numbers larger than 1, in increasing order; then, all the doubles of the odd numbers larger than 1, in increasing order; then, all the quadruples of the odd numbers larger than 1, in increasing order; and so on.
For example, and
Set for every and .
That is: the powers of 2 follow, in the Sharkovskii ordering, any number which has an odd factor.
For example, .
Sort the pairs of the form —i.e., the powers of 2—in reverse order.
The set of positive integer with the Sharkowsky ordering has then the form:
Note that is a total ordering.
Theorem 2. (Sharkovsky’s “great” theorem)Let be an interval on the real line and let be a continuous function.
If has a point of least period , and , then has a point of least period . In particular, if has a periodic point, then it has a fixed point.
For every integer it is possible to choose and so that has a point of minimum period and no points of minimum period for any . In particular, there are functions whose only periodic points are fixed.
Bibliography:
Keith Burns and Boris Hasselblatt. The Sharkovsky theorem: A natural direct proof. The American Mathematical Monthly118(3) (2011), 229–244. doi:10.4169/amer.math.monthly.118.03.229
Robert L. Devaney, An Introduction to Chaotic Dynamical Systems, Second Edition, Westview Press 2003.
We talked to Fraser Tweedale. Fraser works at Red Hat, and is on the Haskell Security Response Team. We talked about security in the context of Haskell, both technical and organizational issues, and also the political issues involved. Fraser's work is both really important and not well-known in the Haskell ecosystem, so it was high time for him to come on the show.
I’ve also been experimenting with agent sandboxes lately. redoubtful is a work-in-progress sandbox that supports:
Linux-only sandboxes: I’m focusing on what Linux supports, specifically, rather than trying to support the lowest-common-denominator features that work cross platform.
Modular configuration profiles: See below.
Isolation using pasta and bwrap.
A shadow filesystem that looks like your home directory, so things like git worktree actually work correctly. You can also selectively mount existing parts of your filesystem in read-only or read-write mode.
Network port forwarding and filtering proxy server.
TODO: Proxy credential support.
But first, a warning: Nearly 100% of this code was written by coding agents, much of it by a local Qwen3.6 27B. I am, however, keeping a very close eye on the output—one of my goals here is to see just what a small agent like this can do. This is maybe only 80% as good as my handwritten code would be a similar point in a project.
And finally, this is an incomplete work-in-progress, and it has not been packaged nicely for anyone besides me yet.
Modular configuration “profiles”
One of the slightly novel parts of all this is the ability to define modular configuration. This allows us to invoke a sandbox with a specific set of credentials:
redoubtful run --uses pi --uses llama-server pi
Here, we’re running the pi.dev coding agent with a locally-served Qwen3.6 27B via llama-server. Qwen3.6 27B is a fantastic lightweight coding model, and it works very well with pi.dev’s minimalist prompt. And since we’re running in a sandbox, we don’t care that pi.dev provides no sandbox and no confirmation before acting.
To set up these two profiles, we first define a node profile:
# Standard Node setup. If you're using `nvm`, you'll need to fix the path_add# entry to point to the correct nvm version.## We might want some kind of plugin system to handle messy things like nvm.[profile.node]mounts=[{host="~/.npm-global"},{host="~/.local/share/nvm",access="rw"},]path_add=["~/.npm-global/bin","~/.local/share/nvm/v24.15.0/bin"]
Then let’s make Rust work:
# A Rust setup, with optional rustup and advisory support.[profile.rust]mounts=[{host="~/.rustup"},{host="~/.cargo"},# Cargo audit/deny support, which needs to take a lock to update the# advisory database.{host="~/.cargo/advisory-dbs/",access="rw"},]path_add=["~/.cargo/bin"]
And then basic git is easy—we just need enough config to read user.name and user.email:
# Things you will likely want for git.[profile.git]mounts=[{host="~/.gitconfig"}]
And then finally, we can set up pi itself:
# Profile for the pi coding agent. Run with:## redoubtful run -u pi pi[profile.pi]uses=["node","rust","git"]mounts=[{host="~/.pi",access="rw"}]
# Pass through llama-server connections.[profile.llama-server]forwards=[{host_port=8080}]proxies=[{host="127.0.0.1"}]
What’s left?
The biggest missing piece is teaching the proxy server how to inject real credentials into network connections. This isn’t a new idea. The goal is to provide access to things like GitHub without giving an agent actual credentials.
After that, it’s just packaging everything up nicely and writing some docs, so that other people (or agents) can easily configure it for different purposes.
I continue to be interested in late-2024-era edit completion, the “Fill in the Middle” (FIM) models. You know, what Copilot used to do, back before it started generating “mini diffs.” Why?
The new-school edit completion offered by Copilot and Zed’s Zeta2 actually slows me down. It overlays diffs on my buffer, which is visually disorienting at speed. And it proposes edits further from the current cursor, which take me longer to mentally process. Personally, the new style feels like hunt-and-peck. The older style felt like really fast touch typing.
Mind you, I’m a very specific sort of user. I want to know how my code works. I want my code to be clean. And I can read a half-page code completion in moments, thanks to way too many years of reading PRs.
Initial experiments
All experiments performed in Zed, which does less post-processing of the raw model output than some tools. All evaluations are purely subjective.
New-school models (generating diffs). Zeta2 is honestly pretty underwhelming right now. The completions are very generic. And Zeta2 seems to be bad about taking the context into account. It will complete a function, sure. But I’d swap Zeta2 for late 2024 Copilot in a heartbeat.
Old-school models (FIM, inserting at cursor). Let’s go down the list so far:
ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF:Q8_0: The classic, default choice. This isn’t terrible, and it gives more context-aware completions than Zeta2. But it’s generations old, and I want to know if anything is new and shiny.
mradermacher/Seed-Coder-8B-Base-i1-GGUF:Q6_K. This is the raw base that went into Zeta2, I think? It doesn’t seem to be useful in Zed, because the inserted text feels pretty raw. This might work better in a smarter harness. But I’m dropping it for now.
JetBrains/Mellum-4b-base-gguf:Q8_0. Downloaded, but not yet tested.
unsloth/Qwen3.6-35B-A3B-GGUF:IQ4_XS. This is unexpectedly good! Worth further experimentation.
Refining Qwen3.6 35B A3B: Changing order from PSM to SPM
Qwen typically uses FIM, “Fill in the Middle” completion. This uses 3 magic tokens:
/// Qwen FIM prefix marker.constPRE:&str="<|fim_prefix|>";/// Qwen FIM suffix marker.constSUF:&str="<|fim_suffix|>";/// Qwen FIM middle marker (model generates after this).constMID:&str="<|fim_middle|>";
We have two possible flavors. The original is “PSM” compeletion, “prefix, suffix, middle”:
{PRE}{prefix}{SUF}{suffix}{MID}
But since the prefix grows with each keystroke, we can’t cache the entire message. We could get much better caching with “SPM” order:
{SUF}{suffix}{PRE}{prefix}{MID}
Here, we can cache everything up to the final {MID} character, and resume generation with a longer prefix. Whooo, speed!
But Zed doesn’t support SPM completion, only PSM. So I fired up a copy of Claude Code (as one does), and asked, “Hey, write me a Rust proxy server (using my standard conventions) that intercepts /completion, and translates PSM to SPM please.”
Results: Extremely disappointing. SPM format confuses Qwen3.6 35B A3B pretty badly. But then I thought, “Hey, even if we’re running in /completion mode, this is still an instruction-tuned model. Can we prompt it?” One unscientific tweak later:
You are a code-completion tool. You receive input in
fim_suffix+fim_prefix+fim_middle order, and your job
is to generate what the user would be likely to type
next. When in doubt, keep it short. Think of this like
generating a diff in agentic coding mode. You're trying
to insert the right text to make a working program that
does what the user wants. If there's no obvious next
step, generate nothing.
{SUF}{suffix}{PRE}{prefix}{MID}
This is still pretty bad, but it’s better. You can tell it’s trying to be an SPM autocompleter, though it’s still the worst of the bunch.
Possible next steps:
What if we modify the proxy to transform /completion into a /chat/completions request, with a real prompt, real text inputs, and tool for insert_at_cursor(text)? Can we access more of the model’s intelligence?
Qwen3.6 35B A3B is small enough to fine-tune! We could look up file completion data sets, and try to create a LoRA adapter. We could even use something like tree-sitter to generate custom completion examples. Would that give us something useful?
I also notice that FIM-style models are notoriously bad at choosing a good stopping place. This can be fixed with a lot of regexes. But what if our fine-tuning data took care to demonstrate good stopping places?
About three months ago, I bought the Onyx BOOX 25.3” Mira Pro Color, an e-ink monitor for desktop use. I’ve used it as my primary monitor since, and I’ve had a lot of questions about it. This is my experience report, from the perspective of a working, still mostly typing, programmer.
This is not a sponsored post, and it is not a product review. I wrote a very similar post about the Daylight DC-1 last year.
Neovim in the morning sunlight.
As explained in last year’s post, the reason I persist with these monitors is because it makes me energetic and happy. Sunlight, direct or indirect, helps me stay clear and focused during my workday. I find spaces illuminated by natural light beautiful and inspiring.
I’m not going to recommend that you buy one of these devices. They’re expensive, about $2000, and the experience is quite different from LCD. Even if this looks cool, it seems to me very possible that most people would not like it in practice. With that said, I am happy with it, and I’ll probably keep investing in these tools as they get even better with time.
Spending a workday in the garden.
Using the Mira Pro as a primary monitor is a continuation of the experiments with my e-ink tablets and Termux as coding environments. But now, with far fewer compromises. I’m running my regular NixOS environment on my work laptop. No SSH and tmux needed, no Android terminal emulator to customize.
What I have done, though, is spent quite some time on making my system more suited for this monitor. The Mira Pro does not work well with dark themes. In fact, it only works well with high contrast light themes.
Luckily, I’m bent towards minimalism, so I already used near-monochrome themes, relying more on typographic syntax highlighting rather than coloring. I now have custom themes for Neovim, Zed, and Ghostty with a few vivid colors for things like selection, comments, and constants. Otherwise it’s largely black on white.
It’s trickier with other applications. In Firefox, I’ve started using the high contrast setting. That works pretty much like an inverse of DarkReader. I now run Spotify in the browser in order to avoid its dark theme.
The monitor has a clunky menu system with which you can change rendering modes; things like contrast and speed. I found an open-source reverse-engineered NodeJS package that I use with Hyprland keybindings to easily change rendering modes and manually refresh. No need for the built-in menu.
In practice I use two modes:
Reading:
This mode renders colors most vividly and text sharply, but typing with it is agony. I use it when reading text documents, web pages, or code diffs.
Writing:
This is by far the most commonly used mode, which compromises colors and sharpness for way better latency. I use this for everything in the terminal, chat, general web browsing, and probably most other things not covered by the reading mode.
See the following photos for a close-up comparison:
Reading mode, where colored regions are pretty smooth and text looks sharp.Writing mode, where colored regions (light gray, red, green) are grainy and text is a bit blurry.
What about latency? Here’s the two short clips of me typing with the reading and writing modes:
Reading mode, with horrible latency for typing.Writing mode, with some but acceptable latency.
Ghosting? In my writing mode it’s minimal. It really doesn’t bother me.
About the color panel: I don’t like it very much to be honest. It was the only version of the Mira Pro available from the Swedish retailer at the time, so I went with it. I think I would’ve been happier with a monochrome panel, because the coloring technology makes it considerably darker.
Here’s a comparison between the Palma 2 Pro (using a similar but smaller Kaleido color panel) and my old Tab Ultra (with a monochrome panel):
Color vs Monochrome e-ink panels without backlight.
Unless the room has great diffuse lighting, natural or otherwise, the color panel does require some backlight. In direct sunlight or outdoors it works without. I might spend more time optimizing the lighting in my office to make this work during the winter months.
So, what’s to make of it? Personally, I enjoy using this monitor a lot, even if it’s not perfect. Should you buy an expensive 25” e-ink monitor? I cannot say. But if you do, let me know how it works out.
My custom themes and keybindings can be found here.
This blog is ancient, in blog years. The first post was on June 30, 1998, and it featured a randomized emboss for MathMap. Back in those days, it was a mix of neat little snippets like that and interesting links. The site was a single, hand-edited HTML file in reverse chronological order. It ran on a Linux mini-tower built from parts from the MIT Swapfest, and it lived under my desk.
Google hadn’t been incoporated yet. The Internet bubble was still inflating.
Over the years, the tech stack changed: for a while, this site used SGML-based rendering via a custom script (or was it XML?), then it was a nice interactive Typo site with comments, and then eventually it migrated to the current Jekyll architecture. Which seems to be about 12 years old. I’m pretty proud to have kept nearly all the inbound links working for decades now.
Around 2007 or so, I did a fun series of high effort posts about probability monads. But high-effort posts are a trap. Soon I started feeling like every post ought to be high effort. And then I wrote less and less.
But blogs are a bit of a retro endeavour these days. RSS readers still exist, but I imagine nearly all my subscribers have disappeared since the heady days of 2007. And apparently it’s trendy to work with the garage door up.
So maybe it’s time to get back this site’s roots. I don’t have any MathMap snippets for you today, sadly, because the last release seems to have been in 2004. But here’s a cool trick!
![An animated GIF that shows a screen recording of a terminal session. The terminal is split into three screens. On the left, taking up the majority of the terminal, is a screen labelled 'otel-tui'. On the right, split vertically, are two screens labelled 'oddball' and 'eventlog-live-otlp'. The 'otel-tui' screen is itself split into three subscreens. A narrow column on the left, labelled 'Metrics', lists all incoming metrics by name. These names correspond to the names described under Heap Profile by Closure Type, except formatted in camel-case and prefixed with 'ghc_eventlog_'. Currently, the metric named 'ghc_eventlog_HeapLive' is selected. The wide column on the right is split vertically. The top row, labelled 'Details', shows the exact structure of the selected OTLP message for the selected metric. The bottom row, labelled 'Chart', shows a text rendering of a line chart that shows the values of the 'ghc_eventlog_HeapLive' metric. This shows a spiky line, which jumps between 10-30MB, though the Y-axis is labelled in bytes rather than megabytes. The 'oddball' screen shows the output of the oddball program, which consists of many lines saying 'Generating X random numbers' and 'Sum: Y' for ever-changing values of X and Y. The 'eventlog-live-otlp' screen shows the debug output of Eventlog Live, which consists of many lines saying '[DEBUG] Received X events.', '[DEBUG] Exported Y metrics.', and '[DEBUG] Exported Z logs.', for ever-changing values of X, Y, and Z.](https://well-typed.com/blog/aux/images/real-time-telemetry-with-eventlog-live/oddball-with-otel-tui-2026-09-22.webp)
which we write as 
and so
, 
and 
.
but we can rescale at any time.)
(a tenth of a full turn). If the shorter edges are of length 1, then the longer edges are of length 
is the golden ratio.
times the scale of the former, to reflect the change in scale.
for a function 
is a point 
such that 
. With this definition, a periodic point of period 
is also periodic of period 
for every 
for every 
from 1 to 
, we say that 
be an interval and 
a continuous function. If 
has a point of least period 3, then it has points of arbitrary least period; in particular, it has a fixed point.
being open or closed, bounded or unbounded.![I=[a,b]](https://s0.wp.com/latex.php?latex=I%3D%5Ba%2Cb%5D&bg=ffffff&fg=333333&s=0&c=20201002)
be a compact interval of the real line and 
it is 
. Then 
be the minimum and the maximum of 
, it is 
and 
. Choose 
such that 
and 
. Then 
is nonpositive at 
and nonnegative at 
. By the intermediate value theorem applied to 
, 
and 
, which is a subset of 
be a closed and bounded interval contained in 
. Then there exists a closed and bounded subinterval 
.![K=[c,d]](https://s0.wp.com/latex.php?latex=K%3D%5Bc%2Cd%5D&bg=ffffff&fg=333333&s=0&c=20201002)
. We may suppose 
, otherwise the statement is trivial. Let 
be the largest such that 
. Two cases are possible.![x\in(u,b]](https://s0.wp.com/latex.php?latex=x%5Cin%28u%2Cb%5D&bg=ffffff&fg=333333&s=0&c=20201002)
such that 
. Let 
, and let ![J=[u,v]](https://s0.wp.com/latex.php?latex=J%3D%5Bu%2Cv%5D&bg=ffffff&fg=333333&s=0&c=20201002)
. Then surely 
, but if for some 
we had either 
or 
, then by the intermediate value theorem, for some 
 we would also have either 
or 
, against our choice of 
for every 
be the largest ![x\in[a,u]](https://s0.wp.com/latex.php?latex=x%5Cin%5Ba%2Cu%5D&bg=ffffff&fg=333333&s=0&c=20201002)
such that ![J=[w,u]](https://s0.wp.com/latex.php?latex=J%3D%5Bw%2Cu%5D&bg=ffffff&fg=333333&s=0&c=20201002)
. Then 
be such that 
, 
, and 
. Up to cycling between these three values and replacing 
with 
, we may suppose 
. Fix a positive integer 
such that 
and 
for every 
.![L=[a,b]](https://s0.wp.com/latex.php?latex=L%3D%5Ba%2Cb%5D&bg=ffffff&fg=333333&s=0&c=20201002)
and ![R=[b,c]](https://s0.wp.com/latex.php?latex=R%3D%5Bb%2Cc%5D&bg=ffffff&fg=333333&s=0&c=20201002)
be the “left” and “right” side of the closed and bounded interval ![[a,c]](https://s0.wp.com/latex.php?latex=%5Ba%2Cc%5D&bg=ffffff&fg=333333&s=0&c=20201002)
: then 
and 
by the intermediate value theorem. In particular, 
, and Lemma 1 immediately tells us that 
in 
. Also, 
, so 
, again by Lemma 1: call it 
. This point 
, but 
which has period 3. As we can obviously take 
, we only need to consider the case 
.
of 
. In turn, as 
, there also exists a closed and bounded subinterval 
of 
, again by Lemma 2: but then, 
. By iterating the procedure, we find a sequence of closed and bounded intervals 
such that, for every 
, 
and 
.
and recall that 
in the role of 
as a closed and bounded subinterval not of 
. In turn, as 
of 
. Following the chain of inclusions we obtain 
. By Lemma 1, 
has a fixed point 
has least period 
, then so has 
, and in addition 
while 
for every ![i\in[2:n]](https://s0.wp.com/latex.php?latex=i%5Cin%5B2%3An%5D&bg=ffffff&fg=333333&s=0&c=20201002)
. Consequently, if 
, then 
. But this is impossible, because 
by construction as 
. 
between positive integers is defined as follows:
, with 
.
in lexicographic order.
and 
for every 
.
.
—i.e., the powers of 2—in reverse order.
be a continuous function.
, then 
integer it is possible to choose 
. In particular, there are functions whose only periodic points are fixed.
