The Pragmatic Open Source Contributor

2025-03-02T00:00:00-06:00

I sometimes have the feeling that salaried programmers shy away from fixing or extending open source code due to some combination of the following:

They don’t think it’s part of their job.
Their employer doesn’t think it’s part of their job, and/or does not have the necessary legal infrastructure and will.
They’ve been frustrated in the past with rejected or ignored patch requests.
They are worried about the time committment for the entire process.

These are understandable barriers that I would like to break down. More broadly speaking, I have two aims for this guide.

First, I hope to encourage fostering the shared knowledge that open source software represents. Especially if you work in larger-scale environments or are early adopters of promising new technologies, you have a great opportunity to spot issues and areas of improvement that can benefit everyone.

Second, I aim to provide a playbook for how to successfully do this kind of work and set expectations for what you can expect and what will be expected of you. I have made contributions to many projects over the years, generally following a pattern of identifying some code that solves 95% of my problem and contributing the 5% delta back. I’ve also worked more extensively in a single open source community, OpenStack, where public contributions and distributed coordination are the status quo.

The Pragmatic Contributor

The community guide to open source succinctly summarizes the benefits of contributing back to open source projects. Still, I find that these guides frame the process as motivated by individual incentives, e.g., personal development or a sense of fulfillment. For the pragmatic contributor, one motivation trumps all: the drive to improve software the business relies on. You want to fix a tool that isn’t enabling you to do the best job you can.

This mindset influences all aspects of the contribution process. If you’re mostly interested in learning something new or making connections, you can pick and choose what projects you want to contribute to, and what problems you’d like to pick off the bug tracker. A pragmatic contributor never tries to find problems; they find you. Maybe it’s a newer project and hasn’t fleshed out all the little details, e.g., support for multiple authentication methods. Or, perhaps you simply found what seems like a bug in the behavior, or an inefficiency when used with high throughput.

A pragmatic contributor also pressure-tests the solution. It’s likely you ran in to some problem nobody else has seen. Do you really need to implement a fix in the open source layer, or is it fair to say your application is just behaving weirdly? What is the wider benefit, really, of contributing this feature back to the community? Software naturally wants to expand in surface area and complexity over time. Some maintainers rule with an iron fist to keep the scope of their code low and steady, others are more willing to give you the benefit of the doubt that expanding scope is going to make things better. Over time I’ve come to appreciate the wisdom of the first approach, though it introduces challenges for you as an outsider. In either case, I have found that a respect for the maintainer’s view (it is their code you’ve been happily using, after all) and a willingness to find the most elegant solution goes a long way.

The Pragmatic Patch Playbook

Once you’ve identified a potential contribution, here are some general steps I like to follow.

Step 0: Talk to legal

Before you do anything else, get approval from your legal department. Even if you’re contributing to a project that does not have a Contributor License Agreement (CLA), you have a duty to ensure you are not adding risk to your company or yourself. Ideally, this conversation results in approval to sign any reviewed CLAs; Company CLAs (CCLA), if supported by the project, can enable self-servicing of licenses to new members of your team, and are much more future-proof than individual CLAs tied to one employee.

This can be a bit of an uphill battle, but in my experience it is mostly a battle of time. You might need some patience to walk all the stakeholders through the business case for contributing back to open source, and any possible risks. I usually lean on the following argument:

We currently use open source system X, and it provides business value through capabilities and cost-efficiency, i.e., it’s usually free—“as in beer.”
Yet, it can’t handle some new business use-case without modification.
Modification effort is small relative to working around the constraint.
We do not need to and will not expose proprietary code.
Privately adapting the code (forking) introduces long-term maintenance burden and adds risk. It’s likely X will be changed in the future in a way that requires significant rework of our adaptations and thus blocks us from performing security upgrades.
Publicly adapting the code reduces this risk and (especially if X is well-known in the industry) can even serve to attract new talent by increasing the company’s visibility.
Contributing to X is thus in the business’s best interest.

I have never seen this argument fail given enough motivation, but I’m sure there are exceptions. One thing to remember is you’re not arguing for actually doing the work or estimating the time commitment or return on investment; you’re making a case for the option. You should however be prepared to give concrete examples of the type of contributions you might make.

Step 1: Get the lay of the land

I see developers often skip this step and go straight to submitting a patch. This can lead to frustration for both you and the maintainers. Doing a quick check of any defined contribution process and putting yourself in the maintainers’ shoes often prevents these issues.

What is the contribution process? Do you need to sign a CLA first? Are pull requests welcome on the repo? Can you find examples of contributions from the outside? How did the maintainers respond to the contribution request? Anything you can learn from this?

See also How to Contribute to Open Source: Orienting Yourself to a New Project

How active are the maintainers? Are they reviewing patches on a daily basis, or does it seem more sporadic? Can you notice any patterns in how and when they respond to queries? Is there a shared maintainer model or is there a single owner? If shared, who seems to be the most active recently?

How long will this realistically take? What is the latency between the time a pull request is open and it is merged? How much of that is waiting for the patch author versus feedback from maintainers? How many patches do you think you’ll need to do, and do they need to be done serially? From this, you can usually get a ballpark estimate, but I also have a heuristic: expect two weeks to one month for a bugfix to land, and three months to a year for major feature work. Much of that depends on how much of your attention you give to tending to the process.

Step 2: Get maintainer buy-in

For small changes, you can usually skip this step, but if you’re thinking of making any significant changes to the codebase, investing time here will make the entire process much smoother. Your objective is to identify at least one maintainer who will help champion your change.

Meet the maintainers where they are. Do they have a Slack channel they use to coordinate changes and share info? A bi-weekly special interest group (SIG) meeting? A mailing list? Figure out what their preferred communication method is and then introduce yourself. Give a bit of background on who you are and why you’re interested in contributing to the project, and what problem you’re trying to solve.

Follow the formal proposal process. If the project uses a proposal system (e.g., the Kubernetes Enhancement Proposal [KEP]), learn about how to submit a proposal. You can either talk to maintainers before making a proposal, or notify them after you have submitted the proposal. I still think a “warm handoff” is important here, to actually reach out in person to the maintainers to let them know that you’re open to discussion and are serious about embarking on the process of making a significant contribution.

Agree on scope and keep a paper trail. If in your conversations with the maintainers you arrive at agreement on what the scope of your contribution will be, and, importantly, what can be considered out of scope, make sure that you write that down somewhere publicly. This can be helpful to give context to other maintainers who might be reviewing the work down the line. Generally, this information should also be in a proposal document, if that process exists, but it’s still useful to have other records.

For this reason, I also like to have conversations about proposals in, e.g., public Slack channels as opposed to private messages. You can hash out details in private, but then post a summary of the conversation in Slack to preserve a record.

Step 3: Do the work

You may have noticed that we haven’t written any code yet. Guess what? This is the only step where we’re going to talk about code! The lion’s share of open source work is communication. That said, there are some general rules that in my experience improve the outcome of patch requests.

Don’t be afraid to fork. Some people recoil at the word “fork.” Personally, when I’m working on open source contributions, I always fork the project, make the patches there, and use the fork internally for a while. You’re making a trade-off between speed of delivery (leveraging your patch immediately) and maintenance burden (carrying your fork through upgrades); in my workplaces the former usually trumps the latter, depending on how often you expect to pull in upstream changes. Forking internally also lets you battle-test your changes in a real environment before you submit the patches upstream. I often find bugs this way, or can correct mistaken assumptions about how something works in practice.

Work backwards from your specific desired outcome to a generic mechanism that helps achieve that outcome (and perhaps others.) For example, in this old webpack patch, what I wanted was a way to put a Git commit SHA in the name of files built by webpack. Rather than code this case explicitly, I proposed a way to enable plugins to provide support for new filename pattern placeholders. This enabled me to handle my needs in a separate plugin, and appears to have been useful to others over the years.

Add tests! If you’ve found a bug in some code, it probably means there wasn’t a good-enough test for that behavior. Add a test that fails without your patch and succeeds with your patch. If you’re adding new functionality, make sure you have good coverage. The maintainers will ultimately be on the hook for bugs in your code, and your job is to reduce that burden as much as you can. It sometimes happens that there is not appropriate test infrastructure to express the tests you need. In that case, reach out to the maintainers to ask their opinion on how to proceed; often, they will be okay with less test coverage as they are working on figuring out a generic solution for testing that aspect of the code.

Keep every patch to one atomic change. The definition of “change” here is open to interpretation. If you are working extensively in an open source project and have a large feature you’re implementing that touches many areas of the code, you should probably break up each piece of the implementation into a single patch that targets a subset of the codebase. This works best if the broader context of your work is known and formally tracked; not all projects have this infrastructure. I like to keep my patches scoped to minimize context overhead for the reviewer. For example, when working on a larger feature, I first identified one (rather large) refactor I could do that would make implementing the feature easier. I submitted one patch for that change, and then one patch for the minimal feature implementation.

If your atomic change is still large, break it into iterative commits. In the latter example patch, I broke it into several commits to make it easier to review and see the thought process. I could have broken those commits into separate pull requests, but it seemed to me to reduce cognitive overhead (for the reviewers) when everything was in a single pull request that could be referred to and iterated upon. You can always break it into separate requests later if you have the commits structured this way.

Keep refactoring to a minimum. You may be tempted to “clean up” other areas of the code not specifically related to your code. It may be difficult, but you should try not to give in to these impulses. Any unnecessary refactoring, especially when it concerns readability or styling of code, adds to the work the reviewer must do and disrupts the message of what you’re actually trying to achieve with your patch. Reduce refactoring to only what is necessary to make your change possible. Later, you can come back and do the refactor if you want. Patch requests are also a good opportunity to ask the maintainers how they would feel about such a refactor in the future, enabling you to get some early buy-in.

Preserve backwards-compatibility. Open source projects are widely consumed and you cannot know all of the ways in which it’s being used today. You have probably been burned at some point by a library changing default behavior or its API surface without a major version bump. Major versions are big steps for an open source codebase and if you are tying your change to a breaking-change release, it will increase the latency of your change being available drastically. As such, work to ensure that whatever you do does not break existing behavior.

Step 4: Do the other work

This is probably the part of the process that developers like the least, but you should plan to spend some time here to have a high-quality contribution.

Write good documentation. You should write docs for any new capabilities you are adding to the project. Sometimes you need to add an entirely new section of documentation! I don’t think I need to argue for the benefits of documentation, and how frustrating it can be when documentation is sorely lacking. If this is a task that is particularly difficult for you, I’m guessing that LLMs are probably pretty good at writing docs these days, and could be a good tool for summarizing how a feature works in a more consumable format. I haven’t tried this myself yet and still write all documentation by hand.

See also these useful resources from WriteTheDocs.

Examples are also documentation. Usually, technical documentation for open source code is of the “technical reference” variety. However, that is only one type of documentation. Examples add more of a “how to” flavor, and sometimes showing really is better than telling. If you’re adding another feature to the code, you should provide some examples of how to use it.

Step 5: Finish line

Once you have all the code and non-code pieces assembled, and have checked all the other required boxes in the process, you’re ready to submit your patch! This can feel a bit like “hurry up and wait,” especially if you’re been working on a patch to deliver something else as part of your job. In my experience, patience is important here. Just because it’s a good time for you to iterate on the patch (because you’ve built up all the context) doesn’t mean it’s the same story for the maintainers. Here are some tips for how to navigate this time.

Proactively reach out to the maintainers. Use those channels and connections you established in Step 2 if it’s a bigger change. Maintainers may want to discuss your change formally as part of their own processes, and giving them a heads-up that your patch is ready for review can help them figure out how to prioritize it. For smaller changes, I don’t usually do this, as I think it’s a bit annoying to just ping maintainers for something they essentially were already notified about via the patch request itself.

Politely check in periodically to raise visibility. If I’m having trouble getting any eyeballs on a patch request, I usually wait a week or two and then nicely ping some maintainers and request they please have a look-see at the patch, or if there is anything else that I should do. If that doesn’t work the first time, I’ll do it again in another week or so. If a month or so goes by, I will start to get more creative and see if I can reach them on another appropriate public social channel. Always be polite, and definitely never be pushy; even though you think you’re helping the project, you’re also taking the time and attention of maintainers.

As soon as you get attention from the maintainers on your patch, leap into action. Especially for larger requests, if you get a first-pass review of your code, try to respond to feedback within a day or so. This keeps the patch conversation pretty fresh for everybody. The maintainer just spent time building up context on your work by reviewing your code and it’s my experience that they appreciate quick follow-through on comments. This can be difficult to balance depending on your own job commitments. I try to set expectations that when a patch is in the final stages, I should have some work time set aside as a buffer.

Step 6: Tie it off

If you made it here, your patch was accepted upstream! There’s always a great feeling (of relief?) when you see that merge complete successfully. This doesn’t always happen—I have some pull requests that have been sitting for years collecting a trickle of sad “+1” comments. As a pragmatic contributor, this isn’t such a big deal, as it usually indicates the code has a low rate of evolution and therefore it’s not too much work to maintain your own fork.

Anyways, there are a few final things I usually do once crossing the finish line.

Thank the maintainers who reviewed your code. Being a maintainer is often a thankless task, so I find a sincere thank-you goes a long way. I often learn a lot from code reviews with open source maintainers, and view that as a gift. You can just leave a thank-you on your patch conversation, or reach out directly, whichever seems more appropriate given your past communication with them.

Reduce your bus factor. Did you start work on a longer set of related features? If so, make sure you clearly document what the next steps are for the work somewhere. Lots of things could happen. You could get involved in other work commitments that take up all your attention, you could be laid off, your company could stop using the software you patched altogether. Still, somebody might want to come along later to finish what you started. It should be possible to do that without your involvement.

Bring your patches back internally. If you’ve been working on a private fork of the code, work to bring your changes back in to your fork. Maybe you’ve even made it possible to stop using your fork entirely!

Conclusion

Let’s revisit some of the reasons I suggested prevent folks from contributing to open source:

They don’t think it’s part of their job. Hopefully I’ve made a brief but decent case for why this is important, both for the wider community, and for your own growth. Familiarity and confidence in this process empowers you to blast through technical barriers, as you might no longer be “blocked” from achieving your goals due to some underlying third-party code not supporting XYZ.

Their employer think it’s part of their job, and/or does not have the necessary legal infrastructure and will. Step 0 describes ways you can try to make the case for contributions. Ultimately this may still be a barrier in practice, but I think it’s worth poking at assumptions here. Sometimes you can be surprised by how open-minded your employer is about work like this.

They’ve been frustrated in the past with rejected or ignored patch requests. Steps 1 and 2 should put you in a better position to set your own expectations and improve the timelines by having better relationships with folks you’re dependent on.

They are worried about the time committment for the entire process. This is the wisest objection to the whole endeavor, in my opinion. It’s hopefully clear that writing the code is a very small part of this entire process. Still, I do find that this gets easier the more experience you have doing it, because you know more about how to keep the ball rolling, and if you’re working within the same ecosystem, over time you should gain more trust from the maintainers, which helps greatly with future contributions.

Write the change you want to see in the world!

K3S on Balena

2021-12-04T00:00:00-06:00

Balena is a software company that specializes in creating solutions for the IoT space. Their flagship product balenaCloud is a fleet management platform that solves a lot of the pain points in an IoT deployment, namely host OS updates, application deployment workflows, and making scaling more manageable. The cloud product is free for up to 10 devices and a single user, making it suitable for a small home lab, but much of the core technology is also open-sourced as openBalena. In my view the device enrollment workflow is stellar: you download an image specific to your target device platform, flash it on an SD card, and then boot up your device, which automatically registers with the fleet and starts running the target application.

The Balena application/deployment model is quite simple:

A Fleet is a collection of some devices.
A device always belongs to exactly one Fleet.
Every device in a Fleet runs some release of a shared application configuration.
The application is either expressed as a single Docker container, or a docker-compose environment with multiple containers and potentially storage volumes.

Balena as undercloud

A consequence of the single-application-per-device model is that it is difficult and/or clunky to divide some set of applications across some set of devices, and as such devices can often be underutilized. For example, in a home lab, you may wish to run an IDS, a PiHole, a Plex server, and maybe some other small applications. In the Balena model, you would either have to split each application out, create a Fleet for each, and assign a whole device to each application, or somehow bundle all the applications in a docker-compose environment, using environment variable switches or similar to control which services should be enabled/disabled on a given host device.

One solution to this problem is to shift the orchestration of applications from Balena to a higher layer, i.e., instead of deploying your target applications directly to Balena, you instead deploy a way to run applications to a single Balena fleet, and then you interact with that system to configure and deploy your target applications. In this architecture, Balena acts as a sort of “undercloud”: you’re using Balena’s cloud platform to bootstrap your own local cloud, combining advantages of each.

I was interested in seeing to what extent this was possible on Balena, and so spent some time evaluating the complexities of packaging something like K8s as a Balena application. The desired end state would be, you can flash all of your devices (Raspberry Pis, Jetson Nanos, etc. – a wide variety of SBCs are supported) with the same base image, and then build one Balena application for your K8S cluster and deploy to all devices simultaneously. Balena will then handle host OS updates for you and any changes you make to the K8S environment (e.g., adding Tailscale integration or something) will be rolled out across the fleet uniformly by default.

Packaging challenges

K3s was the obvious choice for the K8s distribution, as it’s designed specifically with the edge use-case in mind: it has a smaller footprint, uses websockets to tunnel the kubelet API (meaning the kubelet could be running behind several firewalls or NAT layers) and at this point is pretty battle-tested on commodity SBCs such as Raspberry Pi.

Adapting K3s to run on top of Balena was another matter. BalenaOS deploys applications as Docker containers, but it’s not Docker precisely–rather, Balena’s own balenaEngine, which is a fork of moby, the container framework that provides the base for much of Docker itself. So we need to accomplish a few things:

The K3s control plane (API server) needs to be able to run in a container on Balena Engine.
Container storage created by K3s should be allocated from outside the K3s container’s overlay file system. I’ll go in to why.
Any cgroups applied to K3s containers should use the same cgroup manager as BalenaOS. I’ll also go into why this is important.

Besides these challenges, I think it is useful to call out a few non-goals:

The K3s container should run rootless, i.e., not in “privileged” mode. While this may be simpler to do in the future, it is not feasible at the moment in a container context.
All possible container configurations will work! This will likely not be the case, as we must run the K3s control plane in a container in order to deploy it via Balena. In particular, any containers running on K3s in privileged mode will have a slightly different view of things than if you were running them on K3s on the host directly. While this may not affect anything in practice, in theory it could.

Let’s look at each challenge individually.

Running K3S in a container

The principle challenge was how to run K3s itself inside a container. There is some prior art on this, namely the K3d project, which helps developers spin up K3s “clusters” as Docker containers. However, because it’s use-case is K3s development, not running production workloads, K3d wants to be in charge of creating and configuring the cluster and its containers, but Balena requires that we specify this explicitly ourselves as part of the docker-compose declaration. Importantly, it’s not possible to add a new node to a cluster. So, it is out, but we can learn some of the requirements of running K3s in a container by examining its source code, where we for instance learn that K3s server/agent containers must be run in privileged mode.

Separately we need to build our container image. One of the nice things about Balena is their tooling for this. With the understanding that you’re trying to build one application for deployment to potentially many different devices, you can express your Dockerfile as a template and use placeholders to specify which base image you are building from for a given target platform. This prevents a lot of boilerplate as the number of supported platforms increases.

Solutions:

Install K3s in a container image.
Configure the K3s container to run in privileged mode, mostly so that /dev devices are accessible, both from the K3s control plane and so that they can be passed to pods running on K3s.
Similarly, configure the container to run in the host networking stack. This helps most CNIs (Container Network Interfaces) to function as normal.

The right filesystem, at the right time

A Docker image consists of layers of deltas applied to the layer below. This is one of the reasons why Docker containers can be quite size efficient if managed well: if several containers share a common ancestor, there is only 1 copy of that ancestor’s layer(s) on disk. However, a consequence of this design is that when a container starts from an image, the container’s filesystem inherits this layering property: any operations done inside the container happen in a new layer (how exactly this works depends on the overlay file system in use.)

The primary implication here is that if your container is doing a lot of i/o against its default overlay filesystem, it will likely either (a) balloon in size over time or (b) perform so many writes/deletes that your storage on-device (likely, SD card) will get hit pretty hard. The reason for this is that edits to any particular file will copy the entire file from the lower layer to the upper layer and then perform the edit; particularly for large files, this can be a problem.

Fortunately, Docker volumes solve this problem by mounting a separate path as a different filesystem. Balena supports allocating Docker volumes and attaching them to your application containers; they will be formated with ext4 (for our purposes, just important that it’s not an overlay filesystem.) Additionally, it is possible to mount tmpfs volumes, which importantly are volumes that exist only in memory and not on disk. This means they are by definition not persisted, but are great for reducing i/o to the underlying storage medium.

So what directories/paths does K3s need to do its work? It turns out there are two main classes: runtime (ephemeral) and data (persistent). The runtime directory is used mostly by the container runtime, in our case containerd. This includes the gRPC sockets as well as a state file that holds information about the running containers; when the k3s container is stopped, all of the spawned container processes will also exit, and they must be re-created. As such, this “state” file can be considered ephemeral as it is ultimately tied to the lifecycle of the kubelet process.

The data directory on the other hand stores all of the persisted about the containers, including container images, volumes, and the containers’ overlay file systems once created. Importantly, the overlay file systems are created from the non-overlay Docker volume mount passed to K3s via Balena. This critically prevents an overlay-in-overlay situation for K3s pods. While it’s possible to nest overlay filesystems in this way up to a hard limit of 2 levels, it adds nothing but complexity here.

For K3s, the runtime directory is in /run, which is typical, and the data directory defaults to /var/lib/rancher/k3s. Knowing this, we can construct our Balena app to have this configuration:

- tmpfs:
    - /run
    # This is usually symlinked, but symlink behavior can be odd when mounts are
    # involved (citation needed), so we just additionally explicitly mount it here.
    - /var/run
- volumes:
    - k3s_datadir:/var/lib/rancher/k3s

Why not use the Balena socket?

Readers accustomed to Balena may be aware that it’s possible to add a special label to your Balena app container to request that the balenaEngine socket be mounted inside the container and its path stored in $DOCKER_HOST. It should theoretically be possible for this to work by doing the following:

At K3s kubelet container start, symlink the socket to /var/run/docker.sock, where K3s expects to find it (the path is not currently configurable.)
Run K3s with the --docker flag to tell it to use a “dockershim” socket interface rather than the default containerd interface. Docker’s socket is not quite compatible with the K8s CRI (Container Runtime Interface), so K8s historically supported an adapter interface to bridge the gap (while this is officially deprecated, there should still be support for this method, albeit no longer by K8s core.)

There are two reasons I found for why this cannot work for K3s on Balena:

Balena tries to bind-mount the socket at /run, which we already configured as a tmpfs mount. The mount gets screwed up b/c it’s trying to mount over this tmpfs.
It turns out that when K3s uses the Docker driver it expects to be able to introspect the raw filesystem backing the container state. This appears to happen when it tries to allocate the container’s cgroup:

 k3s  E1213 01:22:12.729919      60 manager.go:1123] Failed to create existing container: /system.slice/docker-71d337540e5cd588869350a52a524e9b6e891cc6b5448011fd96a35c21d93530.scope: failed to identify the read-write layer ID for container "71d337540e5cd588869350a52a524e9b6e891cc6b5448011fd96a35c21d93530". - open /var/lib/docker/image/aufs/layerdb/mounts/71d337540e5cd588869350a52a524e9b6e891cc6b5448011fd96a35c21d93530/mount-id: no such file or directory

So K3s actually needs access to /var/lib/docker, which (a) does not exist for Balena (it is actually /var/lib/balena) and (b) cannot be bind-mounted as Balena does not support arbitrary bind-mounts from the host.

Consequences of NOT using the Balena socket

For our purposes, there are really not many benefits to re-using the Balena socket, except if it’s important that host bind-mounts be used: if K3s can’t spawn containers via the socket, it really has no way to provide bind-mounts from the host, because it does not have access to the host filesystem otherwise.

As already mentioned, Balena doesn’t support arbitrary bind-mounts from the host, so this is a moot point in this context. Device mounts (/dev) on the other hand should still work because our K3s container runs in privileged mode.

While Balena Engine has some nice features, like pulling container deltas and limiting memory usage during image pulls (to prevent page thrashing), containerd does a good enough job of replacing the other benefits of balenaEngine (small binary size and decompress-on-pull to avoid excessive disk i/o).

K3S cgroups on a systemd system

BalenaOS uses systemd as the init system (as do many[/most?] Linux distributions now.) This is important because systemd effectively takes control of cgroups itself and organizes them in a specific way. What is a cgroup?, you ask? It is just a way of letting “whoever is in charge” understand what system resources your process is allowed to consume; it is a way of ensuring the total load of the system is not exceeded, and enough headroom is given to this process or that.

K3s by default uses the “cgroupfs” cgroup driver instead, which manages cgroups via access to the /sys/fs filesystem.

Going down a rabbit hole

Under normal circumstances, using the cgroupfs driver in a systemd context means there are effectively two systems with different views of the total resource consumption on the system. My first thought was to try to configure K3s to use the systemd cgroup driver when running inside Balena. However, if you do some searching as I did, you will find this comment by a K3s maintainer, explaining why this is not possible:

systemd cgroup driver is not supported because systemd will not allow statically linked binaries (which k3s is built on). The cgroups manager code needs something from systemd CGO so we have to disable it.

I admit I still don’t really know how to parse this. systemd itself cannot (easily?) be statically linked due to some of its architectural dependencies. But it is not clear to me what cgo has to do with this.

In any event, as far as I can tell, the systemd cgroup driver does indeed work with K3s, but it must be enabled on the K3s kubelet agent with --kubelet-arg=cgroup-driver=systemd. Now, using this driver on the kubelet agent (which, again, is running in our K3s container!) does come with a pretty harsh consequence: we have to be running systemd in the container! Not only that, it needs access to the D-Bus in order to maintain some cohesion with the host systemd! Fortunately the latter can be done with an additional Balena container label. I accomplished the former by stealing a lot of code from this example project published by Balena.

I was able to get K3s going, but there were still issues launching pods; it could not allocate the cgroup. After a lot of searching and debugging, I realized that actually, it worked fine to use the “cgroupfs” driver. But why?

Host OS systemd-cgls memory:

  │ │     └─balena.service
  │ │       ├─1804707 containerd
  │ │       ├─1804896 /var/lib/rancher/k3s/data/86a8c46cd5fe617d1c1c90d80222fa4b7e04e7da9b3caace8af4daf90fc5a699/bin/containerd-shim->
  │ │       ├─1806414 /var/lib/rancher/k3s/data/86a8c46cd5fe617d1c1c90d80222fa4b7e04e7da9b3caace8af4daf90fc5a699/bin/containerd-shim->
  │ │       └─1806511 /var/lib/rancher/k3s/data/86a8c46cd5fe617d1c1c90d80222fa4b7e04e7da9b3caace8af4daf90fc5a699/bin/containerd-shim->

K3s container systemd-cgls memory:

  └─balena.service
    ├─ 82 containerd
    ├─209 /var/lib/rancher/k3s/data/86a8c46cd5fe617d1c1c90d80222fa4b7e04e7da9b3caace8af4daf90fc5a699/bin/containerd-shim-runc-v2 -nam>
    ├─717 /var/lib/rancher/k3s/data/86a8c46cd5fe617d1c1c90d80222fa4b7e04e7da9b3caace8af4daf90fc5a699/bin/containerd-shim-runc-v2 -nam>
    └─792 /var/lib/rancher/k3s/data/86a8c46cd5fe617d1c1c90d80222fa4b7e04e7da9b3caace8af4daf90fc5a699/bin/containerd-shim-runc-v2 -nam>

The pids are different but they are organized and visible in the same way, at least.

Solutions:

Use the default “cgroupfs” driver for K3s, which will happily write to the unified cgroup it sees there. Systemd is OK with this.
Set the limits on the K3s container as high as possible to give it the maximum amount of resources so it can pass those resources to pods.
Do NOT bind-mount the /sys/fs filesystem into the container; this creates the “split-brain” situation between systemd and cgroupfs, and it seems there are some inconsistencies between the state of the filesystem. I think that /proc mounting is also required, and while this is possible in Balena, it adds additional problems, as systemd then sees that another systemd is running as the init process, etc. It’s just a mess this way.

Calico and FlexVol

What? We’re making this even more complicated? Well, yes. I wanted to run Calico as the CNI plugin to provide networking as opposed to the default flanneld plugin. I believe this is optional, though I did have some issues running flanneld due to the default MTU it tries to set on the interfaces it creates and manages (it is higher than the MTU on the interface Balena configures); this may be configurable in flanneld but I did not investigate, because I was interested in running Calico for the additional capabilities it can provide.

The main issue I encountered with Calico+K3s is that it uses a FlexVol driver, which is an earlier solution for the problem that was ultimately solved by the K8s CSI (Container Storage Interface.) FlexVols provide a general-purpose way to mount volumes into pods. Calico uses this mechanism to mount a socket into the pod, which is used to coordinate communication b/w the Calido DaemonSet and its node pod. To be honest, I am not quite sure exactly why this is needed, or what it does. But the main issue was that Calico needed to write this socket file to a place on the container’s filesystem, and that place was being bind-mounted from a different path inside the K3s container that was not backed by a Docker volume mount provided by Balena.

The solution is to create a volume mount just for this and tell Balena about it:

volumes:
  # .. (other volumes)
  - k3s_flexvol:/opt/libexec/kubernetes/kubelet-plugins/volume/exec

I also updated the K3s agent to point to this location instead of its default, rather than tempting fate and trying to add the Docker volume at the default path: --kubelet-arg=volume-plugin-dir=/opt/libexec/kubernetes/kubelet-plugins/volume/exec.

Another problem I encountered was that Calico by default, if you’re using their new “operator”-based deployment, will configure the CNI to use VxLAN for some pieces. IPIP is the usual default so I’m not sure why it changed in the operator migration, but I recall there were similar MTU issues with this setup. You can change this default by updating the Calico “installation” resource’s IPPools to use IPIP encapsulation, if you also encounter this (it’s supposed to be the default acording to the docs.)

Solutions:

Add an additional volume mount for the FlexVol installation path and configure K3s to use it.
Ensure the IPPools for Calico use IPIP encapsulation, or tweak flanneld’s MTUs to align with the values Balena configures on the host network links.

Cluster contextualization

As discussed, Balena’s deployment model assumes that a single application is deployed to all devices in a given fleet. Yet, in a K3s cluster, there are effectively two applications we need: one for the K3s API server, which drives the state of the cluster, and one for the kubelet agents, which create and manage the pods. A K3s cluster must initially start with an API server process. When the server bootstraps, it creates a node enrollment token, which the kubelets use to authorize themselves to join the cluster. This process of the cluster attaining this state, where the nodes have identified eachother and agreed on respective roles, is called contextualization.¹

This could be done in any number of ways, but I wanted to see how much of this could reasonably be automated.

The approach I landed on leverages the fact that you can specify an additional label io.balena.features.balena-api to have a Balena API token available in the container environment. We can use this token to update device variables to auto-configure the cluster. It works like this:

First device enrollment: Device A is enrolled to the fleet; it is the first device. The first application container to start is the k3s_context container, which runs a script to determine what role the device will have. The script pulls a list of all devices in the fleet. If it is the first device, it sets a device variable K3S_ROLE=server on itself. The k3s_context container then goes into a wait loop, where it periodically wakes up to see if the server has started properly. If so, it writes two fleet device variables K3S_URL=... (which points to Device A’s IP) and K3S_TOKEN=... (which has the value of the enroll token.) The enroll token is normally written to disk by the k3s container to its data directory; because this is backed by a Docker volume, we can share the volume with the context container, so it can read it easily.

Subsequent device enrollment: Device B is enrolled to the fleet. Again, the context container starts first. But, it sees that there is another device already in the fleet, which means the server must exits. So it sets K3S_ROLE=agent on itself; the API server URL and enroll token are already available to it because they are fleet variables, which apply to all devices as defaults.

Concluding thoughts

Turns out, it’s indeed possible to run K3s on Balena! The container must be set up in a particular way such that it can provide filesystems properly to the pods launched there. More advanced CNIs like Calico can also be used. Overall, this combination has worked out well for me thusfar and I already have plans on extending the configuration to support more advanced networking capabilities. I have even been able to access peripherals such as the Raspberry Pi Camera Module from a pod launched on K3s this way. Pretty cool!

I have not seen this word used much, and I think Kate Keahey is the originator of its meaning. I haven’t found another word that describes this process. ↩

Keystone federation++

2021-07-17T00:00:00-05:00

At Chameleon, I help develop and operate a series of OpenStack cloud deployments, which have been modded to serve as a powerful general-purpose testbed for Computer Science research. Currently Chameleon is deployed at three separate host institutions and provides over six thousand CS researchers with bare metal access to a diverse range of state-of-the-art hardware configurations.

Last year we worked to replace our old legacy authentication and identity system. Users up until that time had to register for a separate account for Chameleon, and used that username and password to authenticate against Keystone, OpenStack’s identity system. We first had Keystone verify the user’s credentials against our central account database, but this made it a single point of failure, so at some point we switched to provisioning the accounts directly in Keystone at user registration time so Keystone could locally verify credentials. Because we had multiple sites at different locations that we needed to keep in sync with the accounts we had active in the system, we also experimented with running a geo-distributed MariaDB cluster just for Keystone in order to transparently share user accounts between multiple cloud sites: every OpenStack site effectively would be sharing a single user and project database, but would use its own database(s) for the other services. This however required us to coordinate changes to the Keystone database schema (e.g., during OpenStack upgrades) across all the sites, making upgrades much harder than they had to be. It also made it harder to add and integrate additional sites.

It was time for a change. We wanted a system that was simpler for users to use and be onboarded into, while also scaling well to arbitrary numbers of cloud deployment sites. Federated identity provided the solution, but it took some hacking to get us there.

Goals for a federated identity system

Early on, we identified a few goals of the system from a product perspective:

Users should be able to log in with existing credentials. Chameleon caters to the research community, many of whom already have accounts at their host institution or national laboratory. For those who do not have such accounts, we would support a general-purpose identity such as Google or ORCiD.

A user’s login session should carry across applications. Prior, users of Chameleon’s various OpenStack clouds would have to log in to each cloud separately, though with the same credentials. Single Sign-On (SSO) makes sense to use for our systems and is a better user experience.

The system should integrate with all existing applications. Besides various Keystone deployments, Chameleon also has a user management portal, which is a Django web application, and a JupyterHub deployment. The identity system should support all of these, and should be flexible enough to support more applications in the future.

CLI/API authentication must be supported. Many users use CLI interfaces to Chameleon’s cloud systems and we had to ensure we did not break this or make it significantly more onerous. If you’ve used OpenStack’s clients before, you know how burdensome figuring out the right authentication parameters can already be.

Store as little user inforation as possible. We don’t want to be responsible for storing sensitive data about users, such as passwords, which may be re-used on other sites. Similarly, we don’t want to store a lot of contact information about the users. To whatever extent we can rely on information fetched from an upstream source, such as Google or Globus, we should. This approach is also in-line with data privacy regulations such as GDPR.

Additional constraints

There are a few important rules about Chameleon accounts:

Users can be members of one or more projects.
Every project has one user acting as the project owner, who can add other users to the project.
Projects are authorized to use cloud resources by having an “allocation”: this is effectively a “budget” they are allowed to spend over a fixed amount of time.
Both projects and users can be disabled, e.g., if the project’s allocation expires or the project/user demonstrates irresponsible use or abuse of resources.

These constraints added some additional goals:

Authorization policies should be flexible and per-client. When a user first joins Chameleon, they will likely not be a part of any projects. Similarly, if returning after a long time, it’s possible their pre-existing projects have expired. Depending on the application, users not belonging to any authorized projects should still have access, i.e., login shouldn’t simply fail as a general rule.

Changes to project memberships should propagate as soon as possible. Users often wrote in to support asking why they didn’t have access to a project, when in fact they did, but the change had not yet been synced to all the Keystone services.

Users should not be allowed to perform actions under disabled projects. This is somewhat obvious, but important enough to call out, as it’s how we prevent unauthorized usage of cloud resources.

Final design

Ultimately, we decided on a design that uses Keycloak as a central identity provider. Keycloak keeps track of all users, projects (groups) and the group memberships. From an application like Keystone’s perspective, Keycloak serves as the OpenID Provider (OP). The applications use standard OpenID Connect authentication flows to log in the user and obtain some claims about the user from Keycloak. Project memberships are included in claims so that at login time, the client application knows what projects the user belongs to. This is particularly important for Keystone, which has no support for fetching user information after login (via, e.g. OpenID’s UserInfo endpoint).

Keystone needed several adjustments to properly integrate with our design, which demanded more from the system than it could support by default. The rest of this post goes into detail about those modifications. OpenStack is open source, let’s take advantage of that fact!

A brief guide to Keystone federation

Keystone’s own documentation will do a far better job than I of explaining what federated identity is in general, and how Keystone supports it. The main thing to know is that Keystone supports registering a mapping, expressed as a JSON file following a defined schema, which describes how OpenID Connect (OIDC) or SAML claims should map to Keystone entities. A claim is some metadata that is attached to the authentication token generated by the identity provider, and typically has some basic information about the user, such as their username, email address, and possibly other contact details. The specifics are ultimately up to the identity provider. This means that if you control the identity provider implementation, you have a lot of options open to you. Claims for OIDC are encoded as JSON Web Tokens (JWTs), so they can look something like this:

{
  "FirstName": "James",
  "LastName": "Kirk",
  "Email": "jameskirk@example.com",
  "Groups": ["Staff", "Bridge"]
}

Here is a simple Keystone federation mapping, just so you have an idea. With this mapping, when a user logs in, Keystone will automatically provision a federated user named “{FirstName} {LastName}” with the email “{Email}”, where those bracketed parts would be resolved from the OIDC or SAML claims. The bracketed tokens in the mapping refer to “slots” in the “remotes” section; the 0th slot contains the value of the “FirstName” claim.

Note: the best guide for the mapping syntax is the Mapping Combinations documentation, which goes over all the pieces of the syntax and gives some examples. It is, however, not incredibly extensive. It takes a while to grok the mapping system, as it may not work in the way you immediately expect. But, it is ultimately very powerful and gets the job done.

{
  "rules": [
    {
      "local": [
        {
          "user": {
            "name": "{0} {1}",
            "email": "{2}"
          },
        }
      ],
      "remote": [
        {
          "type": "FirstName"
        },
        {
          "type": "LastName"
        },
        {
          "type": "Email"
        }
      ]
    }
  ]
}

Automatically creating users is useful, but really what we want is to associate that user with a Keystone project. Projects are central to Keystone and the other OpenStack services as they’re the main way the system tracks usage and ownership. Therefore we ultimately want to be able to sort users into projects via this mapping. Fortunately, in addition to the “user” mapping target, the engine also supports a “projects” target. When a user is mapped to a project, Keystone will automatically provision the project if it doesn’t already exist, which means the cloud operator has to do very little work ahead of time, and indeed, doesn’t need to do additional work for any additional future projects. This can however cause difficulties if projects which were never intended to wind up in Keystone get auto-provisioned, so some care must be taken to filter these out. Here’s what a mapping with projects looks like. It looks at a “Department” claim containing the name of the user’s department within the organization and puts them into a project for that department.

{
  "rules": [
    {
      "local": [
        {
          "user": {
            "name": "{0} {1}",
            "email": "{2}"
          },
        },
        {
          "projects": {
            "name": "{3}",
            "roles": [
              {
                "name": "member"
              }
            ]
          }
        }
      ],
      "remote": [
        {
          "type": "FirstName"
        },
        {
          "type": "LastName"
        },
        {
          "type": "Email"
        },
        {
          "type": "Department"
        }
      ]
    }
  ]
}

The mapping engine does have a few filtering mechanisms available, allowing operators to filter out some values of a claim containing multiple entries (such as the “groups” claim). The filter mechanism can also be used to disallow users with some claims to log in at all; this can be useful if you only want to allow a subset of users in the identity provider access to the cloud.

As mentioned eariler, rather than directly integrate Keystone with a third-party identity provider, such as GitHub, or Google, or Globus, we decided to deploy and configure our own intermediate identity provider using Keycloak. This ended up being a very good decision because it allowed us to reap the benefits of upstream identity providers such as Globus while being able to control how claims were delivered downstream to our Keycloak clients. For example, we could write a custom OIDC claim provider that returned the precise list of projects that the authenticating user was a member of, and filter that list to only include enabled projects, based on Keycloak group attributes we were managing via a separate accounting system. The ability to create custom claims came in very handy, as we’ll see below.

The missing pieces

After evaluating Keystone’s federation offering, it became clear that we would have some troubles. Fortunately, and it’s a testament to Keystone’s design that I was able to do this, all blockers were resolved with relatively minimal changes or modifications to Keystone, and in all cases we were adding new functionality to Keystone rather than making breaking changes to existing functionality. These kind of changes are easier to carry forward into future OpenStack releases even if they are not accepted by core contributors.

In all cases, I tried to structure the solution to our problem such that it addressed as wide a set of use cases as possible, rather than doing one-off hacks just for our deployment. While more difficult, the hope is this improves the chances of us releasing the patches upstream.

Multiple projects per user

The first thing I noticed was that auto-provisioning of projects didn’t work quite like I expected. In the following simple mapping, consider a test user that has OIDC claims like this:

{
  "preferred_username": "jason@example.com",
  "projects": ["MyProject", "MyOtherProject"]
}

With the following mapping, I would expect this user is added to two projects, one called MyProject and one called MyOtherProject, and each project would be lazily created if it did not already exist.

{
  "rules": [
    {
      "local": [
        {
          "user": {
            "name": "{0}"
          }
        },
        {
          "projects": [
            {
              "name": "{1}",
              "roles": {
                "name": "member"
              }
            }
          ]
        }
      ],
      "remote": [
        {
          "type": "OIDC-preferred_username"
        },
        {
          "type": "OIDC-projects"
        }
      ]
    }
  ]
}

This however was not the case! Instead, a single project was created, with the entire list of project names encoded as the name field:

+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description |                                  |
| domain_id   | b5bb9d8014a0f9b1d61e21e796d78dcd |
| enabled     | True                             |
| id          | 1352f23cd32812f4850b878ae494af78 |
| is_domain   | False                            |
| name        | ["MyProject","MyOtherProject"]   |
| options     | {}                               |
| parent_id   | b5bb9d8014a0f9b1d61e21e796d78dcd |
| tags        | []                               |
+-------------+----------------------------------+

This turned out to be because not all mapping “targets” in Keystone’s federation mapping could support the input being a list. When the input claim is a list, the desired behavior is to expand the list, mapping to N targets instead of 1.

The patch for this, keystone/727891, also adds the missing support for mapping multiple roles. We did not need support for mapping multiple roles, as all users just get the default “member” role on their projects, but it was simple enough to include and makes the behavior more consistent.

Auto-remove users

It turns out that, while users can be added to projects lazily on first login, they are never removed from these projects if their claims change later! This was important to fix, because it meant that users would retain access to any of their past projects forever.

keystone/741785 adds a new Keystone configuration setting remove_dangling_assignments, which is turned off by default to maintain old behavior. If turned on, however, users will be removed from any projects not matching their claims. This only applies to projects within the identity provider domain, so, any projects in other domains or the default domain will be untouched (Keystone isolates users/projects into one domain per identity provider configured.)

Rich claim objects

Keystone reasonably assumes that claims are going to either be strings, or lists of strings. We however had a very specific need, and while it’s a bit of a silly need relative to the technical complexity required to fulfill it, it ultimately drove a few improvements that make the Keystone mapping engine significantly more powerful.

Motivation

When users are logged in to the OpenStack GUI, in the top nav bar they have a project selector dropdown, which displays the name of the project. In Chameleon, all of our projects have immutable names–this is important because the names act as foreign keys, allowing projects to be matched up across cloud deployments. These names are however not very user-friendly: they look like a prefix followed by a set of six numbers. Users who are members of multiple projects very often get them confused. So, at some point we allowed users to “nickname” their projects with text of their choice. We store this field as an extra field on the Keystone project (Keystone’s API already supports this.) We then patched the GUI to show that field value if it was present instead of the project name. Users were happy.

With federated projects being auto-provisioned, we had to somehow sneak that nickname field in there. We could have opted to periodically sync the nickname directly to the project via some out-of-band process, but we had come this far without having to fall back on syncing, and one of the design goals of the architecture was to avoid this. And, the sync solution would have to be maintained for every Keystone deployment we had, now and in the future. So, what can we do?

The first step was to update our Keycloak IdP to return a more complex representation for the projects a user belonged to:

{
  "preferred_username": "jason@example.com",
  "projects": [
    {"name": "P-123456", "nickname": "MyProject"},
    {"name": "P-234567", "nickname": "OtherProject"}
  ]
}

So now instead of having the “projects” claim be a list of IDs/names, we have a richer representation containing both. Now we have to get Keystone to accept this new reality.

Parsing entire assertion as JSON

Keystone delegates the federation protocol’s authentication and authorization to some service sitting directly in front of Keystone. That service is responsible for doing all the handshakes and claim verification, before passing the claims to the Keystone wsgi handler. Keystone then sees those claims as environment variables or HTTP headers. For OpenID, the mod_auth_openidc Apache module is the recommended solution for this. Now, when using a richer claim structure, the first thing I noticed was that the claim wasn’t being properly passed down to Keystone, because the Apache module couldn’t understand how to parse and then re-serialize the claim; it was not expecting a nested JSON structure.

The solution to this was to configure the module to just pass all the claims directly through to Keystone in a big JSON blob. This is possible by changing the “OIDCPassIDTokenAs” mod_auth_openidc setting from “claims” to “claims payload”, meaning that both the individual claims are passed as environment/headers, but also one environment variable will contain the entire payload encoded as JSON.

I then wrote a little patch (not yet submitted) to add a new assertion_payload configuration setting, which can be set to the name of the environment variable that will hold the entire set of claims encoded as JSON. Keystone will then parse the value of that variable as JSON and update its internal representation of the assertion claims accordingly. This allowed the rich claims to make it into the mapping engine.

Note: you’ll see in the mapping examples that the “remote” section references claims without the “OIDC-“ prefix; the way I wrote the patch, the claims that come from parsing the entire JSON payload just get added w/o a prefix.

Mapping “extra” properties

Next, the “projects” mapper needed support for actually specifying this extra metadata to set on the project when it was created. Another patch adds support for this in the mapping engine, allowing you to specify an “extra” field in a project mapper to set additional fields:

{
  "rules": [
    {
      "local": [
        {
          "user": {
            "name": "{0}"
          }
        },
        {
          "projects": [
            {
              "name": "{1}",
              "extra": {
                "extra_key": "extra-value",
                "extra_key2": "extra-value2"
              },
              "roles": {
                "name": "member"
              }
            }
          ]
        }
      ],
      "remote": [
        {
          "type": "preferred_username"
        },
        {
          "type": "projects"
        }
      ]
    }
  ]
}

These extra fields can also contain tokens like “{0}”, so they can contain values mapped from claims.

Map fields over claim list items

We’re still missing the final piece: how to wire in our list of projects such that, for each project in the claim, the “name” attribute is mapped to the project name, while the “nickname” is mapped to an extra field?

To solve this, I updated the mapping engine to support more types of token placeholders than just “{0}”, “{1}”, etc. The engine now can support tokens like “{0[name]}”, which tells it:

Look at the claim referenced in the first slot of my declared ‘remotes’.

If the claim is an object, return the ‘name’ field.

If the claim is a list, return a list of all the ‘name’ fields for each item in the list.

If the claim is neither an object nor a list, or is otherwise malformed, return nothing.

If you’re into functional programming, this effectively provides a way to express a map operation, albeit with very limited inputs.

This now (finally) allows us to express our mapping using Keystone’s mapping definition:

[
  {
    "local": [
      {
        "user": {
          "name": "{0}"
        }
      },
      {
        "projects": [
          {
            "name": "{2[name]}",
            "extra": {
              "nickname": "{2[nickname]}"
            },
            "roles": [
              {
                "name": "member"
              }
            ]
          }
        ]
      }
    ],
    "remote": [
      {
        "type": "preferred_username"
      },
      {
        "type": "projects"
      }
    ]
  }
]

Note: these lookups don’t go any further than one level, so it’s not possible to do like “{0[child][name]}” or something like that. At some point one has to draw the line!

Additional traps

Once the whole integration was more or less working, it was time to polish up the user experience and deal with a few less critical issues I encountered.

Regex whitelist/blacklist

We experimented with using nested Keycloak groups (and fine grained admin permissions) to model the idea of roles within a project: some users should be able to add and remove users, while others cannot. This meant that our Keycloak groups might look like this:

/ProjectA
  /ProjectA-managers
/ProjectB
  /ProjectB-managers

The “*-managers” groups are used to apply custom Keycloak policies, allowing any user in that group to manage the parent group’s memberships. For this to work, a user will have to be in both groups, e.g., in both ProjectA and ProjectA-managers. Yet, we can see that there shouldn’t really be a Keystone project called “ProjectA-managers”, as it’s not really a “real” project.

The solution for this normally is to add the projects that should be ignored to a “blacklist” in the Keystone mapping:

"remote": [
  {
    "type": "projects",
    "blacklist": ["ProjectA-managers", "ProjectB-managers"]
  }
]

However, this doesn’t scale. Keystone’s any_one_of and not_any_of mapping filters support regexes, but this was never added to the blacklist or whitelist filters for some reason. keystone/730423 adds support for regexes to these filters, so we could now do this:

"remote": [
  {
    "type": "projects",
    "blacklist": [".*-managers$"],
    "regex": true
  }
]

This doesn’t work yet though, because we added in those fancy rich claims, which have nested fields! The filter assumes that claim will be a string or list of strings, and raises an exception. I really wanted to tell the mapping, filter out any items in the “projects” claim, if the “name” attribute on that item has a certain value. So, I had to write another patch to be able to apply filters to nested fields. This part of the mapping now finally becomes:

"remote": [
  {
    "type": "projects",
    "blacklist": {
      "name": [".*-managers$"]
    },
    "regex": true
  }
]

Loose constraints

Another interesting thing that came up was how to deal with users who were not a member of any projects. Normally, this would cause the entire mapping (and thus the login) to fail, because for a mapping to pass, all claims must be present and non-empty. From our perspective though, it’s totally fine to allow the user through. OpenStack will disallow these users from doing anything without a project, but it at least gives us a chance to show a nicer error message to them, indicating they need to be added to a project or request to be added to one.

The simplest fix was to add the idea of “optional” claims. If a claim is marked optional, then the mapping can still pass even if the claim is undefined or empty. We can then allow the “projects” list to be empty like so:

"remote": [
  {
    "type": "projects",
    "optional": true,
    "blacklist": {
      "name": [
        ".*-managers$"
      ]
    },
    "regex": true
  }
]

OpenID provider discovery

We use the wonderful Kolla Ansible to deploy and configure our OpenStack clouds. kolla-ansible/695432 added support for configuring all the relevant bits and bobs for OpenID federation, which eliminates much of the burden in setting all this up. However, Kolla Ansible’s implementation configures mod_auth_openidc for multiple OpenID providers. The reason for this is that it’s feasible that a cloud deployer would configure multiple identity providers in Keystone. When mod_auth_openidc is set up this way, though, it will take all login requests through an additional interstitial page, which asks the user to confirm which configured OP they want to use.

This is redundant and unnecessary, as Keystone already allows the user to select which IdP they want to use. We can bypass this by implementing OpenID Connect Discovery and then setting the OIDCDiscoverURL configuration option:

# Defines an external OP Discovery page. That page will be called with:
#    <discovery-url>?oidc_callback=<callback-url>
# additional parameters may be added, a.o. `target_link_uri`, `x_csrf` and `method`.
#
# An Issuer selection can be passed back to the callback URL as in:
#    <callback-url>?iss=[${issuer}|${domain}|${e-mail-style-account-name}][parameters][&login_hint=<login-hint>][&scopes=<scopes>][&auth_request_params=<params>]
# where the <iss> parameter contains the URL-encoded issuer value of
# the selected Provider, or a URL-encoded account name for OpenID
# Connect Discovery purposes (aka. e-mail style identifier), or a domain name.
# [parameters] contains the additional parameters that were passed in on the discovery request (e.g. target_link_uri=<url>&x_csrf=<x_csrf>&method=<method>&scopes=<scopes>)
#
# When not defined the bare-bones internal OP Discovery page is used.
OIDCDiscoverURL <discovery-url>

I wrote a patch to add a new Keystone endpoint hanging off of the same prefix as the rest of the federation endpoints. It is responsible for redirecting to the callback URL and setting the pre-selected identity provider parameters such as “iss”. This is possible because the discover endpoint will receive the target_link_uri parameter, which will be a Keystone OpenID URI, which helpfully contains the name of the Keystone identity provider in the path. So we can be a bit tricky and use this to look up the value for “iss”. With this in-place in Keystone, mod_auth_openidc can use OP discovery and the user no longer sees that awkward extra page in the flow.

Limited OIDC support for rich claims

One last wrinkle I encountered was that not all client libraries had good support for rich OIDC claims; similar to Keystone, they assumed claims would either be strings or a list of strings. The main offender here was JupyterHub’s OAuthenticator, but I imagine there are others.

To support such applications, I configured the Keycloak IdP to return a simpler set of claims to the clients I knew to have issues, such as JupyterHub. Instead of “projects”, we now have “project_names”:

{
  "preferred_username": "jason@example.com",
  "project_names": ["P-123456", "P-234567"]
}

But wait, there’s more…

This change had larger implications because of some capabilities our JupyterHub system has. We wanted our JupyterHub users to be able to transparently interact with the remote OpenStack sites without having to do any wrangling of OpenRC files or logging in again to each site. With federated authentication this is possible because we can piggy-back on the token JupyterHub receives when the user performs their login: the keystoneauth library supports an authentication method called v3oidcaccesstoken, which allows passing an already-generated OIDC access token straight to Keystone, rather than Keystone attempting to obtain this itself.

However, this means whatever claims JupyterHub received from the IdP will be then passed to Keystone, because they are encoded in this token! Since OAuthenticator could only accept simple claims, it would be sending those simple claims back to Keystone, which would reject them, because the mapping was expecting the rich “projects” claim.

Instead of addressing this directly in OAuthenticator, I opted to give Keystone two mappings: a preferred one, which uses the rich claims, and a “dumb” one that it can fall back to. Keystone’s mapping engine will apply the first mapping that successfully passes, so we just put the new simpler mapping after it. You can see the final mapping(s) below.

The final picture

With the patches in place, we were able to get everything we needed to work within Keystone’s existing federation mapping engine. The final mapping we configured looks something like this:

[
  {
    "local": [
      {
        "user": {
          "name": "{0}",
          "email": "{1}"
        }
      },
      {
        "projects": [
          {
            "name": "{2[name]}",
            "extra": {
              "nickname": "{2[nickname]}"
            },
            "roles": [
              {
                "name": "member"
              }
            ]
          }
        ]
      }
    ],
    "remote": [
      {
        "type": "preferred_username"
      },
      {
        "type": "email"
      },
      {
        "type": "projects",
        "optional": true,
        "blacklist": {
          "name": [
            ".*-managers$"
          ]
        },
        "regex": true
      }
    ]
  },
  {
    "local": [
      {
        "user": {
          "name": "{0}",
          "email": "{1}"
        }
      },
      {
        "projects": [
          {
            "name": "{2}",
            "roles": [
              {
                "name": "member"
              }
            ]
          }
        ]
      }
    ],
    "remote": [
      {
        "type": "OIDC-preferred_username"
      },
      {
        "type": "OIDC-email"
      },
      {
        "type": "OIDC-project_names",
        "optional": true,
        "blacklist": [
          ".*-managers$"
        ],
        "regex": true
      }
    ]
  }
]

We now had a login system for all of our OpenStack deployments, and it was generic: every Keystone system could be configured exactly the same (just changing the Keycloak client ID and secret). This is particularly powerful because we have open-sourced Chameleon’s provisioning code so that other host institutions or labs can deploy our specific infrastructure. Login with federated identity is now included by default, so any deployer of Chameleon doesn’t need to worry about it.

When a user logs in to any site, their project memberships are synced immediately. They will lose access to any projects they were removed from and will gain access to projects they were added to since the last login. They can still use the CLI: the simplest thing we found was to override the default OpenRC template in Horizon to properly configure it to use the v3oidcpassword authentication type, which effectively implements support for the OAuth2.0 Resource Owner Password Grant; the user sets a password in Keycloak and that serves as their CLI password. Users can also opt to create Keystone application credentials if that works better for them.

We encouraged all users to migrate over to the new system over a period of about six months, which went pretty smoothly all things considered. To assist in the migration, we wrote some self-service tooling that users could invoke to copy information from their legacy Keystone user and projects (which were under the “default” Keystone domain) to their “new” user and projects in the federated domain.

Full list of patches

If you’re interested in applying any of these patches to your Keystone deployment, here they are. I don’t think the order in which you apply them matters very much, as there are very few, if any, dependencies between each changest.

Submitted

keystone/727891 (not merged): create multiple projects if a claim has multiple values
keystone/741785 (not merged): automatically prune user memberships from projects
keystone/730423 (merged): regex support in whitelist/blacklist filters

Not yet submitted

Compressing PDFs on Mac OS X with custom filters

2021-03-10T00:00:00-06:00

I learned recently that not only is it possible to add custom Quartz filters to Mac OS X, it’s very easy. Quartz is a 2D graphics library that sits behind the scenes on macOS. I’m not sure how much it’s used for these days, but at some point a nice developer made it possible to create simple Quartz filters via the ColorSync Utility app. The app has some pre-defined options for common needs and it can do far more than just adjust colors, despite its name.

The built-in “Reduce File Size” filter is very aggressive: it will downsample all images to 512px wide. For PDFs where the entire page is a single image, this usually renders the document illegible. I created a “Reduce File Size (300dpi)” to downsample to a decent resolution density. It still can save tons of space on most documents.

Adding a new filter in ColorSync Utility

The app opens to the “Filters” tab by default presumably because this is its most common function. The easiest way to create a filter is to duplicate an existing one via the dropdown-arrow to the right of the filter in question. I duplicated the “Reduce File Size” filter, and then tweaked the options. I disabled “Constrain size” and instead enabled “Set Resolution” to 300 pixels/inch. I left Image Compression in the middle where it was.

The filter is automatically available when you’re done editin.

Using the new filter in Preview

You can apply the filter via the “Export…” option (not “Export as PDF…” counterintuitively.) Simple as that!

Bad luck to kill a gull

2020-01-21T00:00:00-06:00

The Lighthouse (2019)

In Robert Eggers’ latest film The Lighthouse both story and setting are stripped down to the essentials: there is a remote lighthouse off the coast; two men are sent on a four-week shift to oversee it; one, Thomas Wake, an old hand (Willem Dafoe) and the other, Ephraim Winslow, a young newcomer (Robert Pattinson). When their replacements never arrive, they are forced to ration their supplies and subsequently descend into a drunken madness admist a tempest. The spartan setup gives the two actors a lot of room to roam, but also a lot of rope to hang themselves. Pattinson’s performance is impassioned, if uneven; Dafoe is the one who manages to keep the whole thing from blowing away in the wind. Eggers’ interest with atmosphere and world-building shines through, and one can feel the quest for authenticity in the compact, grainy film treatment, the heavy clanking lighthouse machinery, the way the characters dress and speak, and everything in between. However, granting these raw materials were there to begin with, ultimately the film’s ability to synthesize them into something interesting is hamstrung by a relentless desire to challenge, subvert, and flagellate.

The wide theatrical release of The Lighthouse was buoyed by press a certain kind of film attracts these days: a cycle of retellings of its origin story and laborious production—notably, the construction of an actual lighthouse on a piece of barren rock in Nova Scotia—as well as the various hijinks of its two leading men. Robert Pattinson claimed he punched himself in the face and drank mud before scenes. The film’s legend preceeded it in a manner similar to Iñárritu’s The Revenant, also infamous for its grueling on-location shooting, dogmatic realism, and debasement of its own star, DiCaprio. In both cases, the mechanics of the film’s production arouse more fascination and accolades than the film itself, a trend that has recently found footing with films shot on an iPhone (Unsane), featuring non-actors (American Honey) or real-life relationships (The Souvenir), or utilizing CGI-assisted fur (Cats). The dual character of an unusual production choice is that it can both make an artistic statement more effective and serve as kindling for the ever-important “buzz” that can make a release critically (if not financially) successful. The danger is in the film being too fashionable for its own good.

From the first moment in The Lighthouse, a few such unique aesthetic decisions with regards to the cinematography are clear: the film is in black and white, it’s gritty, and it’s in a narrow aspect ratio. Eggers used actual black and white film stock as opposed to applying such an effect in post-production, a decision that required incredible intensity of lighting to prevent underexposure. Combined with lenses designed in the 1910s and 1930s, the result is an effect that most viewers would simply describe as “old-looking”. But one wonders if the utility of the approach extends beyond the generation of “did-you-know?”-style factoids. The modified aspect ratio certainly recalls productions of a long bygone era, and while it is used to some effect in sequences such as a close-cropped vertical boom shot up the length of the lighthouse’s exterior, the constrained frame feels gimmicky, as they often do.

The narrative style Eggers employs in the script likewise seems designed to provoke a reaction. As if aware of its own aesthetic preoccupations, the film undercuts itself, regularly featuring crude humor around flatulence and feces but also disrupting dramatic energy with lazy punchlines, such as Wake’s lengthy condemnation of Winslow around the latter’s insult to his cooking; the intensity of Dafoe’s performance in this scene (already canonized as a meme, with those in the know additionally parroting the fact that Dafoe reportedly performed the scene in just one take) is immediately evaporated by Winslow’s retort that, OK, maybe the food isn’t that bad. Moments like this are frustrating because they show that something stronger could have arisen out of the combination of setting, cast, and narrative seed. Instead, the film treats dramatic energy frivolously. While the mixing of humor and horror is of course not intrinsically a mistake and genre-blending or genre-defying films are possible and often interesting, this film reads as merely self-deprecating, using laughs to disguise a lack of confidence in its chosen direction. Eggers’ wife said it best: “I don’t think this is as funny as you think it is.”¹

The question then is what is its chosen direction? Mythology and the supernatural are central to Eggers’ acclaimed 2015 film The Witch (also a period horror, set in a desolate 1620’s New England wilderness), and he carries this focus into The Lighthouse. Much of Wake’s dialog, calling forth Triton, Proteus, Poseidon and the like, seems at times plucked straight from Melville and in fact this is even cheekily referenced by Winslow, who becomes fed up with Wake’s “Captain Ahab horseshit”. Though sound from a historical perspective—the film’s events take place a few decades after the publication of Moby Dick—the idea of these two characters being aware of the book or its characters stretches the imagination. The remark serves as joke shared with the audience and, in conjunction with the decision to casually give the Icarus story a Prometheus ending (Eggers claims the latter as inspiration²), undermines any investment in myth. Throw in a boatload of symbolism around the phallic lighthouse, keys and locks, mermaids, vengeful gulls, a violent sea and so forth, and we’re left adrift, not knowing where we’re meant to be, but also not caring.

Fixing faulty archive behavior with Office365 on Thunderbird

2020-01-09T00:00:00-06:00

I use Mozilla Thunderbird to view mail on my work Office365 account (mostly because the web interface only supports top-posting presently, and several mailing lists I participate in rightly denounce this practice). Overall it gets the job done, despite having a very primitive search capability in comparison to the Outlook web app. However, there is one issue I had, which was very difficult to solve: the archive functionality never worked properly. I recently figured out why.

I have Thunderbird configured to archive to a folder on Office365, not locally. Turns out, either by default or because I absentmindedly was fiddling with this one day, I had a setting in place that wanted to archive my mails into yearly archive folders. You can choose between “a single folder”, “yearly archived folders”, and “monthly archived folders” as of this writing. The “single folder” setting is the only one that works, the others cause the archive button to appear, and you can click it, but does nothing (no errors are displayed either.)

To change this, go to the Settings for the account in question, then go to the “Copies & Folders” subsection. Find the “Message Archives” fieldset and click “Archive options…”, and then select the “single folder” option. You can have it archive to the “Archive” folder in your Office365 account and then when you click the archive button in Thunderbird, it will function identially to the web interface.

Hope this helps someone else.

Shot Analysis: Spike Jonze’s Adaptation

2013-01-26T00:00:00-06:00

I’m currently learning and discovering more about the film medium. One of the more interesting realizations I’ve had in the process is just how much attention goes into constructing individual shots. Before I had assumed that much of this was set design and making sure actors were positioned such that the viewer could always suss out the action in the scene without much trouble. I’ve since learned, both through reading and actually practicing with setting up my own shots, how integral the shot construction is to the narrative, and how technically difficult it is to execute when you’re behind the camera.

As an exercise, I’m applying my rudimentary knowledge of the shot/narrative interaction towards the films I watch. The first film I looked at was Adaptation (2002), directed by Spike Jonze. The zany concept of the film aside - real-life Adaptation screenwriter Charlie Kaufman writes a script about his own attempts to adapt a book into a feature film, wherein his character eventually decides to instead turn the script into a story chronicling his own inability to adapt the book - the film ultimately relies on its portrayal of the shifting emotional and psychological states of the primary characters. There is Charlie Kaufman, screenwriter (played by Nicolas Cage), who lives with his twin brother Donald (also played by Nicolas Cage), also an aspiring screenwriter. Charlie’s failures as he tries to adapt the book The Orchid Thief into a movie are paralleled by his naïve brother’s immediate success with a cliché Hollywood script that Charlie disdains. The film also focuses on the writer of The Orchid Thief, Susan Orlean (played by Meryl Streep), and her relationship with the inspiration for her book, the eccentric yet intellectual orchid poacher John Laroche (Chris Cooper).

The cinematography and direction of the film really work at communicating both the relationships between characters and their individual psychologies. The effect is we, the viewers, can trace the path of Charlie, struggling with his adaptation, or Susan, working to reconcile the two lives she has been living since meeting Laroche, with greater ease because we are being led with a skilled hand.

To examine this in detail, it’s helpful to begin with a study on how the film portrays the psychology if its main characters, as it arms us with knowledge about the character that will aid the reading of more subtle shots focusing on comparing or contrasting characters.

Character Psychology

Since Charlie Kaufman is the main protagonist, he is the focus of the majority of shots that focus on communicating psychology. Early on in the film, we see Charlie at a party with a woman named Amelia who he seems to be in a relationship with. Instead of talking with the rest of the ebullient party guests, the two are secluded up the stairs in an alcove. Charlie’s discomfort, both with the party and with Amelia, is clear when we see an over-the-shoulder (OTS) shot of their conversation.

Charlie’s face is framed tightly and is visually constricted, as Amelia’s shoulder appears to be forcing him into a corner. Much of his face is also hidden in shadow (a convention that we will see many times throughout the film), adding to the sense that Charlie is insecure, or ashamed of himself, and prefers to avoid attention (or perhaps thinks he doesn’t deserve it). We are free to imagine the specific implications, but the shot puts us in the right frame of mind: Charlie is not comfortable with where he is. Besides giving us insight into his psyche, this shot also works to foreshadow the dissolving of Charlie’s relationship with Amelia (due to his inability to react to her advances) further in the film.

Charlie’s self-worth is examined further in shots of him lying in bed. After a romantic fantasy sequence with Charlie and his producer, the film cuts to Charlie alone in his room and we discover he was masturbating.

While the bed is normally an intimate place, here Jonze chose to pull back and emphasize the amount of space and emptiness around Charlie. Lighting also plays an integral role here - the cold light of the room is a sobering reminder that Charlie is nowhere near his fantasy. He is alone, almost without any color or vibrancy. The shot is also positioned so we are looking down on his frame, a classic technique to illustrate a character’s lack of power. All of these details add to the overall sense of failure that Charlie is dealing with as he tries to come up with a screenplay.

There are a few comedic moments in the film that contrasts the initial manic energy of Charlie’s creative brainstorming with the depressed self-doubt that comes when he revisits the results later. In this sequence, Charlie excitedly talks about all of his ideas for how to open his movie, then cuts to him playing back his recorded ramblings later.

Much of the comedic effect in this moment comes from the drastic differences in how Charlie is shown. In the first shot, he is close to the camera and placed in the center of the frame. He has a fair amount of light on his face, and the camera is positioned slightly below him, putting him in a position of confidence and power. In a jarring contast, the next shot shows Charlie further away from the camera (again, emphasizing the emptiness around him), largely washed out due to all the shadow, and his placement is off-center. Additionally, he has less headroom in the second shot and doesn’t fill the frame - his initial energy is depleted and is replaced by a combination of horror and despair as he comprehends how hopeless his situation is. Even if Nicolas Cage wasn’t slumped over and visibly forlorn in this shot, we would still get that impression just through the impact of the contrast with the previous shot.

As Charlie continues, with difficulty, to “find the story” in The Orchid Thief, we get a shot of him that reinforces how challenging this task is.

This is a good example of both framing and careful construction of detail. While Charlie’s room could be filled with lots of different objects, all we really see are books and his typewriter workstation. The homogeny of objects prevents us from trying to read all the individual items in the scene, and instead we have the general impression of disorder. Notice how Charlie is also hunched down and takes up a small amount of space relative to the frame. This allows him to be visually overwhelmed and penned-in by all the materials surrounding him. The piles of books and reference materials surround him almost like a circle of wagons, while his typewriter threatens his position from the left. We are left with the impression of Charlie overwhelmed entirely by his screenplay, and without escape - where is he supposed to move?

Another comedic brainstorm sequence happens later in the film, where Charlie decides that perhaps the solution to his plight is to make the film about his plight. He elatedly starts narrating ideas into his voice recorder again, but is later sieged by more feelings of doubt, wondering how narcissitic he is to make the movie revolve around him instead of the book. Then, Donald enters with the news that his first screenplay (that Charlie derides as sophomoric) was picked up by a studio.

This sequence uses many of the same techniques as the first brainstorm sequence, which actually helps the comedy. At first Charlie is well-lit and dominant, finally he is shadowy and dwarfed. We as the audience have seen this before, and already know how it turns out before it happens. We are therefore left waiting for the punchline from the beginning, which makes it even funnier when Jonze repurposes a convention from the horror/thriller to bring Donald into the scene as a looming silhouette clutching his successful screenplay: it is Charlie’s worst nightmare - that he will fail where his brother succeeds - coming to haunt him. The deliberate choice to make this scene so visually similar to the earlier brainstorm scene functions not only comedically but also thematically: we seem to know, even before Charlie, that until he changes some of his attitudes and loosens some of the pressure he places on himself to be truly original - he will never have his story.

Eventually Charlie decides, with much hesitation, to take some advice from Donald and talk with Susan, the author, in an attempt to find more material for his screenplay. When he goes to visit her office at The New Yorker, he encounters her by chance in the elevator, but is unable to talk with her before she leaves.

Both Charlie and Susan are given lots of headroom here, which actually serves two different purposes for each character. For Charlie, it emphasizes how small and pathetic he feels that he’s actually here trying to rescue his screenplay by talking with Susan. For Susan, it provides a contrast to the powerful presence she normally puts forth as a reporter. Although she may try to act one way, she has many inner doubts about her life and the direction it has taken. The empty space of the elevator weighs heavily upon both characters, and visually surrounds them; the horizontal lines of the elevator chamber help emphasize the scale of this vertical space. The characters are also separated just enough to be not touching, adding to the feeling of isolation for each. In addition to the physical separation, there is a sense of great depth in between Charlie and Susan (aided by a wide-angle lens). We are left with the impression that he has to cross a great distance to initiate conversation with her. Also notice that Charlie is once again positioned in a corner. His uncomfortable position and proximity from Susan emphasizes and illustrates the paralyzation he is experiencing from his case of writer’s block.

For a final examination of a shot that conveys information about a character’s psychology, we can look at a pivotal scene for Susan, where we discover more about the nature of her relationship with Laroche. She sits alone in a hotel room with a package of drugs (derived from orchids) that, it is revealed, she asked Laroche to procure. As she takes the drugs, she transforms, and we see how different her external persona is from her inner desires.

In the first shot Susan is positioned center-frame, as she often is in the film. However, while her centered position normally conveys dominance and confidence, in this shot it serves to emphasize her hesitation and nervousness. The overall brightness of the scene swallows her up. She is dressed in dark colors and is positioned lower in the frame so the room fills in space all around her. The sparseness of the room accentuates the professional sterility that she exhibits throughout most of the film. Once she inhales the drugs, however, it’s a different story. Jonze uses a close-up shot, de-emphasizing the room and bringing our attention on her as she completes her transformation - we are free to focus more on her difference of appearance. Her hair is down, her glasses are removed - she is both more vulnerable and perhaps more comfortable than before. In the close-up she is slightly off-center, drifting a bit from her position of power as she lets herself feel whatever is it the drug is making her feel. However, she is also bathed in the light from the lamp in the corner, a soft glow that evokes warmth and comfort.

Overall, these shots support the underlying narrative of Adaptation by chronicling the various emotional states of the main characters at each point in the narrative. However, because these shots focus on one character’s inner psychological state, they typically only involve one character at a time. For scenes where action is taking place between two or more characters, there are different dynamics at play. Dialog scenes often contain a lot of characterization, and Adaptation features many scenes between two characters that reveal much about the players or themes through comparison and contrast. The construction of the shots in these scenes therefore serves to invite these comparisons - and there are plenty of examples to look at throughout this film.

Characterization through Compare/Contrast

One of the more interesting character dynamics in Adaptation is between Donald and Charlie. While Charlie is self-loathing, anxious, and teeters between mania and depression, Donald is calm, confident, and genial. In the beginning of the film, Charlie is clearly the more successful brother from an outside perspective - coming off the success of his screenplay for Being John Malkovitch, he’s working to maintain his upward trajectory and live up to the “genius” label others have placed on him. Donald is a kind of lovable fuckup who hasn’t found any real success and decides to try screenwriting. While Donald is the more put-together emotionally, he looks up to Charlie to the point of reverence and tolerates Charlie’s harsh criticism of his work.

The power balance between Donald and Charlie is embodied physically in most scenes that contain the two brothers. Donald is consistently shown in a position substantially lower than Charlie, and has to look up to Charlie when they speak. This sustained contrast at times comes across as a sight gag, where Donald appears to be lying around everywhere.

Adaptation gets away with this partly because of a plot device (Donald has a bad back and needs to lie down), but also because it works to enforce the paradox of Donald’s implied inferiority: though he looks up to Charlie and Charlie conversely dismisses much of Donald’s creative contributions, it is actually Donald who finds success with greater ease in the story. His first screenplay is selected for production by a studio, he becomes friends with an actress from Being John Malkovitch that Charlie never received attention from, and his relationships with women blossom while Charlie’s founder. The deliberate framing of the shots with Donald and Charlie serve to amplify the inner conflict that Charlie feels regarding his creativity. Charlie sees in Donald the success that he could have if only he compromised his drive to be original and unique (by rejecting all convention), yet he can’t bring himself to come down to Donald’s level.

Another example of using framing to portray power dynamics are in the first scene where we are introduced to John Laroche. Laroche has just returned from the swamp where he was harvesting orchids with his Seminole crew when a sheriff spots their truck and catches them. Laroche is completely nonplussed by the presence of authority, and the composition of the frames both characters inhabit reinforces this.

Laroche appears in the right third of the frame, his head on a focal point in the upper-right corner. He is standing directly in the path of the road, implying an unwillingness to yield to the sheriff. The sheriff, on the other hand, stands in the center of the frame. Though he is in a dominant position, front and center with the camera pointed slightly up at him, the shot is constructed to undermine his position of power. The road behind him is noticeably wide open (the bright sun on the dusty road helps make this more prevalent), and the camera is pulled back to show less of him, making his presence less intimidating than otherwise. From this brief interchange, Laroche comes across as confident, with a distaste for those who try to exert power over him.

In another example, we see a conversation Charlie has in his agent’s office, where he tries to get out of writing his screenplay. While framing is indeed a factor in this shot, there is also a lot of attention paid to the set to really contrast the agent, who represents the business side of Charlie’s craft, and Charlie.

Notice all of the details in the shot of the agent. In many ways he is set up to be a visual foil to Charlie. His shirt is crisp and bold and includes a tie; Charlie’s unbuttoned flannel looks sloppy. The agent has a football and photos of himself (presumably) snowboarding and jetskiing. It is hard to imagine Charlie doing either of these things. The shot of the agent also includes a window, through which we can see more high-rise buildings and symbols of business. Behind Charlie we mostly see the floor. The vertical camera placement does play a role in this set of shots, similar to the Laroche/sheriff scene. The camera seems just slightly below the agent, making him appear in control and confident, while Charlie is placed below the camera so we look down on him as he admits defeat, trying to be released from his project.

While many of the scenes Charlie is in focus on portraying how at odds he is with his environment and the people around him, there are times when he is visually compared to another character, to provide insight about both. A good example of this is a match-cut leading in to Susan’s hotel scene. As Charlie sites on his bed, despondent and feeling helpless, the next shot is of Susan, in a similar position and with a similar lighting treament.

Notice the similarities between the two shots. Both are taken slightly from above (Charlie’s is from a more extreme angle), both contain lots of headroom on the characters that emphasize their crouched postures, and both cloak the characters primarily in shadow. The visual continuity serves to equivocate the mental state of both characters, one of whom - Susan - has up to this point been shown as confident and in control. By comparing her situation to Charlie’s, we are invited to believe that she is suffering from a similar internal conflict. In Susan’s case, we have less of an idea as a viewer what is going on, but we know to start looking and discovering for ourselves.

It’s In the Details

In taking a closer look at this film, I became more aware of how I respond to scenes in films as a whole. The subtle visual cues with positioning, lighting, or even lens choice all serve to crystallize the narrative in a well-shot film. We have been conditioned over hours and hours of watching movies and television to pick up on cues, and by understanding exactly how to manipulate the presentation of a shot to present cues to a viewer, a filmmaker can elicit an emotional response appropriate for the narrative they are trying to communicate. It is important to note that this post only covers (a small set of) the visual choices a director or cinematographer can make. Audio and sound design, acting, and writing all of course play a role in how the audience reacts to a film. I would simply encourage you to spend a bit of time the next time you’re enjoying a movie to think a bit about the choices the director has made - because, they are choices, and they all have a purpose, even if small.