sharphall.org

I’ve been using NixOS recently on my laptop, because I know how great declarative configuration is from my experience deploying software on servers with tools like Kubernetes and Terraform. Along the way, I’ve had a few experiences with it that have left me impressed with how powerful a declarative approach to the configuration of my personal computers can be.

Declarative vs imperative

If you’ve used any electronic device that you can install software on, you are already aware of the imperative approach to device configuration.

If a piece of software doesn’t exist on your machine and you want it to, you run an installer or tell the package manager to install it:

apt-get install -y neovim

A problem with installing packages this way is that over time you may build a collection of software with incompatible versions of dependencies. You might want to upgrade all the software on your machine, but you don’t have any guarantee that the upgrade will succeed. And if you do break something, the only way to fix it might be to reinstall. You might even want to reinstall every couple years anyways to start fresh.

NixOS lets you define the configuration of your system declaratively. The software you want to be installed on your system is listed (or enabled) in a config file — configuration.nix — and after you add new software to this file, you tell NixOS to rebuild your entire system by running nixos-rebuild switch.

{ config, pkgs, ... }:
{
  # ...
  programs.thunderbird.enable = true;

  programs.fish.enable = true;
  users.defaultUserShell = pkgs.fish;

  programs.neovim = {
    enable = true;
    defaultEditor = true;
  };

  # Allow unfree packages
  nixpkgs.config.allowUnfree = true;

  # List packages installed in system profile. To search, run:
  # $ nix search wget
  environment.systemPackages = with pkgs; [
    #  vim # Do not forget to add an editor to edit configuration.nix! The Nano editor is also installed by default.
    wget
  ];
}

Every time you rebuild your system, Nix “evaluates” all the packages you’ve chosen and their dependencies, and saves the output — the build artifacts — in the Nix store. Usually, nix doesn’t need to build the packages you’ve chosen and they can be downloaded directly from a Nix “cache server”. After download or build, the packages involved in the active configuration have their binaries symlinked into a special directory included in the PATH.

On my machine, this is /run/current-system/sw/, and /run/current-system/sw/bin/nvim is a symlink to /nix/store/xz3jy2lzzzycq3dnavn7v1hyr598pyr9-neovim-0.11.5/bin/nvim, for example. In fact, /run is mounted as a tmpfs that is setup when the machine boots or the configuration is switched. Changing the configuration is an atomic operation that can be reversed easily, including by rebooting and selecting an older configuration from the GRUB startup menu. A NixOS system, including the installed packages and system-wide configuration, is essentially itself a build artifact — the result of an evaluation of your configuration.nix.

Package installation and system configuration happen in one place

There are a number of configuration options available in NixOS …

Recently I’ve been learning and building in Rust to the point where I’m feeling productive with it.

Like many others, I’ve found the “The Rust Programming Language” book (or “the book” as my rustacean friends like to call it) to be an incredibly helpful resource. But as I’ve progressed beyond the book I’ve found that reading the source code of the standard library and the crates I was using was extremely illuminating, and revealed common motifs in Rust development.

There are a few things I’ve found that I thought might be especially helpful for those new to Rust, which I’ve collected here.

Rc<RefCell<T>> (or Arc<RwLock<T>> or Arc<Mutex<T>>)

This one is in the book, but it’s in the last 3/4, and if you’re jumping into the language you might be worried that data structures that were easy in other languages might seem impossible in Rust.

Rc<T> and its thread-safe counterpart Arc<T> allow you to reference-count (or in the case of Arc, atomically reference-count) allocated memory.

.clone() on an Rc<T> will no longer .clone() (and copy) the wrapped value. Instead, a counter representing how many times the Rc is owned will be incremented and the same Rc will be effectively owned in multiple places. As these owned references are dropped the counter decrements, and when the last reference is finally dropped, Rc will drop the wrapped value before it is removed itself.

Rc implements Deref (but not DerefMut), which allows you to easily access the value it wraps via “Deref coercion”.

Meanwhile, RefCell<T> (or RwLock<T> and Mutex<T>) lets you separate the mutability of the type that they wrap from the struct or reference that holds it. This is called “interior mutability”. Anywhere a RefCell, RwLock or Mutex is borrowed as immutable, you can try to mutably borrow the interior value with .borrow_mut or similar.

RefCell<T> and Rc<T> (and their respective thread-safe counterparts) can be combined to create a value that can be shared (with Rc) and mutated in multiple places (inside of a RefCell, which temporarily provides a borrowed mut value).

• Rc<T> can be owned in many places with .clone()
• Each clone of Rc points to the same immutable value
• You can ask to mutably borrow the value inside the RefCell anywhere the RefCell is immutably borrowed
• A RefCell<T> wrapped in Rc can be borrowed immutably anywhere the Rc is owned
• You can ask to mutably borrow the value inside the RefCell anywhere the Rc is owned
• With this pattern, the interface to ask to borrow a shared value mutably can have many owners

One data structure that might be impossible or needlessly difficult to create in Rust without reference counting and interior mutability is a graph. Below is an example that uses reference counting with Rc<T> to create nodes, one of which is referenced multiple times from multiple other nodes. RefCell<T> is utilized …

I tried out the Hyprland tiling window manager today, and wrote up some notes that might make a useful cheatsheet for new users.

Configuration, as it turns out, can get pretty involved. These notes are just intended to give a lay of the land, at times from the point of view of someone who wants to get it to work on Debian.

Using Hyprland

Unlike Gnome or KDE, Hyprland is extremely minimal, and a lot of features provided by desktop environments you will have to add piecemeal.

On first use:

<meta>-q to open up a terminal. kitty must be installed, or after having Hyprland write a basic config on first launch, you can change it to a terminal you have. I use rio and it works great. 🦀

You’ll probably use <meta>-m to exit a number of times while you’re still getting a working config set up.

Open up ~/.config/hypr/hyprland.conf in an editor. You can find out the default keybinds by reading the config, but to get started:

• <meta>-{1, 2, 3, 4...} to switch to a workspace
• <shift><meta>-{1, 2, 3, 4...} to move the current window to a workspace
• <meta>-v to untile or re-tile a window
• <meta>-t to toggle horizontal and vertical split
• <meta>-p to toggle “psuedo” for a tile: a window can be resized to be smaller within its tile
• <meta> and click and drag to move a window
• <meta> and right-click and drag to resize
• <meta>-c close window — think like <Ctrl>-C
• <meta>-s toggles a special workspace that you can stash things in, on top of the other workspaces. <shift><meta>-s to move a window into it.

I changed the movefocus binds to be like vim:

bind = $mainMod, h, movefocus, l
bind = $mainMod, l, movefocus, r
bind = $mainMod, k, movefocus, u
bind = $mainMod, j, movefocus, d

I have have trouble getting a right-click-and-drag with my trackpad, so in the general section I set resize_on_border to true.

I prefer my Caps Lock to be an Esc key, which can be achieved by setting kb_options = caps:swapescape in the input section.

Launcher

In order to start apps outside of a terminal — without using a terminal — you’ll need a launcher. wofi is standard and is configured by default, but since it isn’t included you’ll need to install it separately.

Once you do you can press <meta>-r to run software.

waybar

waybar is a statusbar that can show you what workspace is selected, net stats, battery, etc.

It’s configured by two files, ~/.config/waybar/config.jsonc and ~/.config/waybar/style.css.

Unlike Hyprland there is no automatic reload. killall -USR2 waybar will force one, but watch out: old CSS is not cleared.

The basic config assumes you have a Font Awesome font installed. I did, but only most of the icons showed up. I ended up using a Nerd Font font I already had installed, and overrode the config with NF icons …

A few weeks ago I started working on s-rack, a modular synthesizer written in Rust. An earlier post describes my first week of development.

The big news since then is that this project now runs inside a web browser through WASM as a compilation target.

I followed the example at eframe_template. I found that Trunk is a builder for WASM apps. During development it will watch your code, recompile it when it changes, inject a link to the WASM binary into an HTML template you provide, and cause the page to reload. It will also generate a production deployable build in a dist/ folder when you are ready.

I refactored out the audio setup code into its own struct and associated functions so that the code in main — which is different on native than in WASM — doesn’t have to be concerned with it. After making this change and making it so the audio engine only starts after a user interaction on web, the WASM version of s-rack was building without any issue.

Input and output handles

When I wrote about this project a week ago, I anticipated that I would need to create widgets with drag and drop behaviors to act as the input and output “plugs” next to modules.

I found that with egui it was very easy to create widget like objects that could take an egui Ui, call its .allocate_space(), and test for interactions with the pointer including with drag and drop.

In fact, it was extremely easy to implement dragging an output into an input or vice versa, as the Response.dnd_* functions allow payloads of any type. For example, I used Response.dnd_set_drag_payload on each port widget to make the information about the port (a pointer to the module and which port it is) draggable to other ports that can receive it with Response.dnd_release_payload.

Oscillator aliasing

As I predicted, naively generating waveforms with fast transitions like a square wave or sawtooth wave results in undesired artifacts — extra tones which seem unrelated to the tone being generated.

Luckily, adopting a PolyBLEP function to generate corrections for each hard transition turned out to be really easy and worked like magic.

Warning! Annoying noises!

Execution planning

Though after the user has built a patch a module may have more than one (or no) connections to other modules, when the audio driver asks for audio, we want to run each module’s audio processing code once and only once. Additionally, we want to run the modules in an optimal order.

Consider the graph above, made up of modules. (Direction of arrows signify a dependency, not the direction of signal flow, which is reverse.) Modules take input buffers of a certain size, process them together with their internal state, and produce one or more output buffers. A produces an output buffer consumed by B.

Periodically the sound library asks for a new buffer of audio to send to the sound driver, which triggers the execution of all modules …

To kick off my time at the Recurse Center I decided to build a modular softsynth in Rust to expand my knowledge of Rust and DSP, as well as build something I and others could use to make music. I’m inspired by an old modular softsynth I used to use called SpiralSynthModular. Unfortunately, this software is dated and no longer updated, and doesn’t compile easily in a modern environment.

SpiralSynthModular, an old modular softsynth that left an impression on me

The Synth Engine

To begin implementing a modular synthesizer, I started by implementing the concept of modules.

In the physical world, a module is an electronic circuit with voltage inputs and outputs. Each module does one thing and one thing well. For example, one module could generate a tone whose pitch is determined by an input voltage. Another module could play a sequence of voltages into this “oscillator” to create different tones over time. Yet another module could take the output of the oscillator and filter it to produce sound of different timbre. And another module could prepare the signal for connection with the correct voltage levels to an amplifier and speaker.

A real, physical, Doepfer A-100 modular synthesizer. (Image credit)

In software, I decided to make each type of synth module its own struct, but with a SynthModule trait common across all kinds of modules, providing a common interface for connecting modules together, executing the signal processing defined inside them, and fetching output buffers from them.

pub trait SynthModule {
    // Required methods
    fn get_name(&self) -> String;
    fn get_id(&self) -> String;
    fn calc(&mut self);
    fn get_num_inputs(&self) -> u8;
    fn get_num_outputs(&self) -> u8;
    fn get_input(
        &self,
        input_idx: u8
    ) -> Result<Option<(Arc<RwLock<dyn SynthModule + Send + Sync>>, u8)>, ()>;
    fn get_inputs(
        &self
    ) -> Vec<Option<(Arc<RwLock<dyn SynthModule + Send + Sync>>, u8)>>;
    fn get_output(&self, output_idx: u8) -> Result<&[f32], ()>;
    fn set_input(
        &mut self,
        input_idx: u8,
        src_module: Arc<RwLock<dyn SynthModule + Send + Sync>>,
        src_port: u8
    ) -> Result<(), ()>;
}

Arc<RwLock<dyn SynthModule + Send + Sync>> is a type wrapper around SynthModule that allows the workspace to reference all the modules and for all connected modules to reference each other, even across threads.

After the SynthModules are connected to each other, with inputs referencing the connected shared SynthModule, we need to build an execution order from the graph. I took a naive stab at this just to get the audio flowing. At the very least, it ensures every module in the connected graph connected to the output executes exactly once per cycle, initiated by a request for a buffer from the audio library. This allows for cycles in the graph with the tradeoff that there will be a delay introduced where cycles are formed. At least that’s the theory, anyway. I’m not sure that my current implementation of the execution planner is optimal and I’l probably have to revisit it later.

A few days in, I had an implementation of a Oscillator module – limited to sine wave output — and an Output module which …

nanometer is a small framework I made in TypeScript to control ILDA show lasers via the Ether Dream DAC.

It consists of two parts:

• A WebSocket server, which requests a stream of points (defined as an X/Y position and RGB color values) from a connected client and forwards them to the DAC using its TCP interface.
• A library for writing clients, which connect to the server and provide points. Included in the library are helper classes for defining computational point generators, as well as functions for assembling multiple point generators into a scene, complete with 3-D translation, rotation, and projection!

Splitting it up like this over WebSockets lets you rapidly iterate without having to reconnect to the laser and lets you write a client from within a web browser including within a framework like p5.js.

nanometer uses @laser-dac‘s Ether Dream driver and its virtual laser which can run inside of a web browser to give the developer freedom to prototype a client without a physical laser.

Here’s an example demo that simply generates points, and here’s another one that uses the included vector graphics helper library.

This labor day weekend I decided to see if I could complete a small web project that might be of use to Philadelphians and maybe learn React in the process. I came up with a mobile friendly page that will tell you how far away your bus is with the API that SEPTA makes available.

Behind the scenes, it grabs your geolocation from your device, finds stops near your location, filters all SEPTA vehicles by the routes serving the stops near your location, calculates bearing to the vehicles, compares the bearings to the headings to see if the vehicles are incoming, and displays it in a list sorted by distance.

It does all this without sending your location anywhere.

SEPTA does not provide a big list of stops and the routes that serve them. I had to generate that from their GTFS data with a small script.

Visit it on your desktop or phone at wheresepta.com. Accuracy is not guaranteed; the data is only as good as SEPTA provides it. The data has been known to be shaky and completely wrong at times, even in SEPTA’s official app. Check out the source code at GitLab.

Let me know what you think!

After creating a small analysis of campaign finance data for the Blackwell and Gauthier campaigns, I thought it would be interesting to see on a map where financial contributions come from.

I extended my script to geocode all of the addresses of donors and output them to a CSV file. I wrote another script to read this CSV file and write another file with a point per $50, with the location randomized within its square on a grid.

I then loaded this file into QGIS and overlayed it on top of Philadelphia Neighboorhood outlines and county outlines.

Caveats

• This is a very quick analysis. Some entries were dropped because of geocoding errors, and I didn’t follow up to see if they were big donations or small donations.
• As noted in my earlier analysis, the donor reported addresses aren’t always useful for determining residency (but they are probably good enough in most cases.)
• This misses a whole lot of donations that came from outside the map. I didn’t even spend the time to find out how much, but it’s probably quite a bit. See the list of donors to get a sense of how much came from other cities or states.

As always, questions, comments, etc are welcome at sharp@sharphall.org.

With campaign finance an issue in the 2019 election for councilperson for Philadelphia’s 3rd district, I thought I would list and sort sources of funding for both candidates for both 2018 + 2019 to see if there is anything that can be learned. This information is from the public record at https://apps.phila.gov/campaign-finance/search/all, but after processing the listing I generated sums of donations from the same donor, rather than listing each donation separately.

These data can’t account for non-cash support from entities, for example the flyers and door hangers from Philadelphia 3.0 or flyers of undisclosed origin in support of the Blackwell campaign.

If you have comments about this presentation of data or otherwise want to get in touch, at sharp@sharphall.org.

In full disclosure, I am a Gauthier supporter and have donated to her campaign.

Updates since publishing

• May 8, 10:16am - A misspelling in the source data caused “Laborers District Council (PAC)” to not be summed as one donor. This has been corrected.
• May 9, 10:09am - I re-ran and added the analysis for 2019 only.

Process

• I wrote a python script to parse exported data from https://apps.phila.gov/campaign-finance/search/all to sort donors for each candidate by total contribution, and also generated binned counts of donors by total donation size.
• The 2019 CSV produced by Philadelphia’s site has many errors due to commas not being escaped. (2018 uses tabs instead of commas. As far as machine readability it was totally fine.) I had to write my script to raise an error when input was not in the correct format so I could find the erroneous lines and correct them.
• Once I was generating output, I noticed a couple of records that should have been summed together but were not because of a truncation error or because a word in one address line was abbreviated in one record but not in another. I wrote code into my script to find these records and associate them on the fly, instead of changing the data files. The files were only altered to assist with parsing badly formed comma separated fields, as above. I was still seeing some of these until recently. If you catch one I haven’t, email sharp@sharphall.org.

Initial observations

• Jamie Gauthier made a donation to her own campaign, as did some relatives or family-members (presumably) who share the Gauthier name. The name Blackwell does not appear in Jannie Blackwell’s list of donors.
• Blackwell’s top funders: unions and real estate.
• Gauthier’s top funders: an environmental conservation organization and venture capitalists; yes, Perelman is in there too. Those accused of making dark in-kind donations to Gauthier through Philadelphia 3.0 are also here in plain sight.
• Gauthier has a strong base of small individual donors where Blackwell does not.
• Address may in some cases be misleading for determining residency, for example in the case of Gauthier supporters Josh and Rena Kopelman who …

I’ve got some pictures from when I attended SHA2017.