Greg Wilson was soliciting strategies for lazy-loading datasets in Python modules. There are, of course, many ways to do this, but I didn't see this one being discussed.

Since Python 3.7 (released in 2017) you can define a module-level __getattr__ function that gets used for name resolution (see PEP 562). You can use this to call a data-loading function the first time each module-level property is accessed. This implementation uses only functions, dicts, and the one magic “dunder” function, which might be more approachable than programming with classes (depending on your audience).

A sketch of an implementation:

# Cache loaded datasets
_dataset_cache = {}

# Dictionary mapping dataset names to loader functions
_dataset_loaders = {
  # maps names to argument-less functions that load datasets
}


def __getattr__(name):
    """PEP 562 module-level __getattr__ function for lazy loading"""
    # Check if we should load a dataset for this attribute
    if name in _dataset_loaders:
        # If not already cached, load and cache it
        if name not in _dataset_cache:
            _dataset_cache[name] = _dataset_loaders[name]()
        return _dataset_cache[name]

    # If not a dataset, raise AttributeError
    raise AttributeError(f"Module has no attribute '{name}'")

For example, this module has attributes foo and bar which are only calculated when they're first referenced:

# foo.py

# Cache loaded datasets
_dataset_cache = {}


def load_data(name):
    if name == "foo":
        print("loading foo")
        return 1
    elif name == "bar":
        print("loading bar")
        return 2
    else:
        # This should be unreachable, but just in case...
        raise AttributeError(f"Module has no attribute '{name}'")


# Dictionary mapping dataset names to loader functions
_dataset_loaders = {
    "foo": lambda: load_data("foo"),
    "bar": lambda: load_data("bar"),
}


def __getattr__(name):
    """PEP 562 module-level __getattr__ function for lazy loading"""
    # Check if we should load a dataset for this attribute
    if name in _dataset_loaders:
        # If not already cached, load and cache it
        if name not in _dataset_cache:
            _dataset_cache[name] = _dataset_loaders[name]()
        return _dataset_cache[name]

    # If not a dataset, raise AttributeError
    raise AttributeError(f"Module has no attribute '{name}'")

In use, this looks like:

>>> import foo
>>> foo.foo
loading foo
1
>>> foo.foo
1
>>> # NOTE: didn't load foo a second time
>>> foo.bar
loading bar
2
>>> foo.quuz
Traceback (most recent call last):
  File "<python-input-4>", line 1, in <module>
    foo.quuz
  File "/Users/nknight/tmp/foo.py", line 34, in __getattr__
    raise AttributeError(f"Module has no attribute '{name}'")
AttributeError: Module has no attribute 'quuz'

I've known for a while that uv can run Python scripts that declare inline dependencies.

I learned today that you can also use uv to manage those dependencies.

If you have a file foo.py:

#!/usr/bin/env -S uv run

print("your code here")

you can run uv add --script foo.py numpy and it becomes:

#!/usr/bin/env -S uv run
# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "numpy",
# ]
# ///

print("your code here")

This is obviously nice if, like me, you have trouble remembering the exact syntax for declaring the inline dependencies, but it also means uv does things like dependency resolution so you don't have to manage your script's dependencies by hand.

You can add --script to:

• uv add and uv remove to add and remove dependencies
• uv lock and uv sync to create and synchronize with the script's lock file
• uv tree to print the script's transitive dependencies
• uv export to export the script's dependencies to another format, such as requirements.txt

(via epistasis on HN)

Semantic versioning is difficult. Not everyone has the same idea of what “breaking change” means, and depending on your language and tooling “breaking” changes can sneak in despite your best efforts.

One possible mitigation is to record when you resolved your dependencies and ignore anything published after that date. It's crude, and relies on package dependencies not monkeying with stuff that's already been published, but it's easy to understand and you can do it with uv.

It's documented here.

Put a date in your pyproject.toml file and subsequent invocations of uv will ignore anything published after the cutoff:

[tool.uv]
exclude-newer = "2023-10-16T00:00:00Z"

You can also use it in the “inline metadata” format for scripting! From the uv docs:

# /// script
# dependencies = [
#   "requests",
# ]
# [tool.uv]
# exclude-newer = "2023-10-16T00:00:00Z"
# ///

import requests

print(requests.__version__)

This use case seems particularly valuable: if I write a quick script I probably care more about it not breaking than I care about it getting updated dependencies. It's nice that uv offers a way to keep code running rather than forcing me to update it or throw it out.

I just released version 0.1 of a plugin for Simon Willison's llm called llm-questioncache. It lets you send questions to your default LLM with a system prompt that elicits short, to-the-point answers. It also maintains a cache of answers locally so that you only have to hit the LLM once for each bit of esoteric knowledge.

It uses embeddings of each question to find similar questions so that (for example) if you ask

How do you compare two branches in git

and

How to compare different branches in git

you'll get the same answer.

If you've already got LLM installed you can try it out with

llm install llm-questioncache

Here's the PyPI package: https://pypi.org/project/llm-questioncache/

And here's the source code: https://github.com/nathanielknight/llm-questioncache

On a recent project I found myself needing one classic form-and-template style page in an otherwise API-driven project. I could, of course, just do it with a regular view function, but I had a bunch of authentication and suchlike set up for DRF APIViews.

Turns out it's actually pretty easy to make an APIView kick it oldschool!

from rest_framework.parsers import FormParser
from rest_framework.renderers import TemplateHTMLRenderer
from rest_framework.response import Response
from rest_framework.views import APIView


class OldSchoolView(APIView):
    parser_classes = [FormParser]
    renderer_classes = [TemplateHTMLRenderer]
    template_name = "path/to/a/template.html"

    def get(self, request):
        context = {}
        # do regular view stuff here
        return Response(context)

    def post(self, request):
        formdata = request.data.dict()
        # do regular view stuff here
        return Response()  # or redirect; whatever's appropriate

The indicated template gets rendered with the context you pass to the Response.

I think you could write a view that simultaneously supports HTML and JSON with content negotiation, but it ended up being more trouble that it was worth in my case.

This article describes how to get the Helix text editor set up to be a half-decent Python IDE. You'll need to know a little bit about the command line, but if you're using Helix you'll probably be fine.

Instructions

To assess how well Helix can support a particular language with the tools you already have, you can run:

hx --health python

Which will show you something like:

Configured language servers:
  ✘ pylsp: 'pylsp' not found in $PATH
Configured debug adapter: None
Configured formatter: None
Highlight queries: ✓
Textobject queries: ✓
Indent queries: ✓

Helix needs us to install a Python Language Server. Rather than putting tools in our global Python environment or re-installing everything for each project, we'll use Pipx to install it in its own virtualenv (npx is an analogous tool). If you don't have it installed, you can probably find instructions here.

With Pipx installed, first we'll install the Python LSP Server

pipx install python-lsp-server

Because it aims to support multiple tools for things like typechecking and formatting, pylsp uses plugins to support those features, which need to be installed as separate packages. Luckily, Pipx supports this fairly well with its inject command. For example, to add support for [Black] and [Ruff], we can run:

pipx inject python-lsp-server python-lsp-ruff python-lsp-black

Helix also requires a little bit of configuration in its languages file so that it can tell pylsp which plugins to use (on Mac OS and Linux, this file should be at ~/.config/helix/languages/toml):

[language-server.pylsp.config.pylsp]
plugins.ruff.enabled = true
plugins.black.enabled = true

With that, Helix should be a reasonably fully featured Python IDE.

Why do we need to do all this?

Rather than building plugins for each programming language, Helix embraces the Language Server Protocol. Helix doesn't know about refactoring, type inference, etc. It just knows how to speak LSP, and relies on a separate tool (a “language server”) to provide all those nice features.

The downside of this approach is that the responsibility for providing a language server falls to the user. The benefit is that Helix, despite being a relatively new project, has out-of-the box integration with over 200 languages, from common ones like Python and TypeScript to more niche languages like Gleam, Ada, and Unison.

As I've been learning Helix I've been dreading the point where I had to do some customization to get some of the features I'm used to in an editor like VS Code. I'm not very fond of configuring text editors, having lost a great many focused hours to wrangling VimScript and Emacs Lisp so I wasn't looking forward to learning how Helix handles this, but I admit: setting Helix up this way was much easier than I was afraid it might be.

A useful question for understanding software tools is to peel back a layer of abstraction and ask what the thing underneath is. For example:

• On a computer, text is a sequence of numbers (and a system to interpret that as letters).
• An HTTP request is a blob of text with a particular format.
• An interpreter (like the Python or Ruby interpreters) is a program, whose function is to execute other programs.

Knowing this sort of thing is useful when the abstractions break (e.g. when you open a text file with the wrong encoding) or when thinking about the fundamental contours of a system's possibility space. For example, if you know that programs can be configured with environment variables and CLI flags, knowing that the Python interpreter is a program means you know where to start looking if you need to configure it.

While there are many, many articles on the internet explaining how and why to use them, there's less information about what a Python virtual environment (or “virtualenv”) is. Luckily for us, it ends up not being very difficult to investigate.

Looking inside a virtualenv

Creating a virtualenv is the work of a moment. If you have Python installed, you can get one of your very own like so:

> python -m venv ./venv

This creates a directory called venv. Let's see what's in it!

> cd venv
> ls
...

Listing the directory's contents reveals three directories and a file:

• Include/, which appears to be empty,
• Lib/, which contains a directory called site-packages,
• Scripts/, which contains a bunch of executables, including the Python interpreter and the activate/deactivate scripts for this virtualenv, and
• pyvenv.cfg, a config file with the Python version the path to my system Python interpreter.

(Note: This was run on a Windows machine; you'll get different-but-analogous results if you run it on Linux or MacOS).

I know from prior experience that site-packages is where Python packages get installed, that “include” is probably something to do with compiled extensions. It also makes sense that there would be a central place to put scripts, so that the virtualenv's activation script can add it to your shell's PATH. So a virtualenv is a directory full of things for Python to import or execute. But how does it work?

Understanding virtualenvs

The documentation of virtualenvironments is, again, very helpful if you want to understand how to use a virtualenvironment, but not particularly illuminating on the topic of its inner workings.

Instead the answer is in the documentation for the site package: it turns out that when the Python interpreter starts up it looks for a pyvenv.cfg file one directory above itself. If it finds one, it knows its in a virtualenv and configures itself accordingly.

Now we know a few things about virtualenvs:

• You can use the Python interpreter at venv/scripts/python without activating the virtualenv.

• If you need to find the source code of a dependency (for debugging, say) you can find it in venv/Lib/site-packages/.

• You can ruin your virtualenv by messing with the pyvenv.cfg file (or, possibly, un-mess it to fix a ruined virtualenv).

When I'm doing data analysis or building applications with Python and I have to give entities a unique ID, I like to use random UUIDs instead of sequential numbers. Sequential numbers include information about the order and total number of data, but I want my IDs to be just a unique identifier, nothing more.

Python's standard library includes the uuid module, for working with UUIDs. There's a convenient function for generating random ones:

>>> import uuid
>>> uuid.uuid4()
UUID('189afb2c-1d58-4390-b35e-d5c0e3bb7472')
>>> uuid.uuid4()
UUID('0fde2d22-1918-4e39-8c6c-825c2655cbd5')

Handy. 🙂

I like my UUIDs random, but it can be useful to have them be consistent between runs. That way, you can re-run data processing scripts but keep the same UUIDs. This is a little trickier than I thought it would be, but it's certainly possible.

My first instinct was to set the random seed in the random module, which is usually enough to make the random number generator behave the same way with each run:

>>> import random
>>> random.seed("peanutbutter")
>>> [random.randint(0, 100) for _ in range(12)]
[79, 83, 26, 76, 3, 4, 2, 12, 22, 75, 34, 15]
>>> random.seed("peanutbutter")
>>> [random.randint(0, 100) for _ in range(24)]
[79, 83, 26, 76, 3, 4, 2, 12, 22, 75, 34, 15]

Setting the seed makes random.randint give us consistent results. Maybe it will work for uuid.uuid4 as well.

>>> import uuid
>>> import random
>>> random.seed("peanutbutter")
>>> uuid.uuid4()
UUID('37b88a25-9f0f-4308-9532-84fd9a924c06')
>>> random.seed("peanutbutter")
>>> uuid.uuid4()
UUID('04961188-33db-4ad9-86da-e9fcfc6a22e1')

Huh. That didn't work. What gives? 🤔

Let's look at the source code for uuid.uuid4:

def uuid4():
    """Generate a random UUID."""
    return UUID(bytes=os.urandom(16), version=4)

We can see that it's using os.urandom instead of the random module. That function goes straight to the operating system's random number generator, which isn't affected by random.seed. That's definitely a good thing! The random module is a pseudo random number generator, and using it in some kinds of application could cause security vulnerabilities, so for those applications, os.urandom is the right choice.

However, it's not what I want for my data analysis application, so how can we make it use a pseudo-random generator instead?

We can see that uuid.uuid4 is making uuid.UUID objects. If we can provide our own, pseudo-random bytes, we can generate pseudo-random UUIDs instead.

uuid.UUID wants a bytes object, which we can make from a sequence of integers, like this:

>>> integers = [1, 2, 4, 8, 16]
>>> bytes(integers)
b'\x01\x02\x04\x08\x10'

We can get a sequence of random, 8-bit integers (i.e. bytes) from random using the random.getrandbits function

Putting it all together, we get something like this:

>>> import random
>>> import uuid
>>> def random_uuid():
...     return uuid.UUID(bytes=bytes(random.getrandbits(8) for _ in range(16)), version=4)
...
>>> random.seed("peanutbutter")
>>> random_uuid()
UUID('dad39ff6-a734-4906-8804-182dda97441f')
>>> random.seed("peanutbutter")
>>> random_uuid()
UUID('dad39ff6-a734-4906-8804-182dda97441f')

Success! 🎉

That's how to generate consistent (pseudo) random UUIDs with Python's standard library.

One final note: the code above uses the shared random number generator in the random module, so if you need independent sequences of random UUIDs (e.g. for running isolated tests in parallel) it might be better to use separate random number generators for each sequence. I've written up a sample implementation of how to do that, which is available here.