<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>TIL on James Beith</title>
    <link>https://www.jamesbeith.co.uk/til/</link>
    <description>Recent content in TIL on James Beith</description>
    <generator>Hugo -- 0.155.2</generator>
    <language>en-gb</language>
    <lastBuildDate>Wed, 04 Feb 2026 16:00:00 +1100</lastBuildDate>
    <atom:link href="https://www.jamesbeith.co.uk/til/feed.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>How to build a simple event-driven architecture</title>
      <link>https://www.jamesbeith.co.uk/til/2026-02-04/how-to-build-a-simple-event-driven-architecture/</link>
      <pubDate>Wed, 04 Feb 2026 16:00:00 +1100</pubDate>
      <guid>https://www.jamesbeith.co.uk/til/2026-02-04/how-to-build-a-simple-event-driven-architecture/</guid>
      <description>&lt;p&gt;I recently read this interesting &lt;a href=&#34;https://hakibenita.com/django-reliable-signals&#34;&gt;Reliable Django Signals&lt;/a&gt; article which left me inspired to have a play with building something similar. I wanted build a simple event-driven architecture where:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Publishers can publish an event with a given payload&lt;/li&gt;
&lt;li&gt;One or many subscribers can subscribe to an event&lt;/li&gt;
&lt;li&gt;Events are asynchronous (as opposed to the synchronous &lt;a href=&#34;https://refactoring.guru/design-patterns/observer&#34;&gt;observer pattern&lt;/a&gt;)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I took the opportunity to try out &lt;a href=&#34;https://github.com/RealOrangeOne/django-tasks&#34;&gt;&lt;code&gt;django-tasks&lt;/code&gt;&lt;/a&gt; and &lt;a href=&#34;https://github.com/pallets-eco/blinker/&#34;&gt;&lt;code&gt;blinker&lt;/code&gt;&lt;/a&gt; for this. Here’s what I came up with.&lt;/p&gt;</description>
      <content:encoded><![CDATA[<p>I recently read this interesting <a href="https://hakibenita.com/django-reliable-signals">Reliable Django Signals</a> article which left me inspired to have a play with building something similar. I wanted build a simple event-driven architecture where:</p>
<ul>
<li>Publishers can publish an event with a given payload</li>
<li>One or many subscribers can subscribe to an event</li>
<li>Events are asynchronous (as opposed to the synchronous <a href="https://refactoring.guru/design-patterns/observer">observer pattern</a>)</li>
</ul>
<p>I took the opportunity to try out <a href="https://github.com/RealOrangeOne/django-tasks"><code>django-tasks</code></a> and <a href="https://github.com/pallets-eco/blinker/"><code>blinker</code></a> for this. Here’s what I came up with.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">import</span> dataclasses
</span></span><span style="display:flex;"><span><span style="color:#f92672">import</span> functools
</span></span><span style="display:flex;"><span><span style="color:#f92672">import</span> pkgutil
</span></span><span style="display:flex;"><span><span style="color:#f92672">from</span> typing <span style="color:#f92672">import</span> Protocol
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">import</span> blinker
</span></span><span style="display:flex;"><span><span style="color:#f92672">from</span> django_tasks <span style="color:#f92672">import</span> task
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">def</span> <span style="color:#a6e22e">publish</span>(name: str, <span style="color:#f92672">/</span>, <span style="color:#f92672">*</span>, payload: dict[str, object]) <span style="color:#f92672">-&gt;</span> <span style="color:#66d9ef">None</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#e6db74">&#34;&#34;&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    Publish an asynchronous event with the given `payload` for all receivers
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    subscribed to `name`.
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    &#34;&#34;&#34;</span>
</span></span><span style="display:flex;"><span>    signal <span style="color:#f92672">=</span> _get_namespace()<span style="color:#f92672">.</span>signal(name)
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">for</span> receiver <span style="color:#f92672">in</span> signal<span style="color:#f92672">.</span>receivers_for(blinker<span style="color:#f92672">.</span>ANY):
</span></span><span style="display:flex;"><span>        receiver_identifier <span style="color:#f92672">=</span> _get_receiver_identifier(receiver)
</span></span><span style="display:flex;"><span>        _execute_event<span style="color:#f92672">.</span>enqueue(
</span></span><span style="display:flex;"><span>            receiver_identifier<span style="color:#f92672">=</span>receiver_identifier, name<span style="color:#f92672">=</span>name, payload<span style="color:#f92672">=</span>payload
</span></span><span style="display:flex;"><span>        )
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">def</span> <span style="color:#a6e22e">subscribe</span>(name: str, <span style="color:#f92672">/</span>, <span style="color:#f92672">*</span>, receiver: Receiver) <span style="color:#f92672">-&gt;</span> <span style="color:#66d9ef">None</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#e6db74">&#34;&#34;&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    Subscribe a `receiver` callable for the given event `name`.
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    &#34;&#34;&#34;</span>
</span></span><span style="display:flex;"><span>    signal <span style="color:#f92672">=</span> _get_namespace()<span style="color:#f92672">.</span>signal(name)
</span></span><span style="display:flex;"><span>    signal<span style="color:#f92672">.</span>connect(receiver)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">@dataclasses.dataclass</span>(frozen<span style="color:#f92672">=</span><span style="color:#66d9ef">True</span>, kw_only<span style="color:#f92672">=</span><span style="color:#66d9ef">True</span>)
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">Event</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#e6db74">&#34;&#34;&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    The event object a receiver is called with.
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    &#34;&#34;&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    name: str
</span></span><span style="display:flex;"><span>    payload: dict[str, object]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">Receiver</span>(Protocol):
</span></span><span style="display:flex;"><span>    <span style="color:#e6db74">&#34;&#34;&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    Protocol for event receiver callables.
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    &#34;&#34;&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    __qualname__: str
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">def</span> <span style="color:#a6e22e">__call__</span>(self, event: Event, <span style="color:#f92672">/</span>) <span style="color:#f92672">-&gt;</span> <span style="color:#66d9ef">None</span>: <span style="color:#f92672">...</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Private</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">@functools.cache</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">def</span> <span style="color:#a6e22e">_get_namespace</span>() <span style="color:#f92672">-&gt;</span> blinker<span style="color:#f92672">.</span>Namespace:
</span></span><span style="display:flex;"><span>    <span style="color:#e6db74">&#34;&#34;&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    Return the namespace used for all events.
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    &#34;&#34;&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">return</span> blinker<span style="color:#f92672">.</span>Namespace()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">@task</span>(queue_name<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;events&#34;</span>)
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">def</span> <span style="color:#a6e22e">_execute_event</span>(
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">*</span>, receiver_identifier: str, name: str, payload: dict[str, object]
</span></span><span style="display:flex;"><span>) <span style="color:#f92672">-&gt;</span> <span style="color:#66d9ef">None</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#e6db74">&#34;&#34;&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    Call the identified receiver with the event.
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    &#34;&#34;&#34;</span>
</span></span><span style="display:flex;"><span>    receiver_callable <span style="color:#f92672">=</span> _get_receiver_callable(receiver_identifier)
</span></span><span style="display:flex;"><span>    receiver_callable(Event(name<span style="color:#f92672">=</span>name, payload<span style="color:#f92672">=</span>payload))
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">def</span> <span style="color:#a6e22e">_get_receiver_identifier</span>(receiver: Receiver, <span style="color:#f92672">/</span>) <span style="color:#f92672">-&gt;</span> str:
</span></span><span style="display:flex;"><span>    <span style="color:#e6db74">&#34;&#34;&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    Return the identifier `&lt;package&gt;:&lt;object&gt;` for the given receiver.
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    &#34;&#34;&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">return</span> <span style="color:#e6db74">f</span><span style="color:#e6db74">&#34;</span><span style="color:#e6db74">{</span>receiver<span style="color:#f92672">.</span>__module__<span style="color:#e6db74">}</span><span style="color:#e6db74">:</span><span style="color:#e6db74">{</span>receiver<span style="color:#f92672">.</span>__qualname__<span style="color:#e6db74">}</span><span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">def</span> <span style="color:#a6e22e">_get_receiver_callable</span>(receiver_identifier: str, <span style="color:#f92672">/</span>) <span style="color:#f92672">-&gt;</span> Receiver:
</span></span><span style="display:flex;"><span>    <span style="color:#e6db74">&#34;&#34;&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    Return the receiver for the given identifier.
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    &#34;&#34;&#34;</span>
</span></span><span style="display:flex;"><span>    receiver_callable: Receiver <span style="color:#f92672">=</span> pkgutil<span style="color:#f92672">.</span>resolve_name(receiver_identifier)
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">return</span> receiver_callable
</span></span></code></pre></div><p>I also configured the follow Django settings for <code>django-tasks</code> . Specifically, setting up the dedicated <code>&quot;events&quot;</code> queue.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-diff" data-lang="diff"><span style="display:flex;"><span>  INSTALLED_APPS = [
</span></span><span style="display:flex;"><span>      ...,
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">+     &#34;django_tasks&#34;,
</span></span></span><span style="display:flex;"><span><span style="color:#a6e22e">+     &#34;django_tasks.backends.database&#34;,
</span></span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">+ TASKS = {
</span></span></span><span style="display:flex;"><span><span style="color:#a6e22e">+     &#34;default&#34;: {
</span></span></span><span style="display:flex;"><span><span style="color:#a6e22e">+         &#34;BACKEND&#34;: &#34;django_tasks.backends.database.DatabaseBackend&#34;,
</span></span></span><span style="display:flex;"><span><span style="color:#a6e22e">+         &#34;QUEUES&#34;: [&#34;events&#34;],
</span></span></span><span style="display:flex;"><span><span style="color:#a6e22e">+     }
</span></span></span><span style="display:flex;"><span><span style="color:#a6e22e">+ }
</span></span></span></code></pre></div><p>To subscribe to events I could then use the following.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">from</span> django <span style="color:#f92672">import</span> apps
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">import</span> events
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">Config</span>(apps<span style="color:#f92672">.</span>AppConfig):
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">...</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">def</span> <span style="color:#a6e22e">ready</span>(self) <span style="color:#f92672">-&gt;</span> <span style="color:#66d9ef">None</span>:
</span></span><span style="display:flex;"><span>        events<span style="color:#f92672">.</span>subscribe(<span style="color:#e6db74">&#34;user.created&#34;</span>, receiver<span style="color:#f92672">=</span>user_created)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">def</span> <span style="color:#a6e22e">user_created</span>(event: events<span style="color:#f92672">.</span>Event, <span style="color:#f92672">/</span>) <span style="color:#f92672">-&gt;</span> <span style="color:#66d9ef">None</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">...</span>
</span></span></code></pre></div><p>And the following to publish events.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">import</span> events
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">def</span> <span style="color:#a6e22e">create_user</span>(<span style="color:#f92672">...</span>) <span style="color:#f92672">-&gt;</span> <span style="color:#66d9ef">None</span>:
</span></span><span style="display:flex;"><span>    user <span style="color:#f92672">=</span> <span style="color:#f92672">...</span>
</span></span><span style="display:flex;"><span>    events<span style="color:#f92672">.</span>publish(<span style="color:#e6db74">&#34;user.created&#34;</span>, payload<span style="color:#f92672">=</span>{<span style="color:#e6db74">&#34;user_id&#34;</span>: user<span style="color:#f92672">.</span>id})
</span></span></code></pre></div><p>Thanks to <a href="https://hakibenita.com/pages/about">Haki</a> for the original article, which is well worth the read.</p>
]]></content:encoded>
    </item>
    <item>
      <title>How to create a Django GeneratedField for extracting a timestamp from a UUIDv7</title>
      <link>https://www.jamesbeith.co.uk/til/2025-12-04/how-to-create-a-django-generatedfield-for-extracting-a-timestamp-from-a-uuidv7/</link>
      <pubDate>Thu, 04 Dec 2025 16:30:00 +1100</pubDate>
      <guid>https://www.jamesbeith.co.uk/til/2025-12-04/how-to-create-a-django-generatedfield-for-extracting-a-timestamp-from-a-uuidv7/</guid>
      <description>&lt;p&gt;I recently read &lt;a href=&#34;https://www.paulox.net/2025/11/14/how-to-use-uuidv7-in-python-django-and-postgresql/&#34;&gt;How to use UUIDv7 in Python, Django and PostgreSQL&lt;/a&gt; from &lt;a href=&#34;https://github.com/pauloxnet&#34;&gt;Paolo Melchiorre&lt;/a&gt;. I particularly liked the section demonstrating how to extend the model with a generated column that stores the timestamp extracted from the UUID.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-python&#34; data-lang=&#34;python&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#f92672&#34;&gt;from&lt;/span&gt; django.db &lt;span style=&#34;color:#f92672&#34;&gt;import&lt;/span&gt; models
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;class&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;UUIDv7&lt;/span&gt;(models&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;Func):
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    function &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;uuidv7&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    output_field &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; models&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;UUIDField()
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;class&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;UUIDExtractTimestamp&lt;/span&gt;(models&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;Func):
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    function &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;uuid_extract_timestamp&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    output_field &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; models&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;DateTimeField()
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;class&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;Record&lt;/span&gt;(models&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;Model):
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    uuid &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; models&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;UUIDField(db_default&lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt;UUIDv7(), primary_key&lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;True&lt;/span&gt;)
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    creation_time &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; models&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;GeneratedField(
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        expression&lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt;UUIDExtractTimestamp(&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;uuid&amp;#34;&lt;/span&gt;),
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        output_field&lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt;models&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;DateTimeField(),
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        db_persist&lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;True&lt;/span&gt;,
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    )
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;As Paolo says:&lt;/p&gt;</description>
      <content:encoded><![CDATA[<p>I recently read <a href="https://www.paulox.net/2025/11/14/how-to-use-uuidv7-in-python-django-and-postgresql/">How to use UUIDv7 in Python, Django and PostgreSQL</a> from <a href="https://github.com/pauloxnet">Paolo Melchiorre</a>. I particularly liked the section demonstrating how to extend the model with a generated column that stores the timestamp extracted from the UUID.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">from</span> django.db <span style="color:#f92672">import</span> models
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">UUIDv7</span>(models<span style="color:#f92672">.</span>Func):
</span></span><span style="display:flex;"><span>    function <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;uuidv7&#34;</span>
</span></span><span style="display:flex;"><span>    output_field <span style="color:#f92672">=</span> models<span style="color:#f92672">.</span>UUIDField()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">UUIDExtractTimestamp</span>(models<span style="color:#f92672">.</span>Func):
</span></span><span style="display:flex;"><span>    function <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;uuid_extract_timestamp&#34;</span>
</span></span><span style="display:flex;"><span>    output_field <span style="color:#f92672">=</span> models<span style="color:#f92672">.</span>DateTimeField()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">Record</span>(models<span style="color:#f92672">.</span>Model):
</span></span><span style="display:flex;"><span>    uuid <span style="color:#f92672">=</span> models<span style="color:#f92672">.</span>UUIDField(db_default<span style="color:#f92672">=</span>UUIDv7(), primary_key<span style="color:#f92672">=</span><span style="color:#66d9ef">True</span>)
</span></span><span style="display:flex;"><span>    creation_time <span style="color:#f92672">=</span> models<span style="color:#f92672">.</span>GeneratedField(
</span></span><span style="display:flex;"><span>        expression<span style="color:#f92672">=</span>UUIDExtractTimestamp(<span style="color:#e6db74">&#34;uuid&#34;</span>),
</span></span><span style="display:flex;"><span>        output_field<span style="color:#f92672">=</span>models<span style="color:#f92672">.</span>DateTimeField(),
</span></span><span style="display:flex;"><span>        db_persist<span style="color:#f92672">=</span><span style="color:#66d9ef">True</span>,
</span></span><span style="display:flex;"><span>    )
</span></span></code></pre></div><p>As Paolo says:</p>
<blockquote>
<p>A generated datetime column can be very useful even though UUIDv7 already embeds a timestamp. PostgreSQL computes it at write time, Django manages it declaratively through the ORM and having a proper datetime field makes filtering, ordering, indexing and using the Django admin much simpler without requiring annotations or extra computation on the UUID value.</p>
</blockquote>
<p><a href="https://www.paulox.net/2025/11/14/how-to-use-uuidv7-in-python-django-and-postgresql/#adding-a-generatedfield-for-extracting-creation-time">Read that specific section here</a>.</p>
]]></content:encoded>
    </item>
    <item>
      <title>How to use reserved domains in tests to prevent DNS queries</title>
      <link>https://www.jamesbeith.co.uk/til/2025-07-01/how-to-use-reserved-domains-in-tests-to-prevent-dns-queries/</link>
      <pubDate>Tue, 01 Jul 2025 16:45:00 +1000</pubDate>
      <guid>https://www.jamesbeith.co.uk/til/2025-07-01/how-to-use-reserved-domains-in-tests-to-prevent-dns-queries/</guid>
      <description>&lt;p&gt;When writing tests I use the &lt;code&gt;example.com&lt;/code&gt; domain for any URLs, email addresses, and similar. The reason: if my test unintentionally made a real request, or sent a real email, then it would resolve to a special-use domain. The Internet Assigned Numbers Authority (IANA) reserves the &lt;code&gt;example.com&lt;/code&gt; domain for documentation purposes. Better that the request resolves there than to one that’s privately owned, such as &lt;code&gt;test.com&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;While reviewing some work recently I came across a comment from &lt;a href=&#34;https://github.com/bcdickinson&#34;&gt;Ben&lt;/a&gt; about his suggestion to use the &lt;code&gt;.invalid&lt;/code&gt; top-level domain. I learned that DNS prohibits installing certain TLDs like &lt;code&gt;.example&lt;/code&gt;, &lt;code&gt;.invalid&lt;/code&gt;, &lt;code&gt;.localhost&lt;/code&gt;, and &lt;code&gt;.test&lt;/code&gt;. Using test data such as &lt;code&gt;john@example.invalid&lt;/code&gt; is safer than &lt;code&gt;john@example.com&lt;/code&gt; because requests should never even reach DNS servers. While all four TLDs receive special handling, &lt;code&gt;.invalid&lt;/code&gt; is the most non-resolving. For the technical differences between them, see sections 6.2-6.5 of &lt;a href=&#34;https://www.rfc-editor.org/rfc/rfc6761.html#section-6.2&#34;&gt;RFC 6761&lt;/a&gt;.&lt;/p&gt;</description>
      <content:encoded><![CDATA[<p>When writing tests I use the <code>example.com</code> domain for any URLs, email addresses, and similar. The reason: if my test unintentionally made a real request, or sent a real email, then it would resolve to a special-use domain. The Internet Assigned Numbers Authority (IANA) reserves the <code>example.com</code> domain for documentation purposes. Better that the request resolves there than to one that’s privately owned, such as <code>test.com</code>.</p>
<p>While reviewing some work recently I came across a comment from <a href="https://github.com/bcdickinson">Ben</a> about his suggestion to use the <code>.invalid</code> top-level domain. I learned that DNS prohibits installing certain TLDs like <code>.example</code>, <code>.invalid</code>, <code>.localhost</code>, and <code>.test</code>. Using test data such as <code>john@example.invalid</code> is safer than <code>john@example.com</code> because requests should never even reach DNS servers. While all four TLDs receive special handling, <code>.invalid</code> is the most non-resolving. For the technical differences between them, see sections 6.2-6.5 of <a href="https://www.rfc-editor.org/rfc/rfc6761.html#section-6.2">RFC 6761</a>.</p>
]]></content:encoded>
    </item>
    <item>
      <title>How to optimise the order of HTML head elements using Capo</title>
      <link>https://www.jamesbeith.co.uk/til/2025-06-23/how-to-optimise-the-order-of-html-head-elements-using-capo/</link>
      <pubDate>Mon, 23 Jun 2025 11:00:00 +1000</pubDate>
      <guid>https://www.jamesbeith.co.uk/til/2025-06-23/how-to-optimise-the-order-of-html-head-elements-using-capo/</guid>
      <description>&lt;p&gt;I recently came across &lt;a href=&#34;https://rviscomi.github.io/capo.js/&#34;&gt;Capo&lt;/a&gt;, a tool that helps identify HTML head elements that are out of order.&lt;/p&gt;
&lt;p&gt;Using the Chrome extension, it offered a few suggestions for my project.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Move the &lt;a href=&#34;https://github.com/guybedford/es-module-shims&#34;&gt;es-module-shims&lt;/a&gt; asynchronous script and &lt;code&gt;importmap&lt;/code&gt; synchronous script to be a lot higher&lt;/li&gt;
&lt;li&gt;All the synchronous styles &lt;code&gt;&amp;lt;link rel=stylesheet&amp;gt;&lt;/code&gt; to come before deferred scripts &lt;code&gt;&amp;lt;script defer src&amp;gt;&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Remove some &lt;code&gt;preload&lt;/code&gt; elements which were having little to no effect (as the files were already discoverable by other &lt;code&gt;link&lt;/code&gt; elements)&lt;/li&gt;
&lt;li&gt;Put everything else such as &lt;code&gt;meta&lt;/code&gt;, non-stylesheet &lt;code&gt;link&lt;/code&gt;, and JSON &lt;code&gt;script&lt;/code&gt; elements last&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;You can see how Capo categorises all the elements into 11 groups on the docs &lt;a href=&#34;https://rviscomi.github.io/capo.js/user/rules/&#34;&gt;rules page&lt;/a&gt;.&lt;/p&gt;</description>
      <content:encoded><![CDATA[<p>I recently came across <a href="https://rviscomi.github.io/capo.js/">Capo</a>, a tool that helps identify HTML head elements that are out of order.</p>
<p>Using the Chrome extension, it offered a few suggestions for my project.</p>
<ul>
<li>Move the <a href="https://github.com/guybedford/es-module-shims">es-module-shims</a> asynchronous script and <code>importmap</code> synchronous script to be a lot higher</li>
<li>All the synchronous styles <code>&lt;link rel=stylesheet&gt;</code> to come before deferred scripts <code>&lt;script defer src&gt;</code></li>
<li>Remove some <code>preload</code> elements which were having little to no effect (as the files were already discoverable by other <code>link</code> elements)</li>
<li>Put everything else such as <code>meta</code>, non-stylesheet <code>link</code>, and JSON <code>script</code> elements last</li>
</ul>
<p>You can see how Capo categorises all the elements into 11 groups on the docs <a href="https://rviscomi.github.io/capo.js/user/rules/">rules page</a>.</p>
]]></content:encoded>
    </item>
    <item>
      <title>How to write a custom alias for the Python debugger</title>
      <link>https://www.jamesbeith.co.uk/til/2025-06-19/how-to-write-a-custom-alias-for-the-python-debugger/</link>
      <pubDate>Thu, 19 Jun 2025 16:45:00 +1000</pubDate>
      <guid>https://www.jamesbeith.co.uk/til/2025-06-19/how-to-write-a-custom-alias-for-the-python-debugger/</guid>
      <description>&lt;p&gt;Recently shared at &lt;a href=&#34;https://kraken.tech/&#34;&gt;work&lt;/a&gt; was this TIL from Samuel, &lt;a href=&#34;https://www.samuelliedtke.com/blog/til-rich-pretty-print-in-python-debugger-with-pdbrc&#34;&gt;Custom alias for pretty printing in Python debugger with .pdbrc (including Django models!)&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Here’s my ever so slightly adapted version which also includes printing a reminder to me that these exist.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;alias rp import rich; rich.print(%*)
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;alias rpo import rich; from django import forms; rich.print(forms.model_to_dict(%*))
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print(&amp;#34;Aliases: Use `rp obj` and `rpo instance` to print objects and model instances.\n&amp;#34;)
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;I already had something similar in my &lt;a href=&#34;https://github.com/ipython/ipython&#34;&gt;IPython&lt;/a&gt; startup script using &lt;a href=&#34;https://docs.python.org/3/library/pprint.html&#34;&gt;pprint&lt;/a&gt; but I’ve now adapted it to use &lt;a href=&#34;https://github.com/Textualize/rich&#34;&gt;Rich&lt;/a&gt; too.&lt;/p&gt;</description>
      <content:encoded><![CDATA[<p>Recently shared at <a href="https://kraken.tech/">work</a> was this TIL from Samuel, <a href="https://www.samuelliedtke.com/blog/til-rich-pretty-print-in-python-debugger-with-pdbrc">Custom alias for pretty printing in Python debugger with .pdbrc (including Django models!)</a>.</p>
<p>Here’s my ever so slightly adapted version which also includes printing a reminder to me that these exist.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>alias rp import rich; rich.print(%*)
</span></span><span style="display:flex;"><span>alias rpo import rich; from django import forms; rich.print(forms.model_to_dict(%*))
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>print(&#34;Aliases: Use `rp obj` and `rpo instance` to print objects and model instances.\n&#34;)
</span></span></code></pre></div><p>I already had something similar in my <a href="https://github.com/ipython/ipython">IPython</a> startup script using <a href="https://docs.python.org/3/library/pprint.html">pprint</a> but I’ve now adapted it to use <a href="https://github.com/Textualize/rich">Rich</a> too.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">from</span> django <span style="color:#f92672">import</span> forms
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">import</span> rich
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>rp <span style="color:#f92672">=</span> rich<span style="color:#f92672">.</span>print
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">def</span> <span style="color:#a6e22e">rpo</span>(<span style="color:#f92672">*</span>args):
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">for</span> instance <span style="color:#f92672">in</span> args:
</span></span><span style="display:flex;"><span>        rich<span style="color:#f92672">.</span>print(forms<span style="color:#f92672">.</span>model_to_dict(instance))
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>print(<span style="color:#e6db74">&#34;Helpers: Use `rp(*args)` and `rpo(*args)` to print objects and model instances.</span><span style="color:#ae81ff">\n</span><span style="color:#e6db74">&#34;</span>)
</span></span></code></pre></div><p>Thanks for sharing, Samuel.</p>
]]></content:encoded>
    </item>
    <item>
      <title>How to catch missing referenced files in Django static files manifest</title>
      <link>https://www.jamesbeith.co.uk/til/2025-02-27/how-to-catch-missing-referenced-files-in-django-static-files-manifest/</link>
      <pubDate>Thu, 27 Feb 2025 12:00:00 +1100</pubDate>
      <guid>https://www.jamesbeith.co.uk/til/2025-02-27/how-to-catch-missing-referenced-files-in-django-static-files-manifest/</guid>
      <description>&lt;p&gt;I recently had a deployment fail with the following &lt;a href=&#34;https://whitenoise.readthedocs.io/en/stable/&#34;&gt;WhiteNoise&lt;/a&gt; error.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;-----&amp;gt; $ python src/manage.py collectstatic --noinput
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;       Post-processing &amp;#39;polls/style.css&amp;#39; failed!
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;       whitenoise.storage.MissingFileError: The file &amp;#39;polls/style.css.map&amp;#39; could not be found with &amp;lt;whitenoise.storage.CompressedManifestStaticFilesStorage object at 0x7f5c4b1970e0&amp;gt;.
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;       The CSS file &amp;#39;polls/style.css&amp;#39; references a file which could not be found:
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;         polls/style.css.map
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;       Please check the URL references in this CSS file, particularly any
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;       relative paths which might be pointing to the wrong location.
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;It was easy enough to fix, by adding the missing &lt;code&gt;style.css.map&lt;/code&gt; file, but I wanted to write a test that would help prevent this from happening again in the future. Here’s what I started with.&lt;/p&gt;</description>
      <content:encoded><![CDATA[<p>I recently had a deployment fail with the following <a href="https://whitenoise.readthedocs.io/en/stable/">WhiteNoise</a> error.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>-----&gt; $ python src/manage.py collectstatic --noinput
</span></span><span style="display:flex;"><span>       Post-processing &#39;polls/style.css&#39; failed!
</span></span><span style="display:flex;"><span>       whitenoise.storage.MissingFileError: The file &#39;polls/style.css.map&#39; could not be found with &lt;whitenoise.storage.CompressedManifestStaticFilesStorage object at 0x7f5c4b1970e0&gt;.
</span></span><span style="display:flex;"><span>       The CSS file &#39;polls/style.css&#39; references a file which could not be found:
</span></span><span style="display:flex;"><span>         polls/style.css.map
</span></span><span style="display:flex;"><span>       Please check the URL references in this CSS file, particularly any
</span></span><span style="display:flex;"><span>       relative paths which might be pointing to the wrong location.
</span></span></code></pre></div><p>It was easy enough to fix, by adding the missing <code>style.css.map</code> file, but I wanted to write a test that would help prevent this from happening again in the future. Here’s what I started with.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">import</span> tempfile
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">import</span> pytest
</span></span><span style="display:flex;"><span><span style="color:#f92672">from</span> pytest_django.fixtures <span style="color:#f92672">import</span> SettingsWrapper
</span></span><span style="display:flex;"><span><span style="color:#f92672">from</span> whitenoise <span style="color:#f92672">import</span> storage
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">def</span> <span style="color:#a6e22e">test_collect_static_files_manifest</span>(settings: SettingsWrapper) <span style="color:#f92672">-&gt;</span> <span style="color:#66d9ef">None</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#e6db74">&#34;&#34;&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    Test that all referenced static files exist in the manifest.
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    &#34;&#34;&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">with</span> tempfile<span style="color:#f92672">.</span>TemporaryDirectory() <span style="color:#66d9ef">as</span> directory:
</span></span><span style="display:flex;"><span>        settings<span style="color:#f92672">.</span>STATIC_ROOT <span style="color:#f92672">=</span> directory
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">try</span>:
</span></span><span style="display:flex;"><span>            call_command(<span style="color:#e6db74">&#34;collectstatic&#34;</span>, no_input<span style="color:#f92672">=</span><span style="color:#66d9ef">True</span>)
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">except</span> storage<span style="color:#f92672">.</span>MissingFileError:
</span></span><span style="display:flex;"><span>            pytest<span style="color:#f92672">.</span>fail(<span style="color:#e6db74">&#34;Missing file in the static files manifest&#34;</span>)
</span></span></code></pre></div><p>However, when I ran the test with the <code>polls/style.css.map</code> file missing, it didn’t fail. I checked the configuration of WhiteNoise in my Django settings file.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">from</span> django.conf <span style="color:#f92672">import</span> STATICFILES_STORAGE_ALIAS
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">from</span> environs <span style="color:#f92672">import</span> env
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>MIDDLEWARE <span style="color:#f92672">=</span> [
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">...</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#e6db74">&#34;whitenoise.middleware.WhiteNoiseMiddleware&#34;</span>,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">...</span>,
</span></span><span style="display:flex;"><span>]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>STORAGES <span style="color:#f92672">=</span> {
</span></span><span style="display:flex;"><span>    STATICFILES_STORAGE_ALIAS: {
</span></span><span style="display:flex;"><span>        <span style="color:#e6db74">&#34;BACKEND&#34;</span>: <span style="color:#e6db74">&#34;whitenoise.storage.CompressedManifestStaticFilesStorage&#34;</span>,
</span></span><span style="display:flex;"><span>    },
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">if</span> env<span style="color:#f92672">.</span>bool(<span style="color:#e6db74">&#34;CI&#34;</span>, <span style="color:#66d9ef">False</span>):
</span></span><span style="display:flex;"><span>    MIDDLEWARE<span style="color:#f92672">.</span>remove(<span style="color:#e6db74">&#34;whitenoise.middleware.WhiteNoiseMiddleware&#34;</span>)
</span></span><span style="display:flex;"><span>    STORAGES[STATICFILES_STORAGE_ALIAS] <span style="color:#f92672">=</span> {
</span></span><span style="display:flex;"><span>        <span style="color:#e6db74">&#34;BACKEND&#34;</span>: <span style="color:#e6db74">&#34;django.contrib.staticfiles.storage.StaticFilesStorage&#34;</span>,
</span></span><span style="display:flex;"><span>    }
</span></span></code></pre></div><p>Turned out when running the tests I had static files handled by Django, without manifest support, instead of WhiteNoise. I remembered that, as <a href="https://docs.djangoproject.com/en/5.1/ref/contrib/staticfiles/#django.contrib.staticfiles.storage.ManifestStaticFilesStorage.manifest_strict">mentioned in the Django docs</a>, you shouldn’t use a backend with manifest support when running tests.</p>
<blockquote>
<p>Due to the requirement of running <code>collectstatic</code>, this storage typically shouldn’t be used when running tests as <code>collectstatic</code> isn’t run as part of the normal test setup. During testing, ensure that <code>staticfiles</code> storage backend in the <code>STORAGES</code> setting is set to something else like <code>'django.contrib.staticfiles.storage.StaticFilesStorage'</code> (the default).</p>
</blockquote>
<p>If you do then you’d get errors similar to.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>.venv/lib/python3.13/site-packages/django/contrib/staticfiles/storage.py:516: in stored_name
</span></span><span style="display:flex;"><span>    raise ValueError(
</span></span><span style="display:flex;"><span>E   ValueError: Missing staticfiles manifest entry for &#39;polls/style.css&#39;
</span></span></code></pre></div><p>I wanted to consistently use WhiteNoise and remove the conditional logic within my Django settings file. That is, I didn’t want settings defined based on knowing what environment the application was running in. I made the following change.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-diff" data-lang="diff"><span style="display:flex;"><span>  STORAGES = {
</span></span><span style="display:flex;"><span>      STATICFILES_STORAGE_ALIAS: {
</span></span><span style="display:flex;"><span><span style="color:#f92672">-         &#34;BACKEND&#34;: &#34;whitenoise.storage.CompressedManifestStaticFilesStorage&#34;,
</span></span></span><span style="display:flex;"><span><span style="color:#a6e22e">+         &#34;BACKEND&#34;: env.str(&#34;WHITENOISE_BACKEND&#34;, default=&#34;whitenoise.storage.CompressedManifestStaticFilesStorage&#34;),
</span></span></span><span style="display:flex;"><span>      },
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">- if env.bool(&#34;CI&#34;, False):
</span></span></span><span style="display:flex;"><span><span style="color:#f92672">-     MIDDLEWARE.remove(&#34;whitenoise.middleware.WhiteNoiseMiddleware&#34;)
</span></span></span><span style="display:flex;"><span><span style="color:#f92672">-     STORAGES[STATICFILES_STORAGE_ALIAS] = {
</span></span></span><span style="display:flex;"><span><span style="color:#f92672">-         &#34;BACKEND&#34;: &#34;django.contrib.staticfiles.storage.StaticFilesStorage&#34;,
</span></span></span><span style="display:flex;"><span><span style="color:#f92672">-     }
</span></span></span></code></pre></div><p>In my test environment I set the environment variable to the WhiteNoise backend without manifest support.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>WHITENOISE_BACKEND=&#34;whitenoise.storage.StaticFilesStorage&#34;
</span></span></code></pre></div><p>I then updated the new test to explicitly use the WhiteNoise backend with manifest support.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-diff" data-lang="diff"><span style="display:flex;"><span>  import tempfile
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">+ from django.conf import STATICFILES_STORAGE_ALIAS
</span></span></span><span style="display:flex;"><span><span style="color:#a6e22e">+
</span></span></span><span style="display:flex;"><span>  import pytest
</span></span><span style="display:flex;"><span>  from pytest_django.fixtures import SettingsWrapper
</span></span><span style="display:flex;"><span>  from whitenoise import storage
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  def test_collect_static_files_manifest(settings: SettingsWrapper) -&gt; None:
</span></span><span style="display:flex;"><span>      &#34;&#34;&#34;
</span></span><span style="display:flex;"><span>      Test that all referenced static files exist in the manifest.
</span></span><span style="display:flex;"><span>      &#34;&#34;&#34;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">+     settings.STORAGES = {
</span></span></span><span style="display:flex;"><span><span style="color:#a6e22e">+         STATICFILES_STORAGE_ALIAS: {
</span></span></span><span style="display:flex;"><span><span style="color:#a6e22e">+             &#34;BACKEND&#34;: &#34;whitenoise.storage.CompressedManifestStaticFilesStorage&#34;,
</span></span></span><span style="display:flex;"><span><span style="color:#a6e22e">+         },
</span></span></span><span style="display:flex;"><span><span style="color:#a6e22e">+     }
</span></span></span><span style="display:flex;"><span>      with tempfile.TemporaryDirectory() as directory:
</span></span><span style="display:flex;"><span>          settings.STATIC_ROOT = directory
</span></span><span style="display:flex;"><span>          try:
</span></span><span style="display:flex;"><span>              call_command(&#34;collectstatic&#34;, no_input=True)
</span></span><span style="display:flex;"><span>          except storage.MissingFileError:
</span></span><span style="display:flex;"><span>              pytest.fail(&#34;Missing file in the static files manifest&#34;)
</span></span></code></pre></div><p>Now the test failed as expected with the <code>polls/style.css.map</code> file missing, and passed when I added it.</p>
<p>Bear in mind that I’m using the WhiteNoise backend with compression support in this test, to mimic my production environment. That comes with the overhead of also compressing all files during <code>collectstatic</code>. To avoid that, whilst preserving the same behaviour of the test, consider switching to Django’s <code>&quot;django.contrib.staticfiles.storage.ManifestStaticFilesStorage&quot;</code> backend and catching <code>ValueError</code> instead of <code>storage.MissingFileError</code>.</p>
<p>One last thing I noticed was that I was now getting the following warning when my other tests ran.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>UserWarning: No directory at: /.../static_root/
</span></span></code></pre></div><p>This was because my test environment runs with Django’s <code>DEBUG = False</code>. The simplest way to fix this was to enable the <a href="https://whitenoise.readthedocs.io/en/stable/django.html#whitenoise-makes-my-tests-run-slow"><code>WHITENOISE_AUTOREFRESH</code></a> setting in my test (and local) environment. This stops WhiteNoise from scanning static files on start up but other than that its behaviour should be exactly the same.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-diff" data-lang="diff"><span style="display:flex;"><span>  STORAGES = {
</span></span><span style="display:flex;"><span>      STATICFILES_STORAGE_ALIAS: {
</span></span><span style="display:flex;"><span>          &#34;BACKEND&#34;: env.str(&#34;WHITENOISE_BACKEND&#34;, default=&#34;whitenoise.storage.CompressedManifestStaticFilesStorage&#34;),
</span></span><span style="display:flex;"><span>      },
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">+ # For both performance and security reasons, `WHITENOISE_AUTOREFRESH` should not be used in production.
</span></span></span><span style="display:flex;"><span><span style="color:#a6e22e">+ WHITENOISE_AUTOREFRESH = env.bool(&#34;WHITENOISE_AUTOREFRESH&#34;, False)
</span></span></span></code></pre></div><p>In my test and local environments I set the environment variable.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>WHITENOISE_AUTOREFRESH=true
</span></span></code></pre></div>]]></content:encoded>
    </item>
    <item>
      <title>How to use Django database function expressions directly</title>
      <link>https://www.jamesbeith.co.uk/til/2024-10-02/how-to-use-django-database-function-expressions-directly/</link>
      <pubDate>Wed, 02 Oct 2024 16:00:00 +1000</pubDate>
      <guid>https://www.jamesbeith.co.uk/til/2024-10-02/how-to-use-django-database-function-expressions-directly/</guid>
      <description>&lt;p&gt;Django has a many built-in &lt;a href=&#34;https://docs.djangoproject.com/en/5.1/ref/models/database-functions/&#34;&gt;database functions&lt;/a&gt; and a &lt;a href=&#34;https://docs.djangoproject.com/en/5.1/ref/models/expressions/#django.db.models.Func&#34;&gt;documented&lt;/a&gt; &lt;code&gt;Func&lt;/code&gt; API for writing your own.&lt;/p&gt;
&lt;p&gt;Whilst writing a custom &lt;code&gt;Func&lt;/code&gt; subclass may sometimes be necessary, I learnt that there’s many cases when you can instantiate &lt;code&gt;Func&lt;/code&gt; with the necessary arguments to get what you need. For example, take the following model.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-python&#34; data-lang=&#34;python&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#f92672&#34;&gt;from&lt;/span&gt; django.contrib.postgres.fields &lt;span style=&#34;color:#f92672&#34;&gt;import&lt;/span&gt; ArrayField
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#f92672&#34;&gt;from&lt;/span&gt; django.db &lt;span style=&#34;color:#f92672&#34;&gt;import&lt;/span&gt; models
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;class&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;Post&lt;/span&gt;(models&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;Model):
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    tags &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; ArrayField(models&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;CharField(max_length&lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#ae81ff&#34;&gt;200&lt;/span&gt;))
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Making use of the &lt;code&gt;array_shuffle&lt;/code&gt; Postgres &lt;a href=&#34;https://www.postgresql.org/docs/current/functions-array.html&#34;&gt;Array Function&lt;/a&gt; and an annotation, each of the post’s tags are in a randomised order.&lt;/p&gt;</description>
      <content:encoded><![CDATA[<p>Django has a many built-in <a href="https://docs.djangoproject.com/en/5.1/ref/models/database-functions/">database functions</a> and a <a href="https://docs.djangoproject.com/en/5.1/ref/models/expressions/#django.db.models.Func">documented</a> <code>Func</code> API for writing your own.</p>
<p>Whilst writing a custom <code>Func</code> subclass may sometimes be necessary, I learnt that there’s many cases when you can instantiate <code>Func</code> with the necessary arguments to get what you need. For example, take the following model.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">from</span> django.contrib.postgres.fields <span style="color:#f92672">import</span> ArrayField
</span></span><span style="display:flex;"><span><span style="color:#f92672">from</span> django.db <span style="color:#f92672">import</span> models
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">Post</span>(models<span style="color:#f92672">.</span>Model):
</span></span><span style="display:flex;"><span>    tags <span style="color:#f92672">=</span> ArrayField(models<span style="color:#f92672">.</span>CharField(max_length<span style="color:#f92672">=</span><span style="color:#ae81ff">200</span>))
</span></span></code></pre></div><p>Making use of the <code>array_shuffle</code> Postgres <a href="https://www.postgresql.org/docs/current/functions-array.html">Array Function</a> and an annotation, each of the post’s tags are in a randomised order.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">from</span> django.db.models <span style="color:#f92672">import</span> F, Func
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>posts <span style="color:#f92672">=</span> Post<span style="color:#f92672">.</span>objects<span style="color:#f92672">.</span>annotate(
</span></span><span style="display:flex;"><span>    tags_shuffle<span style="color:#f92672">=</span>Func(F(<span style="color:#e6db74">&#34;tags&#34;</span>), function<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;ARRAY_SHUFFLE&#34;</span>),
</span></span><span style="display:flex;"><span>)
</span></span></code></pre></div><p>A second, slightly more contrived example.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">from</span> django.db <span style="color:#f92672">import</span> models
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">Post</span>(models<span style="color:#f92672">.</span>Model):
</span></span><span style="display:flex;"><span>    published_year <span style="color:#f92672">=</span> models<span style="color:#f92672">.</span>IntegerField()
</span></span><span style="display:flex;"><span>    published_month <span style="color:#f92672">=</span> models<span style="color:#f92672">.</span>IntegerField()
</span></span><span style="display:flex;"><span>    published_day <span style="color:#f92672">=</span> models<span style="color:#f92672">.</span>IntegerField()
</span></span></code></pre></div><p>This time using the <code>make_date</code> Postgres’ <a href="https://www.postgresql.org/docs/current/functions-datetime.html">Date/Time Function</a> to get a <code>date</code> object from the post’s published values.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">from</span> django.db.models <span style="color:#f92672">import</span> DateField, F, Func
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>posts <span style="color:#f92672">=</span> Post<span style="color:#f92672">.</span>objects<span style="color:#f92672">.</span>annotate(
</span></span><span style="display:flex;"><span>    published_date<span style="color:#f92672">=</span>Func(
</span></span><span style="display:flex;"><span>        F(<span style="color:#e6db74">&#34;published_year&#34;</span>),
</span></span><span style="display:flex;"><span>        F(<span style="color:#e6db74">&#34;published_month&#34;</span>),
</span></span><span style="display:flex;"><span>        F(<span style="color:#e6db74">&#34;published_day&#34;</span>),
</span></span><span style="display:flex;"><span>        output_field<span style="color:#f92672">=</span>DateField(),
</span></span><span style="display:flex;"><span>        function<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;MAKE_DATE&#34;</span>,
</span></span><span style="display:flex;"><span>    )
</span></span><span style="display:flex;"><span>)
</span></span></code></pre></div>]]></content:encoded>
    </item>
    <item>
      <title>How to enqueue Celery tasks using RabbitMQ message priorities</title>
      <link>https://www.jamesbeith.co.uk/til/2024-08-28/how-to-enqueue-celery-tasks-using-rabbitmq-message-priorities/</link>
      <pubDate>Wed, 28 Aug 2024 13:30:00 +1000</pubDate>
      <guid>https://www.jamesbeith.co.uk/til/2024-08-28/how-to-enqueue-celery-tasks-using-rabbitmq-message-priorities/</guid>
      <description>&lt;p&gt;I have some slow running, low priority Celery tasks that I don’t want holding up more important tasks. I learnt how to configure Celery tasks, and queues, to support message priorities. Note, the project in question uses Django, Celery, and RabbitMQ.&lt;/p&gt;
&lt;p&gt;Firstly, I needed to decide on my &lt;em&gt;priority values&lt;/em&gt;. Whilst RabbitMQ supports priorities between 1 and 255, their docs highly recommend using values between 1 and 5.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;It is important to know that higher priority values require more CPU and memory resources, since RabbitMQ needs to internally maintain a sub-queue for each priority from 1, up to the maximum value configured for a given queue.&lt;br&gt;
— &lt;a href=&#34;https://www.rabbitmq.com/docs/priority&#34;&gt;https://www.rabbitmq.com/docs/priority&lt;/a&gt;&lt;/p&gt;</description>
      <content:encoded><![CDATA[<p>I have some slow running, low priority Celery tasks that I don’t want holding up more important tasks. I learnt how to configure Celery tasks, and queues, to support message priorities. Note, the project in question uses Django, Celery, and RabbitMQ.</p>
<p>Firstly, I needed to decide on my <em>priority values</em>. Whilst RabbitMQ supports priorities between 1 and 255, their docs highly recommend using values between 1 and 5.</p>
<blockquote>
<p>It is important to know that higher priority values require more CPU and memory resources, since RabbitMQ needs to internally maintain a sub-queue for each priority from 1, up to the maximum value configured for a given queue.<br>
— <a href="https://www.rabbitmq.com/docs/priority">https://www.rabbitmq.com/docs/priority</a></p>
</blockquote>
<p>I decided on using values between 1 and 3, which conveniently map to low, medium and high priority.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">Priority</span>(enum<span style="color:#f92672">.</span>IntEnum):
</span></span><span style="display:flex;"><span>    LOW <span style="color:#f92672">=</span> <span style="color:#ae81ff">1</span>
</span></span><span style="display:flex;"><span>    MEDIUM <span style="color:#f92672">=</span> <span style="color:#ae81ff">2</span>
</span></span><span style="display:flex;"><span>    HIGH <span style="color:#f92672">=</span> <span style="color:#ae81ff">3</span>
</span></span></code></pre></div><p>I then needed to configure Celery by updating a few settings in my Django settings module. Note, I’m not using the <code>Priority</code> class here as that’s declared in another module that I didn’t want my Django settings module to depend upon.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#75715e"># The default priority for all tasks.</span>
</span></span><span style="display:flex;"><span>CELERY_TASK_DEFAULT_PRIORITY <span style="color:#f92672">=</span> <span style="color:#ae81ff">2</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># The default max priority for all queues.</span>
</span></span><span style="display:flex;"><span>CELERY_TASK_QUEUE_MAX_PRIORITY <span style="color:#f92672">=</span> <span style="color:#ae81ff">3</span>
</span></span></code></pre></div><p>I wanted to use the RabbitMQ priority feature on all my queues, but if I’d only wanted it on certain queues I could have configured those using <code>&quot;x-max-priority&quot;</code> like so.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span>CELERY_QUEUE_DEFAULT <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;celery&#34;</span>
</span></span><span style="display:flex;"><span>CELERY_QUEUE_PRIORITY <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;priority&#34;</span>
</span></span><span style="display:flex;"><span>CELERY_TASK_QUEUES <span style="color:#f92672">=</span> [
</span></span><span style="display:flex;"><span>    Queue(
</span></span><span style="display:flex;"><span>        name<span style="color:#f92672">=</span>CELERY_QUEUE_DEFAULT,
</span></span><span style="display:flex;"><span>        exchange<span style="color:#f92672">=</span>Exchange(CELERY_QUEUE_DEFAULT),
</span></span><span style="display:flex;"><span>        routing_key<span style="color:#f92672">=</span>CELERY_QUEUE_DEFAULT,
</span></span><span style="display:flex;"><span>    ),
</span></span><span style="display:flex;"><span>    Queue(
</span></span><span style="display:flex;"><span>        name<span style="color:#f92672">=</span>CELERY_QUEUE_PRIORITY,
</span></span><span style="display:flex;"><span>        exchange<span style="color:#f92672">=</span>Exchange(CELERY_QUEUE_PRIORITY),
</span></span><span style="display:flex;"><span>        routing_key<span style="color:#f92672">=</span>CELERY_QUEUE_PRIORITY,
</span></span><span style="display:flex;"><span>        queue_arguments<span style="color:#f92672">=</span>{<span style="color:#e6db74">&#34;x-max-priority&#34;</span>: <span style="color:#ae81ff">3</span>}
</span></span><span style="display:flex;"><span>    ),
</span></span><span style="display:flex;"><span>]
</span></span></code></pre></div><p>After configuring Celery I ran into the following error starting up a worker.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>amqp.exceptions.PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg &#39;x-max-priority&#39; for queue &#39;celery&#39; in vhost &#39;/&#39;: received the value &#39;3&#39; of type &#39;signedint&#39; but current is none
</span></span></code></pre></div><p>This is because existing queues can’t change from a classic queue into priority queue, or vice versa. More specifically, the number of priorities a queue supports can’t change after queue declaration.</p>
<p>To resolve this issue, on my local machine I used the following <code>rabbitmqadmin</code> command to delete all my queues. Starting up a Celery worker created all my queues again now configured as priority queues.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span><span style="color:#75715e"># Caution! Delete all queues from RabbitMQ.</span>
</span></span><span style="display:flex;"><span>rabbitmqadmin --format<span style="color:#f92672">=</span>tsv --quiet list queues name | <span style="color:#66d9ef">while</span> read name; <span style="color:#66d9ef">do</span> rabbitmqadmin --quiet delete queue name<span style="color:#f92672">=</span><span style="color:#e6db74">${</span>name<span style="color:#e6db74">}</span>; <span style="color:#66d9ef">done</span>
</span></span></code></pre></div><p>In production, the process was somewhat similar in that I created a new RabbitMQ instance and migrated the workers to point to the new instance. Again, starting up a production Celery worker created all my queues now configured as priority queues.</p>
<p>Lastly, I configured my appropriate Celery tasks using the <code>Priority</code> class.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#a6e22e">@app.task</span>(priority<span style="color:#f92672">=</span>Priority<span style="color:#f92672">.</span>LOW)
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">def</span> <span style="color:#a6e22e">generate_report</span>():
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">...</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">@app.task</span>(priority<span style="color:#f92672">=</span>Priority<span style="color:#f92672">.</span>HIGH)
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">def</span> <span style="color:#a6e22e">send_email</span>():
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">...</span>
</span></span></code></pre></div><p>Some final notes. I’d already done so, but it’s worth disabling Celery <a href="https://docs.celeryq.dev/en/stable/userguide/configuration.html#worker-prefetch-multiplier">worker prefetching</a> with priority queues so workers always fetch the highest prioritised task. Also, worth bearing in mind how priority queues <a href="https://www.rabbitmq.com/docs/priority#interaction-with-other-features">interact with other features</a>, in particular queues which have a max-length set.</p>
]]></content:encoded>
    </item>
    <item>
      <title>How to encode Unicode characters to store in AWS S3 object metadata</title>
      <link>https://www.jamesbeith.co.uk/til/2024-08-05/how-to-encode-unicode-characters-to-store-in-aws-s3-object-metadata/</link>
      <pubDate>Mon, 05 Aug 2024 09:00:00 +1000</pubDate>
      <guid>https://www.jamesbeith.co.uk/til/2024-08-05/how-to-encode-unicode-characters-to-store-in-aws-s3-object-metadata/</guid>
      <description>&lt;p&gt;I recently ran into &lt;a href=&#34;https://github.com/boto/botocore/blob/1.34.130/botocore/handlers.py#L629-L638&#34;&gt;this error&lt;/a&gt; uploading images to AWS S3 using the &lt;code&gt;boto3&lt;/code&gt; package.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Parameter validation failed:&lt;br&gt;
Non ascii characters found in S3 metadata for key &amp;ldquo;filename&amp;rdquo;, value: &amp;ldquo;ACME™ Anvil.jpg&amp;rdquo;.&lt;br&gt;
S3 metadata can only contain ASCII characters.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;The character in question was ™. Here’s a simplified example of the code.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-python&#34; data-lang=&#34;python&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;filename &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;ACME™ Anvil.jpg&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;metadata: dict[str, str] &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; {
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;filename&amp;#34;&lt;/span&gt;: filename,
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;client&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;upload_fileobj(
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    Fileobj&lt;span style=&#34;color:#f92672&#34;&gt;=...&lt;/span&gt;,
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    Bucket&lt;span style=&#34;color:#f92672&#34;&gt;=...&lt;/span&gt;,
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    Key&lt;span style=&#34;color:#f92672&#34;&gt;=...&lt;/span&gt;,
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    ExtraArgs&lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt;{&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;Metadata&amp;#34;&lt;/span&gt;: metadata}
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;)
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;I wanted to preserve the original filename, so I learnt to use &lt;code&gt;backslashreplace&lt;/code&gt; when encoding the filename to ASCII. Note, I have to subsequently use &lt;code&gt;.decode()&lt;/code&gt; as both metadata keys and values must be &lt;code&gt;str&lt;/code&gt;, not &lt;code&gt;bytes&lt;/code&gt;.&lt;/p&gt;</description>
      <content:encoded><![CDATA[<p>I recently ran into <a href="https://github.com/boto/botocore/blob/1.34.130/botocore/handlers.py#L629-L638">this error</a> uploading images to AWS S3 using the <code>boto3</code> package.</p>
<blockquote>
<p>Parameter validation failed:<br>
Non ascii characters found in S3 metadata for key &ldquo;filename&rdquo;, value: &ldquo;ACME™ Anvil.jpg&rdquo;.<br>
S3 metadata can only contain ASCII characters.</p>
</blockquote>
<p>The character in question was ™. Here’s a simplified example of the code.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span>filename <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;ACME™ Anvil.jpg&#34;</span>
</span></span><span style="display:flex;"><span>metadata: dict[str, str] <span style="color:#f92672">=</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#e6db74">&#34;filename&#34;</span>: filename,
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>client<span style="color:#f92672">.</span>upload_fileobj(
</span></span><span style="display:flex;"><span>    Fileobj<span style="color:#f92672">=...</span>,
</span></span><span style="display:flex;"><span>    Bucket<span style="color:#f92672">=...</span>,
</span></span><span style="display:flex;"><span>    Key<span style="color:#f92672">=...</span>,
</span></span><span style="display:flex;"><span>    ExtraArgs<span style="color:#f92672">=</span>{<span style="color:#e6db74">&#34;Metadata&#34;</span>: metadata}
</span></span><span style="display:flex;"><span>)
</span></span></code></pre></div><p>I wanted to preserve the original filename, so I learnt to use <code>backslashreplace</code> when encoding the filename to ASCII. Note, I have to subsequently use <code>.decode()</code> as both metadata keys and values must be <code>str</code>, not <code>bytes</code>.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span>metadata: dict[str, str] <span style="color:#f92672">=</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#e6db74">&#34;filename&#34;</span>: filename<span style="color:#f92672">.</span>encode(<span style="color:#e6db74">&#34;ascii&#34;</span>, <span style="color:#e6db74">&#34;backslashreplace&#34;</span>)<span style="color:#f92672">.</span>decode(),
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>The encoded result is now compatible with the S3 metadata requirements.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">&gt;&gt;&gt;</span> filename<span style="color:#f92672">.</span>encode(<span style="color:#e6db74">&#34;ascii&#34;</span>, <span style="color:#e6db74">&#34;backslashreplace&#34;</span>)<span style="color:#f92672">.</span>decode()
</span></span><span style="display:flex;"><span><span style="color:#e6db74">&#39;ACME</span><span style="color:#ae81ff">\\</span><span style="color:#e6db74">u2122 Anvil.jpg&#39;</span>
</span></span></code></pre></div><p>And I could use <code>unicode-escape</code> to retrieve the original filename later.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">&gt;&gt;&gt;</span> <span style="color:#e6db74">&#34;ACME</span><span style="color:#ae81ff">\\</span><span style="color:#e6db74">u2122 Anvil.jpg&#34;</span><span style="color:#f92672">.</span>encode(<span style="color:#e6db74">&#34;ascii&#34;</span>)<span style="color:#f92672">.</span>decode(<span style="color:#e6db74">&#34;unicode-escape&#34;</span>)
</span></span><span style="display:flex;"><span><span style="color:#e6db74">&#39;ACME™ Anvil.jpg&#39;</span>
</span></span></code></pre></div><p>At first, I thought this didn’t work for all Unicode characters, namely <a href="https://www.unicode.org/emoji/charts/emoji-zwj-sequences.html">emoji ZWJ sequences</a> which use <code>U+200D</code> to join the characters into a single glyph, for example.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">&gt;&gt;&gt;</span> <span style="color:#e6db74">&#34;😵‍💫&#34;</span><span style="color:#f92672">.</span>encode(<span style="color:#e6db74">&#34;ascii&#34;</span>, <span style="color:#e6db74">&#34;backslashreplace&#34;</span>)<span style="color:#f92672">.</span>decode()
</span></span><span style="display:flex;"><span><span style="color:#e6db74">&#39;</span><span style="color:#ae81ff">\\</span><span style="color:#e6db74">U0001f635</span><span style="color:#ae81ff">\\</span><span style="color:#e6db74">u200d</span><span style="color:#ae81ff">\\</span><span style="color:#e6db74">U0001f4ab&#39;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">&gt;&gt;&gt;</span> <span style="color:#e6db74">&#34;</span><span style="color:#ae81ff">\\</span><span style="color:#e6db74">U0001f635</span><span style="color:#ae81ff">\\</span><span style="color:#e6db74">u200d</span><span style="color:#ae81ff">\\</span><span style="color:#e6db74">U0001f4ab&#34;</span><span style="color:#f92672">.</span>encode(<span style="color:#e6db74">&#34;ascii&#34;</span>)<span style="color:#f92672">.</span>decode(<span style="color:#e6db74">&#34;unicode-escape&#34;</span>)
</span></span><span style="display:flex;"><span><span style="color:#e6db74">&#39;😵</span><span style="color:#ae81ff">\u200d</span><span style="color:#e6db74">💫&#39;</span>
</span></span></code></pre></div><p>Thanks to <a href="https://github.com/quicklizard99">Lawrence Hudson</a> they pointed out that it was the REPL showing the <a href="https://docs.python.org/3/library/functions.html#repr">printable representation</a> of the result rather than the Unicode string. Using <code>print()</code> on the result highlighted this.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">&gt;&gt;&gt;</span> print(<span style="color:#e6db74">&#34;</span><span style="color:#ae81ff">\\</span><span style="color:#e6db74">U0001f635</span><span style="color:#ae81ff">\\</span><span style="color:#e6db74">u200d</span><span style="color:#ae81ff">\\</span><span style="color:#e6db74">U0001f4ab&#34;</span><span style="color:#f92672">.</span>encode(<span style="color:#e6db74">&#34;ascii&#34;</span>)<span style="color:#f92672">.</span>decode(<span style="color:#e6db74">&#34;unicode-escape&#34;</span>))
</span></span><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010">😵‍💫</span>
</span></span></code></pre></div>]]></content:encoded>
    </item>
    <item>
      <title>How to create a Django model index with an upper case empty string condition</title>
      <link>https://www.jamesbeith.co.uk/til/2024-05-02/how-to-create-a-django-model-index-with-an-upper-case-empty-string-condition/</link>
      <pubDate>Thu, 02 May 2024 15:50:00 +1000</pubDate>
      <guid>https://www.jamesbeith.co.uk/til/2024-05-02/how-to-create-a-django-model-index-with-an-upper-case-empty-string-condition/</guid>
      <description>&lt;p&gt;Let’s say I have the following Django model.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-python&#34; data-lang=&#34;python&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#f92672&#34;&gt;from&lt;/span&gt; django.db &lt;span style=&#34;color:#f92672&#34;&gt;import&lt;/span&gt; models
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;class&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;User&lt;/span&gt;(models&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;Model):
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    name &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; models&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;CharField(blank&lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;True&lt;/span&gt;)
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;And I want to filter those like so.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-python&#34; data-lang=&#34;python&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;users &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; User&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;objects&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;filter(name__istartswith&lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;John&amp;#34;&lt;/span&gt;)
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;That would perform the following query.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-postgresql&#34; data-lang=&#34;postgresql&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;SELECT&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;data_user&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#ae81ff&#34;&gt;.&lt;/span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;id&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;data_user&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#ae81ff&#34;&gt;.&lt;/span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;name&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;FROM&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;data_user&amp;#34;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;WHERE&lt;/span&gt; UPPER(&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;data_user&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#ae81ff&#34;&gt;.&lt;/span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;name&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#f92672&#34;&gt;::&lt;/span&gt;text) &lt;span style=&#34;color:#66d9ef&#34;&gt;LIKE&lt;/span&gt; UPPER(&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#39;John%&amp;#39;&lt;/span&gt;)
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;I can improve the performance of that query by adding the following index to the model. To complement the use of &lt;code&gt;UPPER()&lt;/code&gt; in the query I can use the &lt;code&gt;Upper()&lt;/code&gt; function for the index’s expression.&lt;/p&gt;</description>
      <content:encoded><![CDATA[<p>Let’s say I have the following Django model.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">from</span> django.db <span style="color:#f92672">import</span> models
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">User</span>(models<span style="color:#f92672">.</span>Model):
</span></span><span style="display:flex;"><span>    name <span style="color:#f92672">=</span> models<span style="color:#f92672">.</span>CharField(blank<span style="color:#f92672">=</span><span style="color:#66d9ef">True</span>)
</span></span></code></pre></div><p>And I want to filter those like so.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span>users <span style="color:#f92672">=</span> User<span style="color:#f92672">.</span>objects<span style="color:#f92672">.</span>filter(name__istartswith<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;John&#34;</span>)
</span></span></code></pre></div><p>That would perform the following query.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-postgresql" data-lang="postgresql"><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> <span style="color:#e6db74">&#34;data_user&#34;</span><span style="color:#ae81ff">.</span><span style="color:#e6db74">&#34;id&#34;</span>, <span style="color:#e6db74">&#34;data_user&#34;</span><span style="color:#ae81ff">.</span><span style="color:#e6db74">&#34;name&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">FROM</span> <span style="color:#e6db74">&#34;data_user&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">WHERE</span> UPPER(<span style="color:#e6db74">&#34;data_user&#34;</span><span style="color:#ae81ff">.</span><span style="color:#e6db74">&#34;name&#34;</span><span style="color:#f92672">::</span>text) <span style="color:#66d9ef">LIKE</span> UPPER(<span style="color:#e6db74">&#39;John%&#39;</span>)
</span></span></code></pre></div><p>I can improve the performance of that query by adding the following index to the model. To complement the use of <code>UPPER()</code> in the query I can use the <code>Upper()</code> function for the index’s expression.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">from</span> django.db <span style="color:#f92672">import</span> models
</span></span><span style="display:flex;"><span><span style="color:#f92672">from</span> django.db.models.functions <span style="color:#f92672">import</span> Upper
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">User</span>(models<span style="color:#f92672">.</span>Model):
</span></span><span style="display:flex;"><span>    name <span style="color:#f92672">=</span> models<span style="color:#f92672">.</span>CharField(blank<span style="color:#f92672">=</span><span style="color:#66d9ef">True</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">class</span> <span style="color:#a6e22e">Meta</span>:
</span></span><span style="display:flex;"><span>        indexes <span style="color:#f92672">=</span> [
</span></span><span style="display:flex;"><span>            models<span style="color:#f92672">.</span>Index(Upper(<span style="color:#e6db74">&#34;name&#34;</span>), name<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;name_idx&#34;</span>),
</span></span><span style="display:flex;"><span>        ]
</span></span></code></pre></div><p>As <code>name</code> could be blank, I can improve the index by adding the following condition to only index rows where the name isn’t blank.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span><span style="color:#f92672">from</span> django.db <span style="color:#f92672">import</span> models
</span></span><span style="display:flex;"><span><span style="color:#f92672">from</span> django.db.models.functions <span style="color:#f92672">import</span> Upper
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">class</span> <span style="color:#a6e22e">User</span>(models<span style="color:#f92672">.</span>Model):
</span></span><span style="display:flex;"><span>    name <span style="color:#f92672">=</span> models<span style="color:#f92672">.</span>CharField(blank<span style="color:#f92672">=</span><span style="color:#66d9ef">True</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">class</span> <span style="color:#a6e22e">Meta</span>:
</span></span><span style="display:flex;"><span>        indexes <span style="color:#f92672">=</span> [
</span></span><span style="display:flex;"><span>            models<span style="color:#f92672">.</span>Index(
</span></span><span style="display:flex;"><span>                Upper(<span style="color:#e6db74">&#34;name&#34;</span>),
</span></span><span style="display:flex;"><span>                name<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;name_idx&#34;</span>,
</span></span><span style="display:flex;"><span>                condition<span style="color:#f92672">=~</span>models<span style="color:#f92672">.</span>Q(name<span style="color:#f92672">=</span>Upper(models<span style="color:#f92672">.</span>Value(<span style="color:#e6db74">&#34;&#34;</span>))),
</span></span><span style="display:flex;"><span>            ),
</span></span><span style="display:flex;"><span>        ]
</span></span></code></pre></div><p>Thanks to <a href="https://github.com/timb07">Tim Bell</a> I learnt that for the condition I needed to pass <code>models.Value(&quot;&quot;)</code> as the argument to the <code>Upper()</code> function. Previously, I was only passing an empty string <code>condition=~models.Q(name=Upper(&quot;&quot;))</code>, and whilst Django successfully created a migration for that index, when attempting to migrate I got the <a href="https://github.com/django/django/blob/5.0.4/django/db/models/sql/query.py#L1772-L1775">following error raised</a>.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>Operations to perform:
</span></span><span style="display:flex;"><span>  Apply all migrations: admin, auth, contenttypes, data, sessions
</span></span><span style="display:flex;"><span>Running migrations:
</span></span><span style="display:flex;"><span>  Applying data.0001_initial...Traceback (most recent call last):
</span></span><span style="display:flex;"><span>  File &#34;project/src/manage.py&#34;, line 22, in &lt;module&gt;
</span></span><span style="display:flex;"><span>    main()
</span></span><span style="display:flex;"><span>  ...
</span></span><span style="display:flex;"><span>  File &#34;project/.venv/lib/python3.12/site-packages/django/db/models/sql/query.py&#34;, line 1772, in names_to_path
</span></span><span style="display:flex;"><span>    raise FieldError(
</span></span><span style="display:flex;"><span>django.core.exceptions.FieldError: Cannot resolve keyword &#39;&#39; into field. Choices are: id, question_text
</span></span></code></pre></div><p>It appears in this incorrect case <code>Upper(&quot;&quot;)</code> is referring to applying the function <code>Upper()</code> to the nonexistent model field named <code>&quot;&quot;</code>.</p>
]]></content:encoded>
    </item>
  </channel>
</rss>
